This project focuses on the creation of an interactive art installation that conveys the story, in audible format, of the reduction in genome diversity as consequence of plant domestication. The plants featured in this work are:
1_ Rice
Wild rice ----------------------- Oryza barthii
Domesticated rice --------------- Oryza sativa
2_ Corn
Wild corn ----------------------- Teosinte (Zea mays ssp. parviglumis)
Domesticated corn --------------- Zea mays
The assumption implicit in this project establishes that when DNA sequence from a domestication gene is sonificated (rendered in audible format), the resulting sound composition derived from the wild species will be significantly different to that from the domesticated one. The rationale behind this assumption lies on knowledge indicating that repeated cycles of breeding and recurrent selection since the cultivation of major cereals ten thousand years ago, had shaped the genome of plants and reduced their allelic diversity among populations of cultivated varieties of crops relative to their wild ancestors (free of man-dependent selection pressures). The domestication genes selected for this study are:
(a) Shuttering gene sh4 (rice)
(b) Teosinte branched 1 gene tb1 (maize)
Because seed shattering in wild relatives of rice results in greatly reduced grain yield, the loss of seed shattering habit is considered one of the most important events in rice domestication.
Mutations at the Teosinte branched 1 (tb1) regulatory gene had an important role during the morphological evolution of maize from its wild ancestor, teosinte.
How will the story be presented?
The story will be communicated to a general audience in the form of an interactive art installation by which participants access plant science data in audible format (sound composition) by interacting with a tangible element (physical interface enclosing physical computing circuitry).
Defining user interaction
Users will be given the ability to control the pace of the story, choosing which sound composition they want to hear first (wild or domesticated from rice or maize, respectively).
The context and style of the installation will follow the format of museum-like exhibit.
Establishing collaborations outside the ITP community
This project provided me with the wonderful opportunity to establish new collaborations at NYU (my new institution) and Rutgers University (my previous institution) by reaching out to plant scientists working on research projects related to rice and maize domestication. With the help of Dr. Michael Purugganan and his team, I was able to obtain seeds from wild and domesticated rice, whereas Dr. Andrea Gallavotti kindly provided me with seeds of wild and domesticated maize, respectively.
Dr. Michael Purugganan:
Dr. Andrea Gallavotti:
Both scientists have expressed their interest (and willigness) to include this project on their outreach sections of their National Science Foundation (NSF) grant proposals, indicating the importance of bridging art and science in real life and outside the classroom.
Most importantly, this work allows me to continue on the exploration and implementation of GAGAISMO, a new artistic form of expression that I've invented at the intersection of genomics, creative coding, and abstract art. I am highly interesting in the conceptual development of GAGAISMO (its discourse and theoretical premises) as well as its artistic implementation and practical implications in the world of art, science and emerging media.
Figure 1. Left photograph: seeds of wild rice (Oryza barthii) and domesticated rice (Oryza sativa), respectively. Right photograph: seeds of Teosinte (wild ancestor of maize) and domesticated maize (Zea mays), respectively.
Figure 2. In dialogue with NYU Prof. Michael Purugganan at his office and members of his research team at greenhouse facilities. Shown in the photograph is laboratory manager Katherine Dorph displaying wild and domesticated rice inflorescences.
Figure 3. In dialogue with Prof. Andrea Gallavotti at his office in Rutgers University. He appears in the picture searching for the code of his Teosinte and maize stock of seeds.
System diagram of installation (beta version)
The components of the installation include a wooden box enclosing the physical computing circuitry and an acrylic plate with switch and buttons as the interface in which users will interact to access the audio composition. Rice and maize seeds will be place and displayed within the box as well. Arduino and board inside box will connect to a multimedia computer that will render the audio and graphics. Users will hear sound compositions by wearing headphones as it provides a more intimate experience and allows for hearing subtle sounds. Optional components could include a projector and screen instead of using the screen of the laptop itself.
Figure 4. Sketch showing system diagram of proposed installation
Prototype of physical interface (version 1)
At the time I started prototyping different version of the interface, I had access to rice seeds only. From six possible designs, I decided to focus on an interface containing a single switch, two led lights and surface to place seeds on top of 'capacitive touch' and/or copper tape.
Figure 5. Sketches depicting different prototypes of physical interface at the time I only had access to rice seeds
The intended idea was that user action triggered sound composition as result of touching the capacitive sensor when grabbing seeds for closer inspection. Although interesting for further exploration, I change the concept idea when I got access to maize seeds in order to fit them into a single physical interface within the given box. I kept the original design guidelines and added push buttons to substitute the capacitive touch surface. As I acted the user interaction, I noticed that my fingers not always touch the surface of the box when grabbing or picking up the seeds, action that would not trigger the sound composition. In addition, when giving the box with the interface prototype to my wife, it wasn't clear to her that she had to touch the seeds in order to trigger a sound composition. For these reasons, I propose the following interface update:
Figure 6. Updated design draft for interface containing two led lights, dedicated space for placing seeds, four push buttons and a switch. Labels indicating rice, maize, wild and domesticated will be included.
System behavior
In this interface, user interaction would be defined by moving the switch and choosing which sound composition should be heard first (rice or maize). At this time, light turned on would cue the user of his/her action taken. Once the light is turned on, push buttons are activated and when one of them is pressed (wild or domesticated) the sound composition is triggered. Once the sound composition finishes playing, the corresponding light is turned off and the system goes to its original state.
The problem with this approach is that the switch will stay in the position left by the user and the light has already gone off when the music finished.
On possible solution is to use a dual throw switch instead of a single throw switch.
Figure 7. Proposed system behavior to guide user actions in allowing them to control the pace of the story. User actions start at the top and proceed downward through the diagram
Results of user testing
I had the opportunity to user test the implementation of the current project with at least five classmates. All of users were 'satisfied' with the notion of sound as the sensory form of the story, with more than half of them suggesting that I should not include accompanying visuals as it would distract the perception of sound. Form those in favor of images, they suggested I could visualize either the sound itself or the DNA sequence of the genes in question.
A single user suggested I provide participants with a pack of seeds for them to bring home and remember the installation. I found this idea very interesting and I definitely will follow through when bringing the installation to a public space.
During demonstration time, I asked users the following questions:
#1- Was the story clear?
#2- Was the story interesting?
#3- Was the interface clear?
#4- Did you learn anything new about plant domestication?
#5- Did you like the sound composition that you heard?
#6- Would you like to touch the seeds?
All users expressed the way I conveyed the story wasn't clear enough to them. Aside from the science involved in communicating reduction in genome diversity as consequence of plant domestication, they could not grasp the piece as traditional art installation and/or documentary installation. Answers to this first question was the most valuable insight I received from user testing and indicated I should work harder in communicating the story clearer and more engaging.
For most of the other questions, four out of five users provided positive responses, with the most resounding responses pertaining to liking the sound composition and the idea of touching the seeds.
Regarding the outline proposed for the physical interface, most users questioned the existence of the switch and suggested to remove it so that users could go directly into pressing the buttons to access the sound corresponding each plant.
Gene sonification: auditory rendering of genome diversity
Sonification of rice sh4 gene comprised DNA sequences from 17 different individuals for Oryza bhartii (wild rice) and 20 different individuals for Oryza sativa (domesticated rice) as suggested by Jae Young Choi from the Purugganan lab. The data provided was downloaded from two studies previously published and referenced in the following two publications:
By applying a sonification approach similar to the one previously described by Calvino (2017) in
I was able to generate a sound composition for the entire alignment length of 1,185 base pairs for each of population (wild and domesticated sh4 sequences, respectively). I proceded then to render the audio on a multimedia computer and recorded about 10 minutes of sound using the Zoom H4n pro audio recorder, placed resulting audio files on Adobe Audition and selected an audio segment of less than 2 minutes that displayed remarkable differences from Oryza barthii (wild rice) compared to Oryza sativa (domesticated rice).
Audio 1. Allelic variance at sh4 gene sequence was sonified and an audio file corresponding to the segment displaying drastic differences was recorded. The first 9 sequences for each species were used to address differences in sound diversity between wild and domesticated rice.
In the audio segment highlighted, there is a moment in which the sound is more discrete, punctuated, and less diverse in domesticated rice as opposed to wild rice. This audio segment will be used in the proposed installation as example of reduction in genome diversity as result of domestication.
When the sound compositions derived from sonification of sh4 are visualized in a graph, the difference in sound pattern is also evident.
Figure 8. Visualization of sh4 sonification by drawing amplitude levels for each sound composition relative to the audio files presented in Audio 1. Amplitude levels at any given point across the audio composition were drawn as colored dots (light orange for wild rice, and pink for domesticated rice) starting from the left (beginning of audio) and ending on the right (end of audio). Amplitude level is graphed along the y-axis with higher values corresponding to dots drawn further down the graph.
A similar approach will be pursued for tb1 gene sequences from teosinte and cultivated maize populations when respective data is available.
Building the interface and the enclosure
The enclosure and interface is composed of the following materials:
# wooden box painted black with acrylic paint
# acrylic sheet (1.5 mm thickness) of white opaque color
# large 10 mm diffused LEDs (x4): two LEDs in yellow color and two LEDs in blue color
# 30 mm arcade button SPST (x4): two in yellow color and two in blue color
Figure 9. Materials to build enclosure and interface acquired from Tinkersphere Corp. and Canal Plastics Center Inc.
I proceded then to create the Adobe Illustrator file containing the design of the interface as input for the laser cutter.
Figure 10. Still image of Adobe Illustrator file containing the design for the physical interface of the installation. Red stroke signals laser cut and black stroke signal laser etching, respectively
After three attempts, I came out with the desired interface. Because the acrylic was 1.5 mm in thickness, even though I tried to edge the plate for text and seed containers, it ended cutting through the acrylic. This was not so bad as I can paste a black paper on beneath to contain the seeds and provide background color to the text.
Figure 11. First tangible version of the physical interface for the installation
Prototyping of electronic circuit
The purpose of the electronic circuit is to instigate a sound composition when the corresponding button is pressed, with the assigned LED light turning on to indicate the user his/her action has been initiated. As proof-of-concept, I created the circuit for a single push button and LED light, instigating the sound composition on a multimedia computer (my laptop). Arduino code reads digital input at Pin2 to communicate the state of the button to a Processing sketch via Serial communication. The processing sketch plays the audio file when the button is pressed.
Video 1. Short video demonstrating proof-of-concept for electronic circuit. When button is pressed, the LED light is turned on and the audio file on the multimedia computers plays.
Figure 12. Sketch and images showing proof-of-concept circuitry for push button corresponding wild rice that triggers audio composition on multimedia computer and turns LED light on breadboard
The next step was to solder the connections for both buttons and LEDs and place them into the enclosure and test the system. As the following video show, soldering worked well and circuity worked when placed on the enclosure.
Video 2. Soldering of circuit components prior to complete placement into enclosure
Video 3. Test of enclosure with working circuit
Proposed circuit diagram
Circuit building and uploading of code to Arduino board
List of materials and project cost
Enclosure -
Wood box ------------------------- $
Acrylic paint -------------------- $
Paint brush ---------------------- $
Acrylic sheet -------------------- $
Foam board ----------------------- $
Display board -------------------- $
Physical Circuitry -
Arduino Uno ---------------------- $
Breadboard ----------------------- $
Push buttons --------------------- $
Switch --------------------------- $
Cables --------------------------- $
Other hardware -
Multimedia computer -------------- $
USB cable ------------------------ $
HDMI cable ----------------------- $
Projector ------------------------ $
Screen --------------------------- $
Software -
Arduino software ----------------- $
Processing ----------------------- $
Biological material -
Rice seeds ----------------------- $
Maize seeds ---------------------- $
Stationary -
Paper ---------------------------- $
Pen & markers -------------------- $
Scissors ------------------------- $
Transportation -
Travel expenses ------------------ $
Honorary -
Hours worked --------------------- $
Total project cost --------------- $