This essay is the result of my work as artist-in-residence at the 'Laboratorio de Lenguajes Computarizados, Facultad de Bellas Artes, Universidad de la República, Montevideo, Uruguay' and was supervised by Marcos Umpiérrez and Daniel Argente
1 Background
Tango is a musical genre and social dance in which its music and lyrics is characterized by nostalgia. It has its origins at the suburbs of Buenos Aires and Montevideo at the beginnings of the 20th century. Danced today around the world, the music played at social gatherings called milongas is mainly composed from songs that were created during the years of 1935 to 1959, with this period in time known as guardia vieja. This means that a musical composition and its social context from the past is ‘recreated’ each time a song is played and danced in the milonga. The evocation of the past is an ever-present topic within tango culture. In this work, we addressed the mentioned relationship between present time and the past from a visual and aesthetic point of view in which we integrated elements from photography, abstract art, computer programming, electronic media, and dancing.
The key concept explored in the artwork is the body of the tango dancer as time travel machine. We approached this by tracking the body in motion in real time with an infrared sensor in order to gather positional data. This data is then used to move a pointer in the artwork that ‘scan’ the intensity of black color from each pixel that composes tango photographs from the past. As the dancer scan different pixels from black and white photographs, this data is used to color and modify an abstract artwork created in the present that is projected on the wall in real time.
The creation of a dedicated artwork that is projected on the wall and modified in real time by the dancer’s body motion in response to a tango song adds has as objective to bring a new dimension to tango in its social context, one that includes a visual element in artistic form. With this in mind, we surveyed dancers across three different milongas from Montevideo city measure the response of dancers regarding the addition of a visual element powered by technology placed into the traditional setting of the milonga.
As the artwork created for this project, it is based on the geometric abstract style of Martin Calvino. It was important to keep the artist’s identity and style in the piece as the dancer’s keeps his style of body movement. Both artistic expression being connected and integrated through technology.
We envision this project can serve as starting point to integrate art and technology within the context of tango social dancing.
2 Materials & Methods
Abstract artwork was created by Martin Calvino and coded using Processing programming language version 3.2.1 for Mac Os X. The artwork’s dimensions are 1024 x 768 pixels and correspond to the required size of the projector used.
The code that composes the artwork reacts to input data from a Microsoft Xbox 360 Kinect sensor (model 1414) through the Processing library oscP5 that provides the code for the protocol Open Sound Control for transmission of data over a network.
Processing code required to gather and visualize image depth data from the Kinect sensor was part of the Open Kinect library developed by Daniel Shiffman and Thomas Sanchez.
The Kinect is composed of an InfraRed (IR) light projector and camera that produces a ‘depth image’ of the scene in front of it. Contrary to conventional images in which each pixel records the color of light that reached the camera from the part of the scene, each pixel of this depth image records the distance of the object in that part of the scene from the Kinect. Depth images have the appearance of distorted black and white pictures because the color of each part of the image indicates how far it is, with brightest parts of the image being the closest to the camera.
For this project we used a Processing program that analyzed the brightness of each pixel from the depth image in order to identify the distance of every object, in this particular case the body of the dancer in motion, in front of the Kinect. Taking advantage of the Open Kinect Processing’s library, we used the code example Average Point Tracking (ATP). Motion data from APT code is sent via oscP5 protocol to the code that creates the responsive artwork.
oscP5 libraries and documentation used in this project were based on Andreas Schlegel and downloaded from his website at http://www.sojamo.de/oscP5
Dancers from three different milongas across Montevideo city were surveyed to test the reception of the idea integrating art and technology to tango in its social context. The milongas chose were Joven Tango (on weekly Friday nights), Revoltijo Tanguero (on monthly Friday nights) and Che Madame (on weekly Thursday nights). Fifty dancers were surveyed in total.
The tango songs and dance choreography was selected and prepared by Natalia Mazza.
Black and white tango photographs of public use were downloaded from the Internet.
3 Results
3.1 Strategy used to implement the artwork
Two Processing sketches were created that communicate over oscP5 protocol in a local host network (Figure 1). The first sketch contains the code that receives and process dancer’s motion data from the Kinect sensor, sending this data to a second Processing sketch containing the artwork that reacts to the data being received. The artwork produced is then projected on the wall in real time.
Figure 1. Diagram illustrating the overall implementation of the artwork involving the use of Microsoft’s Kinect sensor, code programmed in Processing and projection of artwork on the wall that is responsive to dancers movement.
3.2 Survey of tango dancers in Montevideo City
In order to have a hint about the thoughts and feelings of tango dancers regarding this project, we surveyed fifty of them across three different milongas of Montevideo city. The survey consisted in asking dancers to read the following text:
“Estoy trabajando en un proyecto que incluye TANGO, ARTE Y TECNOLOGÍA. Sensores registran el movimiento de bailarines en la pista, generando datos que son usados para animar ARTE DIGITAL que es proyectado en la pared de una Milonga. Agregando una dimensión visual y artística al tango en su contexto social.”
Then they were asked to answer by marking a cross (“X”) in a graph with four quadrants when answering the following two questions:
1- Considera que la idea es original? 2- Considera que la idea es valiosa?
Each of the four quadrants on the graph referred to the idea as (a) Poco Valiosa : Muy Original, (b) Muy Valiosa : Muy Original, (c) Poco Valiosa : Poco Original, (c) Muy Valiosa : Poco Original.
In this context ‘valuable’ is referring to the possibility of integrating of art and technology in a milonga could indeed add cultural value of it.
Results from the survey are shown in Figure 2 and indicated that 68% of the surveyed dancers considering the idea of integrating art with technology in the context of tango social dance as original and as valuable. Interestingly, 24% of dancers surveyed considered that the idea did not add any cultural value to tango in social context, showing the prevalent feeling that tradition shouldn’t be changed.
Figure 2. Graphic representation of survey results conducted in three different milongas across Montevideo city during the month of August of 2016. Fifty dancers were survey in total.
3.3 Code used to process motion data from Microsoft’s Kinect sensor
We took advantage of the code contained in the Average Tracking Point example present in the Processing Open Kinect library for analysis of image depth data coming from the Kinect. The basic concept being implemented is based on tracking a group of points closest to the Kinect and finding the average closest point, using the average raw depth data to produce visible output within a dedicated artwork created for this project.
The core of the implementation is the code that relates to the computer storing depth values from each pixel in a singular one-dimensional array:
The depth value for each pixel is accessed and evaluated in a nested loop:
The raw depth values from the Kinect v1 model 1414 that are stored in the array are between 0 and 2048, while kinect.width equals 640 pixels and kinect.height equals 480 pixels.
The way to look at the depth values for each index in the array is performed with the relational expression of
We then created variables called ‘dancerX’ and ‘dancerY’ that stored the positions of raw depth values along the x and y values on the plane, drawing a colored circle on the corresponding coordinates to track body motion (Figure 3)
Figure 3. This figure shows raw depth values that are within a given distance threshold from the Kinect (colored in red) with the green circle displaying the location along the x and y axis of the average point from all the red colored pixels. The upper body of the dancer is shown.
3.4 Open Sound Control (OSC) protocol to send and receive motion data via a local network
OSC is a protocol for communication among computers (or among two programs within the same computer) over a network. OSC was initialized within the Processing sketch that contained the code to receive, analyze and send motion data by including the following expressions
where Sketch1 refers to the Processing code for Average Point Tracking from the example contained within the Open Kinect Library, and Sketch2 refers to the custom-made Processing code for the artwork that responds to motion data sent via OSC from Sketch1.
Motion data from Sketch1 is sent to Sketch2 by declaring and OscMessage object
Similarly, we set up the OSC protocol for Sketch 2 in the following manner
To receive data from Sketch1 in Sketch2 containing responsive artwork a function was defined and named oscEvent. This function runs every time Sketch2 receives data from Sketch1
3.5 Description of the artwork and the code that supports it
The artwork explores the concept of the dancer’s body as time travel machine as means to visually represent in the form of abstract art the dichotomy of tango’s social dance. This dichotomy is present every time we dance, when we are transported back to the time the music and its lyrics were created. While the dancer creates her movement in response to the beat and melody of the tango music, her body is used to move a pointer (represented as transparent yellow circle) across three black and white photographs from the years that tango was at its maximum expression. These photographs are placed on top of the geometric abstract work for the viewer to observe the trajectory of the yellow circle as it explores different sections of the photographs according to the dancer’s moves (Figure 4).
Figure 4. This picture shows a frame saved from the artwork named ‘Tango_Abstracto’ with size of 1024 x 768 pixels. Shown in the upper section of the canvas is the yellow pointer on top of the black and white photograph. The dimension of the artwork is set by the specification of the projector used.
The pointer directly responds to motion data sent from Sketch1 via OSC protocol providing the artwork with an interactive feature. The novelty created on the pointer is that it captures the grayscale color for the particular pixel on the photograph where it is located. The grayscale color from the photograph is then mapped to RGB values used in coloring several elements from the artwork beneath the photographs. In this sense, the body is re-interpreted as an instrument to sample the colors of the past in order to paint the present. This is expressed in code with the following expressions
The relentless pass of time from the beginning of tango until today is represented by the sequences of white lines slowly moving from the left of canvas towards the right. Interestingly, when the lines reach the right side border, the sequence of movement starts again from the left, alluding to time repeating itself in each occasion we honor tradition and culture by dancing a tango song.
By saving each frame of the artwork while the dancer is moving, the sequence of colors patterns in the artwork transform themselves in the recording of past movements, creating a series of artworks influenced by the history of motion while dancing, a new form of notation implicit in the sequence of artworks put together (Figure 5).
Figure 5. Frames saved as images create a sequence of artworks that represents the recorded history of the dancer’s movement during the song.
4 Discussion
In this work we explored the implementation of a technological and aesthetic dimension to tango dance in its social context. We created an experimental setup in which we took advantage of the Microsoft’s Kinect sensor capabilities to track the body motion of dancers and used the associated positional information as data to provide interactivity to a dedicated abstract artwork to be projected on the wall in real time. We considered the experiment as proof-of-concept prior to a subsequent implementation within a real milonga and thus we surveyed the opinion of fifty dancers across three different milongas. Dancers were asked to evaluate the idea regarding its originality and added cultural value to the tango scene. We found that 68% of dancers considered the experiment concept to be original as well as valuable, whereas 24% of dancers considered the concept as original but not culturally valuable, revealing a certain degree of rejection towards the integration of art and technology into the social context of tango dancing. Our work revealed at a small scale the complexity of implementing projects that span across several fields of human activities such as dance and culture, technology and art.
On the technical side, this project made use of Processing programming language and available Open Kinect library as starting point to collect raw image depth data from the Kinect sensor, sending the raw data to a second Processing program over Open Sound Control protocol. The implementation of OSC showed the utility of this system in relaying positional data from a piece of hardware that can be translated into a visual output.
The visual artwork created for this project combined two different mediums: photography and digital art. Pictures taken in the past provided us with a glimpse of tango culture and at the same time with the raw material to ‘scan’ grayscale colors from a different period of our history, colors used to paint the artwork in the present time. Inherent to the artwork created here is its interactive nature, providing a channel for a dialogue that transcends the human-machine-software divide. The style of the piece is based on the artistic style of abstraction geometry being developed by Martin Calvino, and thus it is respectful of his artistic identity.
The current project serves as an initial example for others to explore into the integration of art, technology and tango, stimulating the cross-pollination of artistic and cultural currents constantly evolving in Montevideo city.
5 Acknowledgements
We thank Pablo Sedraschi for his help and guidance with implementation of OSC protocol. We are very grateful to the Faculty of Bellas Artes for providing the necessary space and resources to conduct this work.
6 Contributions
Martin Calvino contributed with the idea and conducted most of the work for the project. Marcos Umpierrez provided support and coordinated the implementation of the work. Natalia Mazza contributed with tango dance part of the project. Daniel Argente supervised the project, provided valuable guidance and bibliographic material and allowed the participant to use his laboratory installations to conduct the work.
7 References
Greg Borenstein (2012): Making things see. Make Books; First Edition, O’Reilly Media;
Daniel Shiffman: Getting started with Kinect and Processing. http://shiffman.net/p5/kinect/ Accessed on September 2016
Andreas Schlegel: An implementation of the OSC protocol for Processing. http://www.sojamo.de/libraries/oscP5/#examples Accessed on August through September 2016
Keywords: Martin Calvino; computational abstract art; motion capture; Kinect; image processing; tango; milonga; Montevideo; geometric expressionism