In an attempt to translate the visual world into audio I wondered what if the shapes of objects could be contours of a sound wave. As an experiment I took the contour of an oak leaf and turned it into a wave form.

I first found an outline of an oak leaf online then cut it in two:

Next, using computer vision (openCV in Cinder), I analyzed the image for feature points:

Using javascript I converted these points from a csv into a table that Max/MSP could use as a waveform. Since some of the points are over/under each other (and with a waveform you can’t have more then one value happening at any given point in time) the waveform in max has some interesting striations:

With an oscilloscope the wave coming from the synthesizer looks like:

It sounds real buzzy:

But in layers and with some filtering it can have a nice effect:

I started looking at work by Ryoichi Kurokawa, an artist whose work I really like, and whom I feel mixes audio and visual worlds really well. His work flow is to story board his entire concept (sometimes just in his head), work out a rough video, make audio for that video, then edit the video and audio together precisely. This process can take months.

I made a quick animation with Sol Lewitt’s wall drawing patterns simulating Kurokawa’s work flow, but with very little (read none) story board or revision.

Two weekends ago I went to Dia:Beacon with my wife (a dancer) and a bunch of other dancers. Dia is a wonderful modern art museum just north of NYC in Beacon. The dancers had a performance planned in the museum, the intention being to “bend public space, but not break it.” This meant that they would interact with each other, the art, the space, etc., but if it became noticeable to general museum goers the dance would fade away, disappear, break apart. If a random museum goer stood still for 10 minutes they would probably notice something out of the ordinary was going on, but otherwise the movements would be too subtle to stand out. I participated in this some, but not being a dancer I didn’t dive in fully, though it really did provide a new way of experiencing a museum. This opened me up to experiencing the museum in other ways too, outside of the dancers’ parameters.

This manifested itself at the Sol Lewitt exhibit. The exhibit was comprised of many of his wall drawings, where he writes out directions, then teams of other people draw the actual drawings later on. When I first saw the pieces in the exhibit I was immediately struck by how computational they looked. It seemed to me that this guy was taking the world of generative computer art and recreating it by hand. This made sense since just like you give a computer step by step instructions of what to do, he was giving step by step instructions of what to draw.

One of the drawings consisted of 4×4 grids of lines; vertical, horizontal, diagonal up and diagonal down. These patterns would alternate from square to square based on a numbering pattern:

Whole walls were filled up with these patterns:

Since I was in this mental space of experiencing things in new ways I wondered what would these drawings sound like if each of these numbers was a scale step of a major scale. I started singing the drawings in my head and was pleasantly surprised to find that many of them were actually nice melodies:

I started wandering around the museum and whatever direction (straight, sideways, diagonal) I was going I would listen to my movements in my head as the corresponding pitches in Lewitt’s directions. I then started experiencing the other art work in the museum as music, and then the architecture. I even started seeing the nature outside and wondering what the trees would sound like, what about the infinite changes in angle of branches? It was quite wonderful. I decided then that this was what I wanted to research next… What would the world I can see sound like, if it could be audible?