Whoever said music is a purely an auditory experience? Whoever it was, it surely wasn’t me, as I would have told you that as much as music can overtake your auditory awareness, it can also envelop your emotions, and project on your mind’s eye. That is to say, one can see music.

I often find myself building compositions on the black surface of my eyelids, as I close my eyes and let myself experience music. A few years ago I learned about a neurological phenomenon called synesthesia, in which stimulation of one sensory or cognitive pathway leads to automatic, involuntary experiences in a second sensory or cognitive pathway. It was at that moment that I realized this thing I do – while certainly not full-blown synesthesia, or even quarter-blown – had a name. After learning this name, my readings became somewhat punctuated with a text here and there on the subject. Now, I’m doing a project based off this very phenomenon that always seems to find its way back to me somehow time after time.

I never managed to find the perfect fit academically, even now. This is very much a result of my interests lying somewhere between computer science and art and design. When I come to this school I considered myself a graphic designer, but now I know I’m really an interdisciplinary artist. Because of this it really comes as no surprise that my Integrative Project would be just as well suited as a computer science project – or even a psychology project – as it would an art project. Just as much as it’s not a surprise that my project hugs the fence of several areas of study, it’s equally not surprising that I chose to explore this subject of synesthesia through art by means of technology. Through the next few pages, I hope to explain my project as thoroughly as I can, while touching on my process, as well as the context in which I worked on it.

The goal of this project was three-fold: blur the line between digital art and fine art, to blur the line between computer engineering and art, and to explore the connection between the very temporal medium of music and static and moving visual representations of music. In the end I aimed to illustrate these connections by showing how I can sometimes see music when I close my eyes, all by way of a series of prints and videos that show these different modes of my perception.

The actual final installation of my project consists of five songs that one can listen to through headphones attached to the walls of the gallery. The music is played in sync with the moving visualization that is associated with that song which plays on a TV set into the wall. Above each TV hangs a print taken from the video as a single frame or a completed drawing that the visualization creates when it runs. This way a viewer can experience the song in real time, while also experiencing the moving visualization, which is the halfway point between the static visual output, and the auditory input. This middle stage of evolution from initial input to final print is important, since it can be appreciated as the synthesis between image and sound, static and temporal, and is really the linchpin in tying the input and output together. Off to the side of the wall of prints and videos sits a computer kiosk which contains a rendering of each song as played through each unique visualizer, as well as an application which allows the viewer, now the user, to select two videos and compare them side by side. These videos can be different songs using the same visualizer, or the same song in different visualizers. The purpose of the kiosk is to allow the user to really explore the effect the input has on the end visual result, or see how a different set of instructions can turn the same input into something very visually different. This is important because the viewer can get a better idea as to the nature of the project, and so they can see how each print is unique to the song despite the same set or graphic rules being applied to every other song.

This project started off very ambitiously. In the beginning I had the idea of creating eight separate fine art prints – in the same vein I have described above – and two bronze pieces, all of which was supposed to draw their content from all manner of data sets, from receipts to census data. The reason I had decided to switch to using solely music is because it was simply more fun to use. Reviewers mentioned that when I showed early concepts using music versus concepts using other data, I smiled much more and talked about what I was doing with much more gusto.

The creation of each individual visualizer is a very unscientific process, at least initially. It really all begins by sketching out very rough and quick representations of how I imagine music could be displayed visually. At this point my preliminary drawings are completely disconnected from any sort of meaningful visualization of music. The next step in the process is to just listen to music; any music at all. I find something I like and then pick one of sketches I drew up earlier that best fits the song to some degree. Then, I go back and start revising the sketch so that the visualization starts to take cues from the song in some aspect. It’s at this point I know how I want the visualizer to look, how I want it to react with the song. Meanwhile I continue to listen to the music, and when I do so I can almost “see” how this visualizer’s code will flow. This I came to think of as some form of synesthesia. I start writing pseudo-code down in a text document basically outlining what I need the program to eventually do. Then I take each bullet point in the outline and expand on that, writing something that looks more and more like actual programming. Eventually I reach a point where I start testing the code and seeing what compiles and what throws big errors, and somewhere down the line, past all the errors and past all the testing, an actual working visualizer application is born. The real joy and excitement in this stage of the process is in conquering the errors and finally seeing on screen what I had be envisioning in my mind.

Once an application is working to my satisfaction I will output a video, which is a simple process of writing out a frame to a video file every time the program loops through and draws a visual to the screen. If the visualizer is one that constantly changes with the music I try to find one specific frame, or a close group of frames, and decide to create my final print from them. If the visualizer is of the kind that builds up upon itself frame by frame without erasing what was previously drawn to the screen, I use the resulting image that is left on screen when the song ends. Otherwise, my final print will be the end result of the program drawing to the digital canvas. When I have what I want, I have a few choices, I can create a screen print, a woodblock print, a linocut, or something else. I choose my medium based on how I think the print would best be achieved. With screen prints I have to create transparencies then expose screens coated in a photo-reactive emulsion. Linocuts require me to cut out my designs into linoleum. With woodblocks I still have to engrave my designs into the block but I can do it by hand or I can utilize the 4-axis computer controlled mill to cut my blocks by having it follow a digital file I feed into it. When I have my blocks, screens, or plates cut or exposed, I can then finally print these illustrations I’ve worked on for so long.

Throughout the project I have been lucky enough to have lots of outside input from friends and professors, as well as a number of individuals who’s work I admire to turn to for inspiration. This year in IP, I have been influenced by a number of contemporary artists, programmers, authors, and others. Some such artist-slash-programmers include such folk as Robert Hodgin, creator of Magnetosphere, a music-reactive grouping of spheres and tendrils coded in the very same software I use to produce my prints. I am also influenced by Eric Natzke, a digital-slash-abstract-slash-programmed painter, who creates mesmerizing, flowing, abstract illustrations using Adobe Flash and what I can imagine is some pretty creative code.

Another artist who has inspired me is Joshua Davis, an artist and design technologist who creates illustrations by utilizing chaos theory in a controlled environment to influence his work. Very critical to the beginning of my project, information design’s granddaddy Edward Tufte and his book Envisioning Information had initially inspired me to venture into visualizing data. And one author that really drove me to think about the psychological aspect of synesthesia and actually visualizing music, as well as code, is Oliver Sacks, a psychologist, neurologist, and professor who recently published a book called Musicophilia, which delves into a number of interesting case studies on individuals with varying forms of synesthesia.

Work from others I’ve seen that has had an impact on my project, from the early stages to completion, includes the breadth of work from the previously described. There are also several communities on the Internet where numerous members have been doing somewhat similar work as I have, such as Processing.org, Vimeo.com, and VisualComplexity.com. In a more historical context my work takes cues from work such as Duchamp’s Nude Descending a Staircase, No. 2 in how it visualizes time statically. Also, Muybridge’s early stroboscopic work, the usage of simple geometric shapes in the work of Russian Suprematists Wassily Kandinsky and Kasmir Malevich, as well as Piet Mondrian’s Broadway Boogie-Woogie.

This project has been very exciting affair for me. I have longed for the opportunity to work on a sustained project of my choice with input from professors for a very long time now. Nearly eight months of work dedicated to one project on a topic that really interests me and isn’t really explored much in the Art & Design curriculum has been an incredible boon to me, as with this class I’ve had the time set aside to do some self-directed exploration – which is how I best learn. It has been nice to use media that I haven’t used to the extent I wish I had in my academic career. I’m very pleased with the exploration that has taken place and the work I’ve created over the course of the academic year.