Can you hear what I see?

•

This is a project about Chromesthesia or also called sound-to-color Synesthesia. The main concept is to recreate the sight or vision of a chromesthet while listening to a certain song.

Synesthesia is a neurological phenomenon in which one sense stimulates another one as a direct reaction. The combination of senses has many variations and directions of the connections. That's why I decided to concentrate on a specific combination which is the chromesthesia (when the person listen to a song and sees colors).

Cover of the documentation with the color palette used for the experiment.

After the research phase I started with the experimentation. Why experimentation phase if this is a design project?

Because to express and transmit what a synesthet sees while listening to certain song, you have to understand it first.

In the experiment the synesthets and non-synesthets had a color palette on which they had to assign a color depending when they thought of a color (for non-synesthets) or when they saw a color (for synesthets). After that the synesthets had to draw a layout of what forms they saw and how they were moving while the song was playing.

Results of the experiment with synesthets and non-synesthets

The Application

To find the best way to transmit a scenario that happens in the eyes of another person was a challenge. At the end I decided to use virtual reality as the tool to transmit the information and (of course) to make the project be an experience for the visitor.

I built our room for virtual reality at the university (FH Aachen) 1:1 in Cinema 4D to have the exact same room with the same proportions in the virtual reality project. In that way the visitor/user would be experiencing the same room as a synesthet when the music starts playing.

A surprising experience for the visitors was not only to be able to see how a synesthet does but also to be able to touch the furniture that was present in the room (since it was built 1:1 to the real room). To be able to play with their senses and be able to touch what was really there created a familiar trust to move around the space, what also gave the animation of the colors more meaning of "being real".