Piano Projection Mapping using Max/Msp & Resolume Arena
Part 1 - Ideation
When it came to ideating the project there were several avenues I looked at for visual inspiration. Before even touching Max/Msp I decided to go down these avenues and mock up how I wanted them to look in illustrator to get some idea of how in my head how the opening stages of the project might look.
First Ideation - Star based visualiser
The inspiration behind this idea stems from the movie "Doctor Strange and the Multiverse of Madness". The aspect that inspired this idea was the way the character "America Chavez" uses her multiversal transport powers and how they are visualised in the movie with a tunnel-like portal in the shape of a star. The way I would translate this to interact with a Midi Keyboard is by having the pitch correspond to the colour of the star. I would then have the size of the star dependent on the velocity of the key pressed. The modulation or pitch bend could affect the rotation of the star when drawn also.
Second Ideation - Traditional Audio Visualiser
This idea was one of the original inspirations that I had when first undertaking this project. It is based on traditional audio visualisers from music equipment or from different media players from the early 2000s. This would work exactly like a typical Visualiser with the pitch of a key affecting the highs and lows on the graph and velocity dictating how quickly they change. The Modulation/Pitch bend wheels could be used to change the colour of the piece to make it more visually striking.
Third Ideation - The Beach
This idea was inspired by the piece "Dearly Beloved" from the game Kingdom Hearts 2 composed by Yoko Shimomura. Dearly Beloved is the game's main menu theme and is a calming yet sad piano track with motifs of hope and reflection. These motifs are also complemented by the sound of waves in the background giving the feeling of standing on the beach overlooking the sea. The way I could translate this into a mapped piece is by having chords or key combinations control which ocean-based items wash up on the shore and have the mod wheel act like a wiper and control the tide coming in and going out.
Ideation 4 - Moon over a Hill
For this piece there was no explicit inspiration for it, it was more of an idea that came to me whilst designing the others. The pieces would feature a moon traveling across a landscape based on the duration of the total piece. The stars would fade in and out based on the velocity of the keys played and the hill height would change based on the pitch of the key played. The piece has a ton of scope for experimentation with different midi inputs like synthesizers as you could create alien-esque landscapes and change the sound accordingly.
Ideation 5 - Hyperdrive and Stars
This piece was inspired by Star Wars; specifically scenes where any character engages their ship into hyperdrive. Mapping this piece would lend itself well to a two-state system where you can have both pitch and velocity mapped to two different aspects at the same time. you could have the velocity tied to both the stars and the speed lines and have the pitch control the amount of both of those aspects.
Ideation 6 - Hexagon Colour Wave
The idea for this piece was a lot more simple than the others with colour waving through the hexagons. The way I could map this one is by having the speed of the colour wave dependent on the velocity of the key pressed and the choice of colour waving through be dependent on the pitch of a key. I could also use the pitch/modulation wheel to affect the colour by maybe turning it black and white or by changing the spectrum of colour that gets waved through. The wheels could also change how the colour ripples through by changing to ripples or droplets or even changing the direction of the colour.
Ideation 7 - Tetris
As the name implies this was inspired by the 1984 game of the same name with sections of the keyboard allowing these shapes to be created and fall. The way I could map this further is by having it so only one shape at a time could appear and making the velocity of the key pressed to rotate the shape. Though this is a far more gamified idea that may not work it definitely has some aspects that could be expanded upon and explored.
Part 2 - After Effects Mock ups
Before work on the patch began I decided to create some simple after effects animations to visualise how mapping graphics on to the piano would look. I made three animations themed around pieces of Video Game and Film media that I enjoy and they are as follows.
Animation One - Simple Starfield
Inspired by Sci-fi films where spaceships travel through outer space and go to hyperdrive this animation shows a camera moving through a simulated starfield created with a simple array of 3D balls and two layers of Gaussian Blur.
Animation Two - Rolling tide
Inspired by the Main menu of the game "Kingdom Hearts 2" which features a static image of the games protagonist "Sora" standing in water as the track "Dearly Beloved" plays in the background featuring the sound of the ocean to add to the aesthetic. The animation features a simple beach as the tide roles in complete with seafoam leaving the sand wet where the water has touched.
Animation Three - Lifestream
This animation was inspired by the opening of the movie Final Fantasy VII Advent Children where one of the characters is giving a recap of the original game that the movie follows on from. The scene features green energy trails that I have pulled inspiration from to create my own animation featuring smoke and green wisps.
Part 3 Iteration of patches
During my journey through this project I created 6 individual patches that were all visually different but also interacted with both live and recorded music. They also demonstrate different aspects of MAX 8's Jitter system.
First Patch - A simple growing circle based on the volume of the audio input which the user could use three individual sliders that would alter the RGB values of the circle.
Second Patch - This Patch is named "Cymatics" visually simulates how sand vibrates when around sound to form patterns.
Third Patch - This patch uses frequency analysis to then translate the audio into a visual. In this example, it maps the audio analysis to a sphere to create what looks like a meteor traveling through space.
Fourth Patch - This patch uses Polar Rose mathematics to create visuals with a simple tweak to two core values to allow users to generate visuals.
Fifth Patch - In this patch dubbed "Reactive Dots" music is plugged into the patch which drives the way the dots move and frequently they move based on how loud the volume gets. The volume threshold can be tweaked to make it react more or less often. The feedback and force can also be altered to change how the dots react to see them have more/less form or erupt from the centre more quickly.
Sixth Patch - The last patch that I created was labelled "Tapestry" and is a simple patch that creates visual wavelengths.
Part 4 - Final patch
After weeks of iterating and meetings discussing patches myself and Max agreed upon the "Reactive Dots" patch and after weeks of making visual improvements, tweaking settings, and implementing the Midi Keyboard into the patch so that the keyboard drives the visuals. The patch now looks like this:
The Visuals now look like this:
Altogether the piece looks like this:
