Prototype 1
For the first round of user testing, I used an Azure Kinect, ml5.js, p5.js, and tone.js to create a generative piece whose visuals and sound were affected by user interaction.
My goal for this prototype was to a) get a tracking mechanism working that I can use to control data points, b) to draw something using those data points, and c) to create sounds based on (b). The visuals themselves aren’t too sophisticated yet, and the sound sounds like an eerie monotone as opposed to anything resembling music, but they both respond decently to user interaction.
Technology
Azure Kinect
This is mostly used as a high tech camera in this iteration.
It is difficult to import real time data from the Kinect into the browser without advanced C#. I ended up using a Kinect library built by folks at NYU ITP, kinectron.js, that broadcasts data from the Kinect into the browser.
The one downside of this mechanism is that it can only broadcast one feed at a time. I made a choice to use the depth feed, and let ml5.js do the pose tracking.
ml5.js
ml5.js is an image processing library that has a pre-trained neural network that can recognise poses.
It is fairly reliable, even with a webcam.
In this iteration, I used only the nose, both shoulders, and both hips to draw the visuals.
I aim to incorporate more gestures based on node positions in the next iteration.
Testing
Key Observations
I did notice it’s a little hard to move up and down as the points are currently mapped. This was later confirmed by general user testing, as no one thought to crouch down or climb higher.
Many people’s instinctive reactions were to move their arms and/or bodies around. No reaction is yet mapped to these actions, so they didn’t yield much of a result.
The sound isn’t distinctive enough yet that people could easily differentiate. When they did, they spent some time moving side to side to see what happened.
The pose tracking kept glitching. Since it would automatically snap to whoever has most of their body on screen, sometimes it favoured people in the back over people right in fron of the camera.
Feedback
Setup
Marking out or demarcating a physical area where people are supposed to move around would help.
Visuals
Trailing visuals could be added to supplement the ghostly, eerie vibe given off by the synth sounds.
The literal movement is fun. The visuals should be a little obvious.
The random picking of people also adds a fun, competitive element. Could be used to an advantage by creating a situation where people have to pass to each other.
A more simplistic visualisation could involve only a few dots that are all affected by each other and the music.
Sound
Multiple notes could be played with multiple body parts.
Speed of movements would have a huge impact on the sound in this case.
Incorporating wrist or hand movement into the music would simulate a feeling of playing music.
A distinct note could play at regular intervals instead of a continuous sound.
If music only plays at certain positions/places, people would be incentivised to find the music.
A shape could be built as music is constructed.
[Testers featured: Jon, Rui, Ally, Nikita]
