With the use machine learning and AR technologies, we created a digital representation of a theremin that participants can play, but can’t actually see. The piece stands as a hinged instance that presents participants to the duality of the real and the unreal at the same time. The theremin, as a 3d virtual object, is at the same time present and absent, as it is “real” by generating sound, but also is “unreal” by being invisible. The dynamic is also reinforced by the installation setting. Even thought the main participant or player can’t see the instrument, there will be a screen set near the installation where other participants will be able to see the theremin, and therefor, interact between each other.
References / Motive
We were very much inspired by Magritte’s work to create this project. We wanted to create an interactive object that could have a tangible effect in the world, but at the same time, reveal the media and by doing so, pointing out the duality of the contemporary world.
Theremin makes sound depending of the distant of the hands from the instrument. The thereminist stands in front of the instrument and moves his/her hands in the proximity of two metal antennas, which act as capacitor plates. The output of a fixed radio frequency oscillator is mixed with that of an oscillator whose frequency is affected by the variable capacitance between the antenna and the thereminist as that person moves her or his hand near it. The difference between the two oscillator frequencies produces a tone in the audio range. The distance from one antenna determines frequency (pitch), and the distance from the other controls amplitude (volume).
So, to replicate the theremin functioning we needed to detect hand movement in reference to the (for now) invisible theremin antennas. We use posenet to detect hands positions according to a frame that we referenced to the distant of the shoulders of the person. Code can be found here.
After having this running in the browser, we used Osc and WebSockets to send the values for distance X (pitch controller) to distance Y (volume controller) to Max for live. Based on what we learned from the paper Physics of the Theremin we scaled the data algorithmically so we could replicate the physical behavior of the variable capacitance of an actual theremin.
And then for the final step for the audio, we send the output from max to ableton where we added harmonics and a low pass filter to the sound.
For creating the augmented reality version of the theremin we used Unity’s Vuforia following this tutorial. The AR scene is composed by an image target and a child object attached to it, the theremin. We build the theremin using a variety of primitive 3d forms.
The scene is set up to find a webcam (declared in the setting of the project) when the project goes to game mode. So when the game is on this camera will look for the image target, and when the target is found, any child of the image parent parent will pop-up as an augmented reality object.
Now the trick is to set up the theremin in the world in the same spot where we are getting the data for the human position. For now we have been using two webcams, but we want to try to have everything working with from the same videofeed.