Interactive Multimedia and Music

Virtual Instrument research


(1)Human Computer Interaction is becoming a major component in computer science related fields allowing humans to communicate with machines in very simpler ways exploring new dimensions of research.

(2)Kinect, the 3D sensing device introduced by Microsoft mainly aiming computer games domain now is used in different scopes.

(3)We use kinect for controlling sound signals producing aesthetic music.


(1)Kinect is a sensor which is capable of capturing depth and color information of the user in front of it using an array of RGB and infrared cameras. Further it is capable of capturing the sound input though an array of microphones.

inter 1

inter 2

(2)Musical Instrument Digital Interface (MIDI)

(3)Technical standard that describes a protocol, digital interface and connectors and allows a wide variety of electronic musical instruments, computers and other related devices to connect and communicate with one another.

inter 3

(4)MIDI controller is hardware or software which generates and transmits MIDI data to MIDI-enabled devices.

(5)Our Virtual Instrument is a novel MIDI controller design based on Kinect!


(1)Visual C++: Programming Language

(2)OpenNI: Kinect Driver and API functions

(3)OpenCV: Visual Interaction display

(4)MIDI: Music signal to control MIDI instrument

(5)Cubase and VST instrument: MIDI instrument (Audio Library)

inter 9

(6)Once the depth information is captured using Kinect, user skeleton can be obtained with the available functions of OpenNI with 24 joints.

inter 10

 Midi Programming and Audio

(1)For the MIDI signal, we use the RtMidi API from Gary P. Scavone from McGill .Sending MIDI signals to audio library for controlling the note key, duration and velocity.

(2)VST instrument from Steinberg Company is the audio library which is capable of simulating the sound of real instruments.

(3)Using MIDI as the input mechanism, VST instruments output the sounds vividly as instructed.

inter 11

Virtual Drum

(1)Three areas in front of the user is identified as the Kick, Snare, Hi-Hat and Cymbal.

(2)Left Hand ,right hand and right knee are used as the triggers against the above specified regions.

(3)When the coordinate of the triggering point is larger than a specified threshold with respect to the defined regions of the virtual drum sets, program triggers MIDI signals, and then MIDI signals trigger the sound in audio library.

inter 13  inter 14

Virtual Guitar

(1)Same like in Drum, using Kinect and OpenNI, user skeleton is captured first.

(2)Then according to the user skeleton coordinates, we set a Chord selection position in front of the user’s left hand. In the Chord selection position, we have defined six different areas, each representing a chord.

(3)To play the virtual guitar, we also set a virtual guitar string in front of the user’s right hand. When the right hand coordinate is in a special value interval, the program sends MIDI signal to the audio library and then triggers the relevant sound.

inter 15inter 16inter 17

Spider King

(1)The program virtually draws a circle around the user. We divide the circumference into several intervals. Then users’ hands are used to control the key and volume.

(2)When the hand is closer to the circumference, the volume becomes louder. The sound is also from MIDI signal as with drum and guitar, and we connect MIDI signals to different audio libraries.

inter 18 inter 19

Advanced Spider king

inter 20 inter 21

The Virtual Cello

  • Use two sensors to play
  • Totally with 17 scales for the instrument
  • Can use another MIDI signal such as violin or others



  • Get the X and Y axis gyroscope values from the sensor
  • When a gyroscope value is over the threshold, send a message to the cello program



The Virtual Piano (Leap motion)

  • Use OpenGL to design the piano
  • Separate keys into the upper and lower arrays
  • Pre-selection of key using colors to match keys and fingers
  • Use a relative average threshold to trigger sound


  • Use the left most two keys at the same time to trigger chords
  • Use hand gesture (move left or right) to shift to higher or lower keys



Demo Video


Keynote Speech and Plenary Talk in International Conferences

  1. Keynote Speech, “The MINE Virtual Band – Music Performance via Sensors,” The third International Conference on Visual Informatics 2013 (IVIC’13), Malaysia, November 2013

Demo Videos

All the virtual instruments presented here, Guitar, Drum and Spider King and several other virtual instruments are presented in a live performance concert in NCU. –May,9th,2013

—103 CSIE Concert – MINE Studio Virtual Instrument Performance (2/2)

—103 CSIE Concert – MINE Studio Virtual Instrument Performance (1/2)

—102 CSIE Parent-teacher meeting

—102 CSIE Concert – MINE Studio Virtual Instrument Performance

—102 CSIE MINE Lab – MINE Studio Virtual Instrument Demonstration

—101 CSIE MINE Lab – MINE Studio Virtual Instrument Demonstration


%d bloggers like this:
search previous next tag category expand menu location phone mail time cart zoom edit close