The Blog

1 Digitizing Facial Movement During Singing : Original Concept

A machine that uses flex sensors to detect movement and change in facial muscles/mechanics during phonation, specifically singing. Sensors can then be reverse engineered for output. Why? Because fascination with the physics of sound, the origin of the phonetic alphabet (Phoenician/Greek war tool later adapted by the Romans), and the mechanics of the voice (much facial movement/recognition research at the moment leans in the direction of expression rather than sound generation).

Found two exceptional pieces on not just the muscles of the face but the muscular and structural mechanics of speech AND two solid journal articles about digitizing facial tracking. After reading the better part of The Mechanics of the Human Voice, and being inspired by the Physiology of Phonation chapter,  we decided to develop a system of sensors to log the muscle position and contraction of singers during different pitches in an attempt to funnel the data into an audio output that translates the sound. For example, is there a particular and common muscle contraction/extension that occurs during high C? We save that, then  while a different non-singing user contracts in the same way the computer recognizes that and plays the corresponding note.

2 Digitizing Facial Movement During Singing : Prototype

We decided to use a three-part apparatus for tracking facial movements during simple singing: a barometric pressure sensor to monitor airflow, electromyography (EMGs) to monitor extrinsic facial muscles and a camera to track facial movement. While this data feeds in we will be manually recording fluctuations at different frequencies. Specially, the aim is to record changes in pressure, muscle contractions and position for each note in one octave—comparing to the face’s resting state. Ideally, data mining will be performed on at least five professional vocalists.

The barometric pressure sensor will be placed in front of the text subject, discretely mounted to a tiny stand. The EMGs will be placed (via medical adhesive) to the extrinsic muscles of the larynx involved in articulation; relevant muscles include the orbicularis oris (lips), geniohyoid (tongue and lower jaw), mylohyoid (mandible), masseter (cheek and jaw) and the hyoglossus (tongue). And finally, facial movement and geometry will be monitored using an external camera.

Luckily we found an Instructables DIY EMG guide.

3 Digitizing Facial Movement During Singing : Sensor Assembly

Pressure Sensor Diagram:

DIY Electromyography Circuit Diagram & EMG Map:

FaceOSC/Syphon/Processing Source Code:

import codeanticode.syphon.*; import oscP5.*;

PGraphics canvas;
 SyphonClient client;
 OscP5 oscP5;
 Face face = new Face();

public void setup() {
   size(640, 480, P3D);
   println(“Available Syphon servers:”);
   println(SyphonClient.listServers());
   client = new SyphonClient(this, “FaceOSC”);
   oscP5 = new OscP5(this, 8338);
 }

public void draw() {
   background(255);
     if(client.available()) {
     canvas = client.getGraphics(canvas);
     image(canvas, 0, 0, width, height);
   }
     print(face.toString());
   }
 }

void oscEvent(OscMessage m) {
   face.parseOSC(m);
 }

+ LINK to the (ORIGINAL) Face class.

4 Digitizing Facial Movement During Singing : Geometry Troubleshooting

We ran tests on the FaceOSC/Syphon/Processing facial geometry detector and were able to mine the following data (seen in the above screen capture):

pose
 scale: 3.8911576
 position: [ 331.74725, 153.13004, 0.0 ]
 orientation: [ 0.107623726, -0.06095604, 0.085640974 ]
gesture
 mouth: 14.871553 4.777506
 eye: 2.649438 2.6013117
 eyebrow: 7.41446 7.520543
 jaw: 24.912415
 nostrils: 5.7812777

Using the on-camera feed to track geometric displacement in the face (relating to muscle movement), only the data in the “gesture” category really applies—categories with two values are recording x and y fluctuation. Fortunately, scale, position and orientation do not bias the gesture readings. Unfortunately, there is a fair bit of natural fluctuation in the readings. We’ve gathered that fluctuations of over 0.5 are actual movement while those under 0.5 are natural oscillations in the camera’s reading.

Digitizing Facial Movement During Singing : Sensor Build

5 Digitizing Facial Movement During Singing : Interface Development

With all that established, it was time to work on building an interface that could help us to track all elements of the sensing system (pressure, electromyography, and geometry), compile these with audio frequency/pitch, display them in an easy to understand manner and then save everything to a data file for reference. We came up with a system of sliders (data is mapped to the tested highest and lowest points of each input so that sliders sitting at the top of the interface are reading at zero fluctuation and sliders which have reached the bottom are extended to their full potential). Actual number values are printed to the console and saved to a local data fie for reference. Geometry (mouth height/width, jaw protrusion) are measured in centimeters (cm), air pressure in pascals (Pa) and muscle fluctuation is to be measure by an unaltered analog reading. Additionally, the interface display and logs the frequency/pitch (using the minim library’s Fast Fourier Transform class and a popular frequency to MIDI formula).

Current monitoring interface and datafile screencap:

Testing Interface Code:

import codeanticode.syphon.*;
import processing.serial.*;
import oscP5.*;
import ddf.minim.*;
import ddf.minim.analysis.*;

PGraphics canvas;
SyphonClient client;
OscP5 oscP5;
Serial port;
Minim minim;
AudioInput in;
FFT fft;
Face face = new Face();

// Analog Inputs
int muscleFluc;
float airPressure;
int[] serialInArray = new int[2];
int serialCount = 0;
boolean firstContact = false;

// Data File
String dataFile = "/Users/mcortese_/Documents/Processing/FaceProcessing2/data/data.txt";

// Frequency Detection
String note;
int n;
float hertz;
float midi;
int noteNumber;
int sampleRate= 44100; 
float [] max= new float [sampleRate/2];
float maximum;
float frequency;

// Interface
PFont font;

public void setup() {
  size(570, 400, P3D);
  println("Available Syphon servers:");
  println(SyphonClient.listServers());
  client = new SyphonClient(this, "FaceOSC");
  oscP5 = new OscP5(this, 8338);
  println(Serial.list());
  String portName = Serial.list()[0];
  port = new Serial(this, portName, 9600);
  font = createFont("GothamHTF-Light-48.vlw", 24);
  minim = new Minim(this);
  minim.debugOn();
  in = minim.getLineIn(Minim.MONO, 4096, sampleRate);
  fft = new FFT(in.left.size(), sampleRate);
}

public void draw() {
  background(255);
  fill(0,100);
  text("MOUTH WIDTH", 25, height/2);
  text("MOUTH HEIGHT", 133, height/2);
  text("JAW", 275, height/2);
  text("MUSCLE FLUC", 357, height/2);
  text("AIR PRESSURE", 460, height/2);
  stroke(160);
  line(15,350,555,350);
  findNote();
  text (frequency-6+" hz   /   NOTE "+ note, 15, 375);
  if (client.available()) {
    canvas = client.getGraphics(canvas);
    image(canvas, 0, 0, width, height);
  }
  if (face.found > 0) {
    print(face.toString());
    println();
    saveData(dataFile, face.toString(), true);
    fill(0);    
    rect(15,map(face.mouthWidth,9,18,5, 345), 100, 10);
    rect(125,map(face.mouthHeight, 1, 10, 5, 345), 100, 10);
    rect(235,map(face.jaw, 20, 29, 5, 345), 100, 10);
    rect(345,muscleFluc, 100, 10);
    rect(455,airPressure, 100, 10);
  }
}

void oscEvent(OscMessage m) {
  face.parseOSC(m);
}

void serialEvent(Serial port) {
  int inByte = port.read();
  if (firstContact == false) {
    if (inByte == 'A') { 
      port.clear();
      firstContact = true;
      port.write('A');
    }
  } 
  else {
    serialInArray[serialCount] = inByte;
    serialCount++;
    if (serialCount > 2 ) {
      muscleFluc = serialInArray[0];
      airPressure = serialInArray[1];
      port.write('A');
      serialCount = 0;
    }
  }
}

void saveData(String fileName, String newData, boolean appendData) {
  BufferedWriter bw = null;
  try {  
    FileWriter fw = new FileWriter(fileName, appendData);
    bw = new BufferedWriter(fw);
    bw.write(newData + " " + System.getProperty("line.separator"));
  } 
  catch (IOException e) {
  } 
  finally {
    if (bw != null) {
      try { 
        bw.close();
      } 
      catch (IOException e) {
      }
    }
  }
}

class Face {
  int found;
  float poseScale;
  float mouthHeight, mouthWidth;
  float jaw;

  Face() {}

  boolean parseOSC(OscMessage m) {
    if(m.checkAddrPattern("/found")) {
        found = m.get(0).intValue();
        return true;
    }
    else if(m.checkAddrPattern("/pose/scale")) {
        poseScale = m.get(0).floatValue();
        return true;
    }
    else if(m.checkAddrPattern("/gesture/mouth/width")) {
        mouthWidth = m.get(0).floatValue();
        return true;
    }
    else if(m.checkAddrPattern("/gesture/mouth/height")) {
        mouthHeight = m.get(0).floatValue();
        return true;
    }
    else if(m.checkAddrPattern("/gesture/jaw")) {
        jaw = m.get(0).floatValue();
        return true;
    }
    return false;
  }

  String toString() {
    return "found: " + found + "\n"
           + "scale: " + poseScale + "\n"
           + "mouth: " + mouthWidth + " " + mouthHeight + "\n"
           + "jaw: " + jaw + "\n"
           + "muscle: " + muscleFluc + "\n"
           + "pressure: " + airPressure + "\n"
           + "frequency: " + frequency + "\n"
           + "note: " + note + "\n";
  }

}
/*
---------------------------
FREQUENCY ANALYZER COMMENTS
FAST FOURIER TRANSFORM CLASS / FRQ TO MIDI / VIA MINIM LIBRARY
---------------------------
Analyses amplitude between 0 and 22050 hertz.
Corresponds each index in [f] to a frequency and amplitude value.
Finds peak volume via max(max). Compares.
If value is equal to amplitude of peak, get index, corresponding to freq.
Uses frequency to MIDI formula: 69+12x(logxlog(freq/440Hz). Casts product as a int.
And finally, casts n to a note based on division of 12 tones/semitones.
---------------------------
*/

void findNote() {
  fft.forward(in.left);
  for (int f=0;f<sampleRate/2;f++) {
    max[f]=fft.getFreq(float(f));
  }
  maximum=max(max); 
  for (int i=0; i<max.length; i++) {
    if (max[i] == maximum) {
      frequency= i;
    }
  }

  midi= 69+12*(log((frequency-6)/440));
  n= int (midi);

if (n%12==9){
    note = ("a");
  }
  if (n%12==10){
    note = ("a#");
  } 
  if (n%12==11){
    note = ("b");
  }
  if (n%12==0){
    note = ("c");
  } 
  if (n%12==1){
    note = ("c#");
  } 
  if (n%12==2){
    note = ("d");
  }
  if (n%12==3){
    note = ("d#");
  }
  if (n%12==4){
    note = ("e");
  }
  if (n%12==5){
    note = ("f");
  }
  if (n%12==6){
    note = ("f#");
  } 
  if (n%12==7){
    note = ("g");
  } 
  if (n%12==8){
    note = ("g#");
  }
}

void stop(){
  in.close();
  minim.stop();
  super.stop();
}

6 Digitizing Facial Movement During Singing : Data Collection and Reverse Engineering

While building the interface we had to keep in mind that we would soon be reverse engineering our findings into a “singing machine” and that our code should begin to reflect that. Our solution to solved two late-game issues: what is the form and application of this singing machine and what does it look like? Our testing interface of sliders became the perfect jumping off point to a visual interface that resembling a soundboard. Once we have reviewed enough data to know which configurations of mouth height/width, jaw protrusion, sternocleidomastoid fluctuation and air pressure produce each of the twelve tones and semitones in an octave we can program Ableton to output those notes upon a non-singing user’s input. The user can then view their biomechanical data through a familiar sound-generating interface: a soundboard.

Aesthetic direction for final interface:

For the next three weeks: official data recording/review, reversing the system/sending data to Ableton and finally refined the aesthetics of the interface.