Run C++ Extractor Client Code

[loretoparisi]

Looking at the examples like the rhythm extractor:

https://github.com/MTG/essentia/blob/master/src/examples/streaming_rhythmextractor_multifeature.cpp

in the essentia architecture an extractor is used in pipe with an audio input node:

Algorithm* mono = factory.create("MonoMixer");
Algorithm* rhythmextractor = factory.create("RhythmExtractor2013");
rhythmextractor->configure("method", "multifeature");
connect(audioloader->output("audio"), mono->input("audio"));
connect(audioloader->output("numberChannels"), mono->input("numberChannels"));
connect(audioloader->output("sampleRate"), pool, "metadata.sampleRate");
connect(mono->output("audio"), rhythmextractor->input("signal"));

and then connected to the ouput:

connect(mono->output("audio"), rhythmextractor->input("signal"));

Finally the network run the nodes:

Network network(audioloader);
network.run();

Is it possible to run the extractor directly i.e. using the constructor class:

essentia::init();
essentia::streaming::RhythmExtractor2013 *rythmExt =
new essentia::streaming::RhythmExtractor2013();

/// pass the pcm input signal

rythmExt->process();

[dbogdanov]

You can pipe from VectorInput (http://essentia.upf.edu/documentation/doxygen/classessentia_1_1streaming_1_1VectorInput.html):

vectorInput = new streaming::VectorInput();
*vectorInput >> rhythmextractor->input("signal");

vectorInput->setVector(&signal);
Network network(vectorInput);
network.run();

Btw, I've simplified the code of this extractor, check the latest commit.

[loretoparisi]

Basically I cannot use the operator >> in the mixed code ObjectiveC / C++,

so can I do something like

 essentia::init();

essentia::standard::AlgorithmFactory& factory = essentia::standard::AlgorithmFactory::instance();

essentia::standard::Algorithm* rhythmextractor = factory.create("RhythmExtractor2013",
                                            "method", "multifeature");

essentia::Pool pool;

std::vector<essentia::Real> audio;

/// TODO: here we have to fill the real audio vector from WAVE/PCM samples

std::vector<essentia::Real> ticks;
essentia::Real confidence = 0;
essentia::Real bpm = 0;
std::vector<essentia::Real> estimates;
std::vector<essentia::Real> bpmIntervals;

rhythmextractor->input("signal").set(audio);
rhythmextractor->output("ticks").set(ticks);
rhythmextractor->output("confidence").set(confidence);
rhythmextractor->output("bpm").set(bpm);
rhythmextractor->output("estimates").set(estimates);
rhythmextractor->output("bpmIntervals").set(bpmIntervals);

pool.add("rhythm.ticks",ticks);
pool.add("rhythm.ticks_confidence",confidence);
pool.add("rhythm.bpm",bpm);
pool.add("rhythm.estimates",estimates);
pool.add("rhythm.bpmIntervals",bpmIntervals);

rhythmextractor->compute();

const std::vector<essentia::Real>& ticksR = pool.value<std::vector<essentia::Real> >("rhythm.ticks");
const essentia::Real ticksConfidenceR = pool.value<essentia::Real>("rhythm.ticks_confidence");
const std::vector<essentia::Real>& estimatesR = pool.value<std::vector<essentia::Real> >("rhythm.estimates");
const std::vector<essentia::Real>& bpmIntervalsR = pool.value<std::vector<essentia::Real> >("rhythm.bpmIntervals");
const essentia::Real bpmR = pool.value<essentia::Real>("rhythm.bpm");

//TODO: map to std:vector NSArray
//TODO: print out

// clean up
delete rhythmextractor;

// shut down essentia
essentia::shutdown();

[loretoparisi]

I explain my knowledge better here.

The basic issue here is that in the ObjC / C++ mixed mode is hard to write the operators overload >> << so I cannot write something like with another example from the standard example codes:

 rhythmextractor->output("ticks") >> PC(pool, "rhythm.ticks");   

But I have to figure out how to write this in a different way, may be something like that:

std:vectoressentia::Real ticks = 0.0;
rhythmextractor->output("ticks").set(ticks);
pool.add("rhythm.ticks",ticks);

So far my understanding looking at the standard examples is that to compute I have to do something like

Algorithm* fc = factory.create("FrameCutter",
                                    "frameSize", frameSize,
                                   "hopSize", hopSize);   


    Algorithm* w = factory.create("Windowing",
                                   "type", "blackmanharris62");

then I can connect them together in this way:

  // set frameCutter:
  vector<Real> frame;
  fc->input("signal").set(audioBuffer);
  fc->output("frame").set(frame);

  // set windowing:
  vector<Real> windowedFrame;
 w->input("frame").set(frame);
 w->output("frame").set(windowedFrame);    

Then I have to process the frames like so:

 // process frames:
 while (true) {

   // compute a frame
    fc->compute();

   // if it was the last one (ie: it was empty), then we're done.
   if (!frame.size()) {
    break;
  }

    w->compute();

    pool.add("spectral.mel_bands_energy", bands);
 }    

As soon as I have in the essentia:Pool the values I can get the values or pass them to another extractor:

// compute rhythm transform:
  vector<vector<Real> > rhythm;
  rd->input("melBands").set(pool.value<vector<vector<Real> > >("spectral.mel_bands_energy"));
  rd->output("rhythm").set(rhythm);
  rd->compute();   

At this point computation was done and I can get the values

  for(int i=0; i<(int)rhythm.size(); i++)
      printf("%i",rhythm[i]);

[loretoparisi]

Just another example for the PredominantMelody if I'm following the guide here in the right way:

http://essentia.upf.edu/documentation/howto_standard_extractor.html

-(void) essentiaCalculatePredominantMelody {

// register the algorithms in the factory(ies)
essentia::init();

/////// PARAMS //////////////
// don't change these default values as they guarantee that pitch extractor output
// is correct, no tests were done on other values
int framesize = 2048;
int hopsize = 128;
int sr = 44100;

// instantiate algorithms
essentia::standard::AlgorithmFactory& factory = essentia::standard::AlgorithmFactory::instance();


essentia::standard::Algorithm* equalLoudness = factory.create("EqualLoudness");

essentia::standard::Algorithm* predominantMelody = factory.create("PredominantMelody",
                                              "frameSize", framesize,
                                              "hopSize", hopsize,
                                              "sampleRate", sr);

// data storage
essentia::Pool pool;

std::vector<essentia::Real> audio;

/// TODO: here we have to fill the real audio vector

equalLoudness->input("signal").set(audio);

std::vector<essentia::Real> audioEq;
equalLoudness->output("signal").set(audioEq);
predominantMelody->input("signal").set(audioEq);

std::vector<essentia::Real> pitches;
std::vector<essentia::Real> pitchConfidences;
predominantMelody->output("pitch").set(pitches);
predominantMelody->output("pitchConfidence").set(pitchConfidences);

pool.add("tonal.predominant_melody.pitch",pitches);
pool.add("tonal.predominant_melody.pitch_confidence",pitchConfidences);

predominantMelody->compute();

const std::vector<essentia::Real>& pitchesR = pool.value<std::vector<essentia::Real> >("tonal.predominant_melody.pitch");
const std::vector<essentia::Real>& confidenceR = pool.value<std::vector<essentia::Real> >("tonal.predominant_melody.pitch_confidence");

NSLog(@"number of frames: %lu", pool.value<std::vector<essentia::Real> >("tonal.predominant_melody.pitch").size() );

float tm = 0.0;
for (int i = 0; i<pitchesR.size(); i++)
{
    tm = tm + float(hopsize)/float(sr);
    NSLog(@"%f %f %f",pitchesR[i], confidenceR[i], tm);
}

// clean up
delete predominantMelody;
delete equalLoudness;

// shut down essentia
essentia::shutdown();

}

[dbogdanov]

if I understood correctly, the problem is in << >> operators, can you use this instead (which is equivalent)?
connect(rhythmextractor->output("ticks"), pool, "rhythm.ticks");

[debugworking]

@dbogdanov
I compiled on iOS platform a static library .It can be work , but does not like the right results, I would like to use raw audio data to detect rhythm.I use the pcm data, but I feel the result is wrong, so I did a test, I used the zero pcm data, but the results still can output, to ask where I was set up wrong!