What is OpenSL ES?

OpenSL ES is royalty-free, cross-platform, hardware-accelerated audio API tuned for embeded systems.

It provides a standardized high-performance, low-latency method to access audio functionality for developers of native applications.

OpenSl ES uses callback mechanism. This call back notify application that new buffer is able to added on buffer queue.

All operation of OpenSl Es is done by inteface. The interface provide basic method call like Java Interface.

There are general interface to use in OpenSl ES.

1. SLObjectItf : Object Interface

2. SLEngineItf : Engine Interface

3. SLPlayItf: Playback Interface

4. SLBufferQueueItf: BufferQueue Interface

5. SLVolumeItf: Volume Interface

<Initialization>

There are three initialization of audio engine, audio recorder and audio player.

- OpenSL ES engine initialization

Main role of OpenSL ES engine initialization is to connect with JNI, to make new engine object and to set several sampling parameters.

The important thing is the parameter of audio data's buffer queue and sampling parameter between transmitter and receiver must be same

SLresult result;

memset(&engine, 0, sizeof(engine));

//set sampling parameter

engine.fastPathSampleRate_ = static_cast<SLmilliHertz>(sampleRate) * 1000;

engine.fastPathFramesPerBuf_ = static_cast<uint32_t>(framesPerBuf);

engine.sampleChannels_ = AUDIO_SAMPLE_CHANNELS;

engine.bitsPerSample_ = SL_PCMSAMPLEFORMAT_FIXED_16;

//new object

result = slCreateEngine(&engine.slEngineObj_, 0, NULL, 0, NULL, NULL);

SLASSERT(result);

//initialization

result = (*engine.slEngineObj_)->Realize(engine.slEngineObj_, SL_BOOLEAN_FALSE);

SLASSERT(result);

//get engine interface and get other object

result = (*engine.slEngineObj_)->GetInterface(engine.slEngineObj_, SL_IID_ENGINE, &engine.slEngineItf_);

SLASSERT(result);

// compute fast audio buffer size.

// we need two things for low-latency.

// minimize buffer size (adjust it here) AND

// minimize buffering time before playing received audio recorder's buffering data

// modify buffer size adapted to request [before it busts]

bufSize = engine.fastPathFramesPerBuf_ * engine.sampleChannels_

* engine.bitsPerSample_;

bufSize = (bufSize + 7) >> 3; // bits --> byte

engine.bufCount_ = BUF_COUNT;

engine.bufs_ = allocateSampleBufs(engine.bufCount_, bufSize);

assert(engine.bufs_);

//remained buffer and received buffer

engine.freeBufQueue_ = new AudioQueue (engine.bufCount_);

engine.recBufQueue_ = new AudioQueue (engine.bufCount_);

assert(engine.freeBufQueue_ && engine.recBufQueue_);

for(uint32_t i=0; i<engine.bufCount_; i++) {

engine.freeBufQueue_->push(&engine.bufs_[i]);

}

The slEngineObj is not be able to use even though it is newly produced engine object.

You should implement engine interface by using slEngineObj. (ex recorder interface and player interface).

The fastPathFramesPerBuf is number of samples each buffer has.

The overall bufsize is twice the number of samples for all channels because samples are 16bits(2 bytes).

The freeBufQueue represents an idle buffer queue that primarily provides an empty sampling array.

The RecBufQueue is a receive buffer queue that is primarily used to store collected audio data and is also a source of playback data.

After engine is initialized, the freeBufQueue is initialized and 16 empty arrays of which size is 480 bytes are initialized.

- OpenSL ES recorder initialization

Recorder initialization mainly includes sound source settings, data format settings, sampling buffer queues, and interface acquisition.

sampleInfo_ = *sampleFormat;

SLAndroidDataFormat_PCM_EX format_pcm;

ConvertToSLSampleFormat(&format_pcm, &sampleInfo_);

//Set up a sound source

SLDataLocator_IODevice loc_dev = {SL_DATALOCATOR_IODEVICE,SL_IODEVICE_AUDIOINPUT,

SL_DEFAULTDEVICEID_AUDIOINPUT,NULL };

SLDataSource audioSrc = {&loc_dev, NULL };

//Set up an audio data pool

SLDataLocator_AndroidSimpleBufferQueue loc_bq = {

SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE,

DEVICE_SHADOW_BUFFER_QUEUE_LEN };

SLDataSink audioSnk = {&loc_bq, &format_pcm};

//RECORD_AUDIO permission is required to create a Recorder

const SLInterfaceID id[2] = {SL_IID_ANDROIDSIMPLEBUFFERQUEUE,

SL_IID_ANDROIDCONFIGURATION };

const SLboolean req[2] = {SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE};

result = (*slEngine)->CreateAudioRecorder(slEngine,&recObjectItf_,&audioSrc,

&audioSnk,sizeof(id)/sizeof(id[0]),id, req);

//Configuring Presets for Speech Recognition

SLAndroidConfigurationItf inputConfig;

result = (*recObjectItf_)->GetInterface(recObjectItf_,SL_IID_ANDROIDCONFIGURATION,&inputConfig);

if (SL_RESULT_SUCCESS == result) {

SLuint32 presetValue = SL_ANDROID_RECORDING_PRESET_VOICE_RECOGNITION;

(*inputConfig)->SetConfiguration(inputConfig,SL_ANDROID_KEY_RECORDING_PRESET,

&presetValue,sizeof(SLuint32));

}

//Implementation of recording objects

result = (*recObjectItf_)->Realize(recObjectItf_, SL_BOOLEAN_FALSE);

//Get Recording Interface

result = (*recObjectItf_)->GetInterface(recObjectItf_,

SL_IID_RECORD, &recItf_);

//Get Recording Queue Interface

result = (*recObjectItf_)->GetInterface(recObjectItf_,

SL_IID_ANDROIDSIMPLEBUFFERQUEUE, &recBufQueueItf_);

//Register Recording Queue Callback

result = (*recBufQueueItf_)->RegisterCallback(recBufQueueItf_,

bqRecorderCallback, this);

//Initialize audio acquisition centralized forwarding queue

devShadowQueue_ = new AudioQueue(DEVICE_SHADOW_BUFFER_QUEUE_LEN);

First, define sound source which is SLDataSource which included DataLocator and Data type.

OpenSL Es use general PCM data for data type.

Data Locator means saving position after collecting sound.

There are four types of Data Locator which is position of midi buffer, position of buffer queue, position of input/output device and position of memory.

OpenSL ES use PCM data to verify Data Locator and saving position of buffer queue to collect data.

For initialization of audio data pool, it means output of data and sets output position and output format of audio data which Recorder should set.

After initialization of recorder object, get recItf which means recorder interface.

This is necessary for recording.

The recBufQueueItf is interface of recording queue to register callback interface.

- OpenSL ES player initialization

The initialization of Player is similar to Recorder.

This include sound setting, collect data format setting, sampling buffer queue and getting interface.

sampleInfo_ = *sampleFormat;

//Initialize OutputMix to output audio data

result = (*slEngine)->CreateOutputMix(slEngine, &outputMixObjectItf_,

0, NULL, NULL);

//Implementation of OutputMix

result = (*outputMixObjectItf_)->Realize(outputMixObjectItf_, SL_BOOLEAN_FALSE);

//Configuring sound source data

SLDataLocator_AndroidSimpleBufferQueue loc_bufq = {

SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE,

DEVICE_SHADOW_BUFFER_QUEUE_LEN };

SLAndroidDataFormat_PCM_EX format_pcm;

ConvertToSLSampleFormat(&format_pcm, &sampleInfo_);

SLDataSource audioSrc = {&loc_bufq, &format_pcm};

//Configuring the Audio Data Output Pool

SLDataLocator_OutputMix loc_outmix = {SL_DATALOCATOR_OUTPUTMIX, outputMixObjectItf_};

SLDataSink audioSnk = {&loc_outmix, NULL};

/* Initialization Player */

SLInterfaceID ids[2] = { SL_IID_BUFFERQUEUE, SL_IID_VOLUME};

SLboolean req[2] = {SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE};

result = (*slEngine)->CreateAudioPlayer(slEngine, &playerObjectItf_, &audioSrc, &audioSnk,sizeof(ids)/sizeof(ids[0]), ids, req);

//Implementation of Player

result = (*playerObjectItf_)->Realize(playerObjectItf_, SL_BOOLEAN_FALSE);

SLASSERT(result);

//Get Player Interface

result = (*playerObjectItf_)->GetInterface(playerObjectItf_, SL_IID_PLAY, &playItf_);

//Get Volume Interface

result = (*playerObjectItf_)->GetInterface(playerObjectItf_, SL_IID_VOLUME, &volumeItf_);

//Get Buffer Queue Interface

result=(*playerObjectItf_)->GetInterface(playerObjectItf_,SL_IID_BUFFERQUEUE,&playBufferQueueItf_);

//register buffer interface callback

result=(*playBufferQueueItf_)->RegisterCallback(playBufferQueueItf_, bqPlayerCallback, this);

In comparision with initialization of Recorder, there is initialization of OutputMix.

OutputMix is used to output data by speaker.

The acquired playBufferQueueItf is interface of playback buffer queue and data source of playBufferQueueItf is same with recBufQueueItf of Recorder.

OpenSL ES collect data in data buffer queue and transmit it to playBufferQueueItf to play sound.

<Collect audio data>

The main process of collecting audio data is consist of initialization of buffer queue, starting record setting and starting record.

The size of starting is 2. Before recording, put 2 record array into record memory.

After starting record, record data is collected into these two arrays.

If record arrays are filled with data, callback is triggered taking recorded data from Recorder.

sample_buf *dataBuf = NULL; //Array of audio data collected

devShadowQueue_->front(&dataBuf); //Take out the collected array

devShadowQueue_->pop(); //Remove the head.

dataBuf->size_ = dataBuf->cap_; //Callback only after the array is full, so size can be set to the maximum length.

sendUdpMessage(dataBuf); //Send by UDP

sample_buf* freeBuf;

while (freeQueue_->front(&freeBuf) && devShadowQueue_->push(freeBuf)) {

freeQueue_->pop(); //Delete the used free array

SLresult result = (*bq)->Enqueue(bq, freeBuf->buf_, freeBuf->cap_); //Continue to the next collection.

sample_buf *vienBuf = allocateOneSampleBufs(getBufSize());

freeQueue_->push(vienBuf); //Add a new free array

}

The above code is included in callback.

First, take collected audio data from devShadowQueue and keep collecting audio data.

Using while loop, put array into collection buffer as much as possible so that collection buffer is free.

Collection buffer is used to store audio data from microphone.

<Playback audio data>

Playback starts after receiving necessary buffed data.

sample_buf *buf = NULL;

if(!playQueue_->front(&buf)) {

uint32_t totalBufCount;

callback_(ctx_, ENGINE_SERVICE_MSG_RETRIEVE_DUMP_BUFS,

&totalBufCount);

break;

}

if(!devShadowQueue_->push(buf)) {

break; // PlayerBufferQueue is full!!

}

(*playBufferQueueItf_)->Enqueue(playBufferQueueItf_,buf->buf_, buf->size_);

playQueue_->pop(); //Delete an array that has already been played

The playQueue is playbackQueue and if it is empty, there is no buffered data.

After playBufferQueueItf play audio data, playQueue pop played audio data which means delete it.

If playback finish, the callback registered in playBufferQueueItf is called.

sample_buf *buf;

if(!devShadowQueue_->front(&buf)) {

if(callback_) {

uint32_t count;

callback_(ctx_, ENGINE_SERVICE_MSG_RETRIEVE_DUMP_BUFS, &count);

}

return;

}

devShadowQueue_->pop();

buf->size_ = 0;

if(playQueue_->front(&buf) && devShadowQueue_->push(buf)) {

(*bq)->Enqueue(bq, buf->buf_, buf->size_);

playQueue_->pop();

} else{

sample_buf *buf_temp = new sample_buf;

buf_temp->buf_ = new uint8_t [BUF_SIZE];

buf_temp->size_ = BUF_SIZE;

buf_temp->cap_ = BUF_SIZE;

(*bq)->Enqueue(bq, buf_temp->buf_ , BUF_SIZE);

devShadowQueue_->push(buf_temp);

}

Above code is in callback.

First, take played audio data out from devShadowQueue which is transmission queue.

If audio data exists, delete it and take audio data out from playQueue to push into devShadowQueue.

The devShadowQueue has two roles, making sure continuation of playback and storing audio data temporarily.

If playQueue is empty or devShadowQueue fails to receive audio data, OpenSL ES played empty audio data.

Share: