|
|
Multichannel Sound using CMdaAudioOutputStream (v1.1)
Welcome to my little tutorial. I will try to give you the directions on implementing your own little multi-channel mixer. Due to lack of time I can not supply working example code.
We will implement our system in the following order:
- Getting CMdaAudioOutputStream to work
- So how does mixing work?
- Using Threads
- Additional Hints
1. Getting CMdaAudioOutputStream to work
This probably is the easiest chapter. First I want to recommend you the read of a presentation which was released at the Symbian Exposium 2003. It does not go into detail too much, but it is a good base.
You can download it here: http://www.symbian.com/events/expos...
I also want to recommend the WaveGen-example by Nokia which is a bit rough, but hopefully gives you a starting point. Get it here (registration required): http://www.forum.nokia.com/files/nd...
If you already have a game you want to add sound to, I recommend you of disabling all processing time consuming operations (such as drawing, game AI) first. You can still enable these later when your sound engine is ready.
The first thing you need to do is to write a class which extends MMdaAudioOutputStreamCallback.
The following methods need to be implemented:
#define BUFFERSIZE 512
class MySound : public MMdaAudioOutputStreamCallback {
// callback functions for MMdaAudioOutputStreamCallback
public:
MySound();
void Open(); // opens the stream
public:
virtual void MaoscOpenComplete(TInt aError);
virtual void MaoscBufferCopied(TInt aError, const TDesC8& aBuffer);
virtual void MaoscPlayComplete(TInt aError);
protected:
// this method fills the buffer and writes it into the stream
void UpdateBuffer();
// this methoed needs to implemented by YOU
void FillBuffer(TInt16* buffer, TUint len);
// we will also need these...
protected:
CMdaAudioOutputStream* iStream; // handle to the stream
TMdaAudioDataSettings iSettings; // stream settings
TInt16* iSoundData; // sound buffer
TPtr8* iSoundBuf; // descriptor for using our soundbuffer
}
MaoscOpenComplete will be called after you initialized the CMdaAudioOutputStream. MaoscBufferCopied will be called after your data buffer was copied to the internal buffer. MaoscPlayComplete will be called after sound output stopped.
So how do we open the OutputStream? We have to make sure that iStream was initialized. We will also initialize our sound buffer. I use a BUFFERSIZE of 512.
iStream = CMdaAudioOutputStream::NewL(*this);
iSoundData = new TInt16[BUFFERSIZE];
iSoundBuf = new TPtr8((TUint8*)iSoundData, BUFFERSIZE*2, BUFFERSIZE*2);
}
Then we just call Open():
iStream->Open(&iSettings);
}
Note we did not initialize iSettings.
If everything worked right, MaoscOpenComplete() will be called. This is the place we will do things like setting samplerate, etc. We will use a sampling rate of 16khz and a bit depth of 16 bit as this worked best for me.
if (aError==KErrNone) {
// set stream properties to 16bit,16KHz mono
iStream->SetAudioPropertiesL(TMdaAudioDataSettings::ESampleRate16000Hz, TMdaAudioDataSettings::EChannelsMono);
// note that MaxVolume() is different in the emulator and the real device!
iStream->SetVolume(iStream->MaxVolume());
iStream->SetPriority(EPriorityMuchMore, EMdaPriorityPreferenceNone);
// Fill first buffer and write it to the stream
UpdateBuffer();
}
Now the stream is ready for usage. The packet writing is actually quite simple.
// calculate the contents of the buffer
FillBuffer(iSoundData, BUFFERLENGTH);
// call WriteL with a descriptor pointing at iSoundData
iStream->WriteL(*iSoundBuf);
}
Of course it's your job to fill the buffer with useful data. As you may have noticed the buffer format is 16 bit signed which actually is the same format used in 16 bit WAV-files.
So a short time period after we wrote the packet into the stream MaoscBufferCopied() will be called. I recommend on filling the next buffer and writing it right in this callback.
if (aError==KErrNone) {
UpdateBuffer();
}
}
So what if an underflow happens? Underflow means that the audio device runs out of data as we were not writing it fast enough. If this happens MaoscPlayComplete() will be called. We will just write a new packet to restart the sound.
I use this code here:
// we only want to restart in case of an underflow
// if aError!=KErrUnderflow the stream probably was stopped manually
if (aError==KErrUnderflow) {
UpdateBuffer();
}
}
Note that the Exposium document states that in case of underflow the stream needs to be Closed() and Opened() before writing the next packet. I found no difference in doing so.
If you want to close the stream just call iStream->Close(). MaoscPlayComplete() will be called shortly after that.
Things you may want to improve:
dynamic buffer size, so you can write doublesized packets after opening or in case of underflow.
Against what I previously suggested (moving the buffer filling/writing into a Timer) I get very good latency by using a small buffersize.
2. So how does mixing work?
In this (rather short) part I will explain to you how to mix several sound channels. Mixing by itself is not so hard. Just take the average of all input sounds.
TInt16 *input1, *input2;
for (int i=0; i<BUFFERSIZE; i++) {
output[i] = (input1[i] + input2[i]) / 2;
}
The problem with this is that all input channels will have the same volume. You will probably want to have different volumes for each channel. I recommend using 8 bit samples. You will not hear the difference on that tiny phone speaker anyway.
If you want to know how to convert 8bit WAV files to 8bit signed format, download my Sound Library. A converter including source is in the ZIP file.
Here is how the mixing with different volumes works:
TInt8 *input1, *input2;
TUint8 volume1, volume2;
for (int i=0; i<BUFFERSIZE; i++) {
output[i] = input1[i]*volume1 + input2[i]*volume2;
}
We get away with one multiplication per channel per output sample which should be ok perfomancewise. Please note that the sum of the volume factors (e.g. volume1+volume2) should be equal to 256 (or less if you want the sound to be low).
How do we handle the situation if the samplerate of your input sample does not match the samplerate of your output stream?
The processing of your input sample to change it's sample rate is called "resampling".
Let's pretend we have an input sample with 8khz sample rate and want to resample it to 16khz. We basically need to duplicate each input sample.
TInt16* output;
for (int i=0; i<BUFFERSIZE; i++) {
output[i] = input[i/2];
}
This looks easy but gives us an additional performance hit. Especially if we used a sample rate like 12000hz as we needed an additional multiplication. We can work around this by using fixed point arithmetics.
TUint step=8000*0x100/16000;
for (int i=0; i<BUFFERSIZE; i++) {
output[i] = input[pos >> 8];
pos+=step;
}
Much better performancewise. I think you basically get the idea.
As nanard pointed out in the comments below it is faster to use pointer arithemtics in this case:
do {
*(output++) = *input;
*(output++) = *(input++);
} while(--i);
However tricks like these only work with simple factors.
3. Using Threads
Until now we looked at our multichannel engine in an isolated environment. But what if we want to use it while other stuff is happening (e.g. graphics drawn, game logic, ...)? The problem is with active objects. These provide an easy way of doing multithreading. The problem is that the execution of an active object can not be cancelled by another active object. This is very bad for our sound engine as the delayed call to our sound routine will result in underflows. A way out of this problem is provided by threads which is not that easy as it sounds.
To use Symbian threads the right way we need to implement a simple Client/Server architecture. I will not go into more detail here as it is not in scope of this article.
Examine these examples which come with the Series 60 SDK 1.0:
Epoc32Ex\Base\IPC\ClientServer\Simple
Series60Ex\ClientServerAsync
These examples differ to what we need in the following aspects:
We probably do not want a standalone server. We want the server to be dependent on our main program.
We want to share heap between our threads. Make sure you use the alternative thread constructor which allows specifying the heap.
CMdaAudioOutputStream uses active objects. So make sure you install an active scheduler.
I have some other hints for you:
Do not try to access CEikonEnv from your thread. It will crash unless you implement an AppUI in your thread or use a dedicated RWsSession.
If you want to close the thread do that in the MaoscPlayComplete method.
Set the thread priority to high. We do not want to get underflows at all costs.
4. Additional Hints
So now you know how to mix sounds and write them into the sound stream. But how should your game organize samples the best way?
I am using something like this:
public:
TUint16 samplerate;
TInt8* data;
TUint length;
};
class CChannel {
public:
CChannel() {
sample=NULL;
}
void PlaySample(CSample* snd) {
sample=snd;
pos=0;
step = snd->samplerate*0x100/16000;
}
void MixSample(TInt16* buffer, TUint len) {
if (sample==NULL) return;
for (int i = 0; i < len; i++) {
// check for end of sample...
if (pos >= sample->length << 8) {
sample = NULL;
return;
}
buffer[i] += sample->data[pos >> 8] * volume);
pos += step;
}
}
public:
CSample* sample;
TUint8 volume;
TUint pos;
TUint step;
};
class CMixer {
public:
void MixChannels(TInt16 *outputBuffer, TUint len) {
int i;
for (i=0; i<len; i++) {
outputBuffer[i]=0;
}
for (i=0; i<4; i++) {
channels[i]->MixSample(outputBuffer, len);
}
}
public:
CChannel* channels[4];
};
Of course this code is not complete but you should get the point. You have a number of "virtual" channels which can play one sample each. If you want to play a sample just find a channel that is not playing a sample currently. Otherwise stop the channel which is playing for the longest time.
To fill your buffer just call CMixer like this:
// calculate the contents of the buffer
iMixer->MixChannels(iSoundData, BUFFERLENGTH);
// call WriteL with a descriptor pointing at iSoundData
iStream->WriteL(*iSoundBuf);
}
I hope the revised version of this tutorial gives answers to some of the questions asked. If you want to try out my sound engine library which includes a player for MOD music, just download it from symbian.jep.de.
> Little correction
Hi!
As pawel pointed out there a code snippet somehow got wrong. As I did not find a way to edit this article, I post the correction here.
This is the wrong code:TUint pos=0;
TUint step=8000*0x100/16000;
for (int i=0; i>8];
pos+=step;
}
Here is how it should be:TUint pos=0;
TUint step=8000*0x100/16000;
for (int i=0; i<BUFFERSIZE; i++) {
output[i] = input[pos >> 8];
pos+=step;
}
> Little correction
> Little correction
TUint pos=0; TUint step=8000*0x100/16000;
for (int i=0; i> 8];
pos+=step;
is still VERY SLOW.
you should better do :
int i = BUFFERSIZE; do *(output++) = *input; *(output++) = *(input++); while(--i);
> Little correction
Hi!
Thanks for your suggestion. But due to an error on my part the original code became a little bit corrupted. But Eric fixed this.
Your code definitely is faster at copying stuff. But my resampling code does more than that (copying).
So if you can come up with a nice optimization of that, I will be happy to incorporate it into the article.
zeep
> Little correction
> Multichannel Sound using CMdaAudioOutputStream
> Multichannel Sound using CMdaAudioOutputStream
HI!
Its really a helpfull tutorial, but i need the complete source code. I dont know how FillBuffer() is implemented.
Regards Fek
> Multichannel Sound using CMdaAudioOutputStream
Hi!
FillBuffer() just fills the sound data into the buffer. For testing purposes you could use the raw data of a 16bit mono 16khz WAV file as testing data.
For the structure of a WAV file just use Google.
I am sorry I don't have the source code of my mixing engine separated. It's incorporated into my games lib which uses stuff like custom file formats etc.
zeep
> Multichannel Sound using CMdaAudioOutputStream
> Multichannel Sound using CMdaAudioOutputStream
Hi!
Yes, I am using threads when using the sound engine in a game. This is due the fact that the callback functions apparently are dependent on active objects.
The problem with that is that the execution of an active object can not be paused or interrupted. This results in the sound engine callback to be called too late which results in underflows/skipping.
Another problem is that the WriteL()-method seems to block for a short time (although it's asynchronous - I don't get this either). So your main thread gets paused now and then.
After I used a dedicated thread for my sound engine the performance hit was much lower - close to unnoticeable. There still is enough opportunity to optimize nevertheless.
So the consensus basically is: if you want to use multichannel sound in a Symbian game, better do use threads. If you found another way to make it work, please tell us ;)
zeep
> Multichannel Sound using CMdaAudioOutputStream
> Multichannel Sound using CMdaAudioOutputStream
Hi!
Basically the work is done when you moved the CMdaAudioOutputStream stuff into another thread.
You can then use the normal IPC-means of Symbian to send pointers to samples that should be played to your sound thread. Not too much magic there.
Perhaps we need another tutorial which explains Symbian threads?
zeep
> Multichannel Sound using CMdaAudioOutputStream
Hi!
I call the writeL function when a key was pressed. The problem is, when i press a key 3 times one after another the sound is playing 5 or more times. Why?
Regards Fek
> Multichannel Sound using CMdaAudioOutputStream
Hi!
I think there is a misunderstanding on how the multichannel sound works.
The sound is streamed to the output device. This basically means that the whole time data is sent to the device. Even if no sound output is currently needed by your game. The WriteL()-method should be used just in the callback method and not by yourself. So calling WriteL() on keypresses is not a good idea.
If you want to play a sound you have to mix it into the audio stream in the (here not implemented) FillBuffer()-method.
zeep
Source Code
Hello,
If any one implemented the code then please send the code in this forum. No matter if there is some errors. I need the code very much. I think zeep can help us a lot about this.
Tipu