Audio input streaming problem on Series 60 v2.6 based phones (6630/6680)

Hi

I posed the question I asked to Forum Nokia Pro.  They admitted that there is no way to get around only recording 20ms frames of audio with each call to CMdaAudioInputStream::ReadL().  This affects only the Nokia 6630 and 6680 so far.  The 320 bytes PCM is locked.  They have promised to look into making the audio API more flexible again, like it used to be.  50 callbacks per second does put quite a performance strain on an application.

I came to a satisfactory solution to my problems in the end.  First an explanation of what the problem was, then a quick tour of the solutions and failures:

My application is recording voice data, encoding to AMR on the fly and streaming using RTP over GPRS.  It's for a Push-To-Talk application.  In addition to the audio callbacks, the application itself is quite busy behind the scenes with timers, network and UI callbacks all over the place.  The application is still only single-threaded.

When I found out that the 6630/6680 was only recording 320 bytes at a time, my first stop-gap measure was to quickly buffer the recorded 320 bytes recorded, appending it onto another buffer then return from the callback straight away.  Once 700ms were buffered, I encoded the buffer from start to finish in one go.

This seemed to work, however bug-reports started to come in reporting audio glitches after several seconds.  It seems that the above solution was not recording audio in real-time, so on the receivng side, since less than 1 second of audio per second was received, glitches occurred.  The problem was that while the buffering scheme worked, the 320 bytes PCM could be recorded and the callbacks handled at a rate of 50/sec (just), but when it came to encoding 700ms of audio into AMR in one uninterrupted block of execution, the time it took to complete was greater than the 20ms period between callbacks from the audio input stream.  It never managed to catch up the lag created each time this 700ms of audio was encoded.  So each second, I was dropping 5-6 frames of audio whilst this encoding took place.

My eventual solution was the encode each 320 byte frame as it was recorded (something that I'd tried before but didn't work).  I also stripped down the code so that it was executing the bare minimum of code in the audio input stream callback:

Since the 6630/6680 are forced to use the MMF to encode/decode AMR (the old AMR APIs available in 2.0/2.1 based phones (CPcmToAmrEncoder, CAmrToPcmDecoder) have been removed) and I'd based the MMF code on top of the existing media server code.  The first improvement to make was to convert ALL descriptor audio buffers into CMMFDescriptorBuffers that have to be used by the CMMFCodecs in use.  This one step removed a significant amount of memory copying from buffer to buffer, (which proved to be really time-consuming as it turns out).

The other improvement, as mentioned, was to encode each frame as it was recorded in the MaiscBufferCopied() callback (following another call to CMdaAudioInputStream::ReadL() - that HAS to be the first line in the callback after checking the error code).

These improvements took the audio recording from around 93% speed to 99.9% speed, good enough for the application in hand.

I also tried another solution that I never got to work.  This was based on the assertion that the audio was the critical path of the application and it HAD to have it's audio callbacks handled in real-time.  Since I have no control over the priority of the active object in CMdaAudioInputStream since it's hidden by the MMdaAudioInputStreamCallback interface, I thought about running that active object on it's own in a real-time priority thread, so taking the application multi-threaded.  This proved more trouble than it was worth after all the shared memory and synchronisation.  I based that solution on the multi-threaded audio-mixer example available from Forum Nokia.

Furthermore, CMMFDevSound works no better than the CMda* stream classes that wrap it.  CMMFDevSound also exhibits this same 20ms frame at a time behaviour.

Hope this helps.

Regards.

Andy

Hi

For the Push-To-Talk application I'm working on, we're using AMR narrow band with DTX turned off, encoding using mode 1 AMR 5.15kbit/s.

I don't have real hard performance figures, but using discontinuous transmission as you are, I'd imagine the AMR encoder would have to put in quite a lot of extra work analysing the input PCM against observed background silence to determine whether anyone was speaking.  This is probably done on a frame-by-frame basis I'd imagine?  I'm sure that I read somewhere that the AMR codec running on the kind of ARM processors available in smartphones today could encode one 20ms frame of PCM in around 4-5ms.  Obviously this will go up when the overhead of DTX is added onto it.

Perhaps these are clock cycles you can't afford, especially if you are encoding video frames on the fly at the same time! 

Could you perhaps turn DTX off, so generating a constant stream of AMR audio even during silence?  While it would increase the amount of data you are transmitting, you may be able to encode audio in real-time.  Would whatever standard for VTC you're using allow this?

If you could use AMR with DTX turned off, perhaps you could take the octet aligned data generated by the codec and pack it into bandwidth efficient mode using a little bit-shifting to save an extra 5-10% of data transmission?

As for AMR decoding, you should have no performance issues there.  As with all lossy compression, encoding is intensive, decoding is quick and easy.

When you say you're working with HWDevice, do you mean your accessing hardware acceleration directly using RHwaDevice?  I'd be interested to learn of how you're doing that and what benefits that can bring.

You said you were playing around with the data sending object in your system too.  Would it be possible to send larger data packets out over the network (or send 2 packets in every asynchronous callback?)  so generating less asynchronous callbacks in that area, allowing more operating space for the audio?

Regards.

Andy.

p.s.

I don't know whether it's of any interest, but as of Symbian 8.1a (Series 60 2.6 Feature Pack 2), CMdaAudioInputStream and CMdaAudioOutputStream have had a function added to their API, SetDataTypeL().  You can pass this function the FourCC identifier of a codec and you can stream directly to/from that format instead of PCM.

I have been told that it does work for AMR, but you have less flexibility to set up the encoding than you would using CMMFCodec.  I don't know whether it offers any performance gains either, I doubt that it would.  Still, it makes streaming AMR code a little neater if you're not too fussy about what it's generating.

Hi

There's been a technical post on Forum Nokia:
http://forum.nokia.com/main/technical_library/FNTL/Customizing_the_buffer_size_of_CMdaAudioInputStream.htm

Apparently the maximum buffer size returnable from any audio recording API in S60 2nd edition FP2 & FP3 is 320 bytes (this includes CMMFDevSound).

On S60 3rd edition, this has increased in size to 4096 bytes maximum, giving more flexibility if necessary.

It's all to  do with tailoring the buffer size to best suit the underlying audio hardware.

Regards.

Andy "The Oracle"