|
|
Optimization of real-time video processing on Nokia N95 smartphone
To work with real-time video one needs at least two things - direct access to bitmaps and fast way to get frames from a video camera. To achieve the first goal one needs to keep in mind that Bitmaps are located on a shared heap that needs to be locked before any direct access can safely take place. This is because some other applications and/or memory management can shift things around so if you get an address of a Bitmap it might become invalid while you are still accessing Bitmap elements.
Direct access
In this example, we access CFbsBitmap iSrcBitmap created elsewhere:
iSrcBitmap->LockHeap(ETrue); // lock global heap (once regardless of the number of bitmaps) TUint8* ptr = (TUint8*)iSrcBitmap->DataAddress(); // direct access with the code that never leaves ptr[33] = 255; iSrcBitmap->UnlockHeap(ETrue); // unlock global heap
TIP: My experience shows that in bitmap-intensive applications in order to avoid unexplained crashes it is a good idea to change the priority of an application e.g. add the following string in AppUi::ConstructL()
RProcess().SetPriority (EPriorityHigh);
YUV to RGB conversion
2. In order to outperform BitBlt() function that is particularly slow when copies frames from a video camera (YUV format)
to RGB bitmaps one has to use his/her own conversion function from say YUV244 to RGB 16M. If done properly, this will speed up your code approximately 3 times. For example, YUV244 uses a unique gray (Y) and shared color (U, V) for two neighboring pixels in such a way that 4 bytes in YUV244 format result in 6 BGR bytes. In short U,Y1,V,Y2 -> G1,B1,R1,G2,B2,R2 according to a following formula:
U -=128;
V -=128;
R1 = Y1+1.370705*V;
G1 = Y1-0.698001*V-0.337633*U;
B1 = Y1+1.732446*U;
R2 = Y2+1.370705*V;
G2 = Y2-0.698001*V-0.337633*U;
B2 = Y2+1.732446*U;
By the way, somebody just gave me an excellent idea to use Y1, Y2 as gray channel in case you don't need color. In this case there is no need for conversion and multiplications! Note that in general Y is not an average RGB and is calculated as Y = R.299+G*.587+B*.114 which actually looks better then (R+G+B)/3
TIP: A good idea is to clip the resulting RGB values to make sure they stay within 0..255
In order to avoid above multiplications that are too slow one has to create look up tables for each value of intensity from 0 to 255. A good starting point to learn more about this conversion is http://local.wasp.uwa.edu.au/~pbourke/dataformats/yuv/
3. Also a good idea for using all capabilities of your video camera is to switch to a Landscape mode right away regardless of how you are going to hold a phone. It is especially important for image recognition and localization applications since it guarantees a wide vertical angle (so a user don't need to scan vertically) and a narrow horizontal angle (so the azimuth error of localization is small). In order to switch to a Landscape mode do two things. First, update your AppUi::ConstructL():
BaseConstructL(CAknAppUi::EAknEnableSkin|EAppOrientationLandscape); // code in your AppUi::ConstructL()
And second, Choose an additional option : "Read User Environment" in your mmp file ->Options->Capabilities.
TIP: Note that in a Landscape mode the longest screen dimension (320 pixel) corresponds to the width of a bitmap.
typo
just changed my own typo bits->bytes
Re: Optimization of real-time video processing on Nokia N95 smar
Is it only for Nokia N95 ?
Because for Nokia 6600 I'm receiving bitmap from ViewFinderFrameReady(CFbsBitmap& aFrame)
already in RGB (16 bits), even though I have EFormatYUV420Planar for camera. And I'm getting data from bitmap not by BitBlt but through DataAddress.
BTW for RProcess().SetPriority (EPriorityHigh) you need MultimediaDD capability in Symbian Signed, which require manufacturer approval, correct ?
Re: Optimization of real-time video processing on Nokia N95 smar
I tired using frames (BitMaps as you said) from a ViewFinder in Nokia N95 and they aren't 16M RGB.
May be they are 16MU (Display mode 11) or something, as opposed to RGB 16M (Display mode 8 ).
I never had time to find it out exactly but I suspect that BitBlt is pretty fast converting 16MU->16M so if
you build a simple application frames from a ViewFinder might suffice.
In fact, I have just checked the speed of such BitBlt operation (I isolated it in a loop so the frequency of
a call back function is not a factor) and it was twice slower than a correctly implemented YUV->RGB
conversion from video frames. Also if you need only gray channel Y component of YUV gives
it right away. Apart from that, the reason why I prefer using real video frames are twofold:
1. A camera can fill a buffer with several frames that you can process for motion estimation
and it then tells you how many frames were skipped (not saved in the buffer) while processing;
2. At the same time you can use a view finder to transfer its data to the given portion of the
screen using direct screen access, which saves a couple of copying operations. To do this just
put the following code in CCamera::PowerOnComplete()
CCamera::StartViewFinderDirectL() // with a proper list of parameters of course
Concerning your second question about MultimediaDD capability - I cannot say for sure.
In your IDE you can check a .mmp file option "Process Priority".
We buy unlocked phones in a sense that we don't stick to any service provider and
we use self-singed sisx files. When I go to the phone menu I am able to set up the following
option from the menu: Tools >Settings > Applications > Application Manager > Software Installation: "All "
as opposed to "Signed only". But I might confess - I am not an expert in all this licensing stuff.
Re: Optimization of real-time video processing on Nokia N95 smar
Don't rely on any symbian drawing code to be quick, specially not conversions of modes.
Sad, but true.
Its one of the biggest bottle necks in symbian and the main reason symbian phones have a reputation of being "slow".
(That, and the window server design)
Another tip for performance is to as long as possible avoid any byte accesses.
Reading a byte from memory takes exactly the same time as reading a whole 32 bit word, so if you need also the next byte in the stream, you should make sure to do 32 bit reads and mask and shift.
Also... please note that this SetProcessPriority hack is just that.. .a hack.
Most probably your app has some logic error that is just hidden because your app gets higher priority.
A good guess is the LockHeap calls... Or maybe something completely unrelated...
I can assure you that in a correctly written app, it will not matter how intensely you use bitmaps, it will not crash...
I would be _very_ careful in giving "TIPs" like that...
Re: Optimization of real-time video processing on Nokia N95 smar
If you are using self-signed sisx that mean this capability definitely not in MultimediaDD, but user-grantable.
Thanks for very useful info, I'm going to port app to Nokia N95 soon too.
Re: Optimization of real-time video processing on Nokia N95 smar
Alh is right and it is a very good comment about 32 bit (word) alignment. In fact, bitmap's scan lines are also aligned for efficiency reasons and the alignment is even more tricky. I also agree that setting high priority is a hack but a useful one when you need your application to start working and cannot debug it anymore.
The reason why this hack works is not completely clear to me but I doubt it is locking the heap twice or forgetting to unlock it since it usually leads to completely frozen state of the phone when you need to apply take_out_battery restart. I can even blame it on inperfections of OS since some bitmap intensive applications already PRESENT on the phone (like a bar code reader) repeatedly crash.
So the tip is just a tip, I am sure a lot of people who need running application or are under the pressure of a deadline would use it making a mental note that there is something wrong either with their code or with the system. There is a need for more research on the topic and it should be open to discussion.