Core Image memory management problem

Hi, everyone.
I'm working on the codebase of Apple's example application CIVideoDemoGL to create a computer vision software capable of face detection techniques. I'm currently extending the method renderCurrentFrame to catch Core Video buffer and store it into a CIImage instance, which I'll later draw inside a CIContext. I'll post the code fragment I'm working on:

- (void)renderCurrentFrame
{
// bla bla bla

if(currentFrame)
{
// bla bla bla

inputImage = [CIImage imageWithCVImageBuffer:currentFrame];

// bla bla bla

// calling of the method that make me do ugly dreams at night
[detectedFaces addObjectsFromArray:[faceFilter findFaces:inputImage]];
// I'll talk better about it later
// PS: detectedFaces is a NSMutableArray defined, faceFilter is an instance of a class I wrote.

// creation of outputImage as I need: bla bla bla

// drawing of outputImage: bla bla bla

}
QTVisualContextTask(qtVisualContext);

}

As I announced, within this method, I pass the CIImage instance to the method findFaces belonging to a class I wrote. That method is expected to return a NSMutableArray filled with NSDictionry instances. This array should contain the exact positions of the faces my App finds within every frame. This is the code I used.

-(NSMutableArray*) findFaces:(CIImage*)image
{
[image retain]; // here retainCount is 2...

// bla bla bla

VJIntegralImage *tempImage = [[VJIntegralImage alloc] init];
[tempImage doTheMagicWithImage:image];
[tempImage release];

// bla bla bla

[image release]; // ...and here retainCount comes back to 2
return faces;
}

VJIntegralImage is some class I wrote that takes in input a CIImage and creates an internal data structure to model the concept of Integral Image I need to implement such algorithm (formally a float**). This is the method that does the trick:

-(BOOL)doTheMagicWithImage:(CIImage*)image
{
[image retain]; // retainCount == 3
// creating a NSImage from the CIImage through NSCIImageRep
inputNSCIImageRep = [[NSCIImageRep alloc]initWithCIImage:localCIImage];
localNSImage = [[NSImage alloc] initWithSize:NSMakeSize(w, h)];
[localNSImage addRepresentation:inputNSCIImageRep];
// extracting NSBitmapImageRep from NSImage
inputImageRep = [[NSBitmapImageRep alloc] initWithData:[localNSImage TIFFRepresentation]];
[image release];
}

By releasing objects properly in -(void)dealloc (I'm talking about localNSImage, localCIImage and inputImageRep), I made sure that input CIImage ' s retain count is 2 at the end of the method findFaces.

Now, the question.
Using MallocDebug I found a substantial memory leak located upon the call to TIFFrepresentation. Appearently the software leaks 10 to 15 MB of memory every few seconds (as the autorefresh of the Activity Monitor can see). According to my analysis, I'm experiencing memory anomalies around the deallocation of CIImage's instance. In fact, by ending the method renderCurrentFrame with a retain count on that object equal to 1, my App crashes and GDB retuns a objc_msgSend as catched error, while if I avoid releasing the CIImage just one time I get that huge memory leak.

It appears that the problem is located within the code I wrote (yeah, figures...). I'm just a newbie to obj-c programming and I'm beginning to think that I misunderstood something really basic about memory managament. Can somebody help?

I hope my english and my code were plain enough. If not, I'd be happy to give more informations.

macbook pro, Mac OS X (10.4.7), cool!

Posted on Jul 4, 2006 8:48 AM

Reply
11 replies

Jul 6, 2006 11:15 AM in response to iSasha

Hi iSasha,

you say that you've released localNSImage, localCIImage and inputImageRep in the dealloc method of VJIntegralImage, but have you also released inputNSCIImageRep?

Chances are you've simply forgotten to release an instance somewhere so if the above is just an oversight in your post you'd be best to start by checking you have released everything that needs to be. If you use alloc, retain or copy it is your job to release.

If you still can't find the bug maybe you could post a link to the whole project for the forum to check.

PowerMac G5 1.6Ghz Mac OS X (10.4.6) 4 gig RAM & NVidia 6800 Ultra

Jul 8, 2006 10:58 PM in response to iSasha

Ok, I think I've figured out what is going on, or I at least have a couple of theories.

I'll explain the crashes first.

In doTheMagicWithImage:
localCIImage = image;
This holds the same reference as the image you pass to the method which you retain and then release within the method. If you then try and release localCIImage in dealloc you are releasing more than you are retaining.

[inputImageRep release];
You release this in both In doTheMagicWithImage: and dealloc.

And now the theory.
Apart from the above oversights you are releasing all that you should be. As you suspected, your problem is with the line:
inputImageRep = [[NSBitmapImageRep alloc] initWithData:[localNSImage TIFFRepresentation]];

Here the system retains both your CIImage and inputImageRep. Then your VJIntegralImage object is deallocated and the system is left holding some dead references. I believe that many CI methods are performed lazily (so you may not have an NSImage at this point) and it might be that your references are not released until the data has been used at least once by your program.

Once you finish the method and use the retained data things could be corrected by themselves.

Another possibility could be that some work is been carried out on another thread by the system and your VJIntegralImage object is being deallocated before the work is finished. In this case you will need to rewrite your VJIntegralImage class to allow it to stay open and use 'get and set' techniques to access your image data.

Also, by using local instances which are released at the end of the method you will keep things a lot cleaner and easier to follow. You could also use class methods to create your objects which would then be owned by the autorelease pool.

For example:
inputNSCIImageRep = [NSCIImageRep imageRepWithCIImage:localCIImage];

is the same as your:
inputNSCIImageRep = [[NSCIImageRep alloc] initWithCIImage:localCIImage];
[inputNSCIImageRep release];


The first example is autoreleased.

I hope I've been of some help, I know there's a lot here to think about.

PowerMac G5 1.6Ghz Mac OS X (10.4.6) 4 gig RAM & NVidia 6800 Ultra

Jul 10, 2006 2:17 AM in response to iSasha

Hi again,

I had a eureka moment on my way to work this morning and I may have found the actual cause of your problem. I am away from my Mac for a few days so it is not something I can investigate.

In VJIntegralImage the varibles ‘w’ and ‘h’ are not being set. This could mean that your NSImage is being initialized with no size. Maybe this is the reason for TIFFRepresentation not completing its task and failing to release its objects?


PowerMac G5 1.6Ghz Mac OS X (10.4.6) 4 gig RAM & NVidia 6800 Ultra

Jul 10, 2006 5:34 AM in response to Martin Buck

One first thing: I launched a release over inputImageRep also in the doTheMagicWithImage method because I noticed that, after its creation, inputImageRep has a retain count of 2 yet. So I thought that, for a correct deallocation, it would be necessary another release. Other thing: yes, in the code I gave you, I forgot to set w and h in initWithSize method of NSImage. Corrected, but the problem wasn’t here...

By the way, I did some modifies...

first_mod
In this code, I made the instance of VJIntegralImage always alive, so it’s initialized only one time and it’s accessed by the method doTheMagicWithImage, which worries about releasing objects. ...But same leak problem.

second_mod
In this code, instead, I used only convenience methods, except for the calling of TIFFRepresentation. Always the same problem... and if I set the [localNSImage TIFFRepresentation] object as autoreleased, it crashes.

About your second theory, I tried to launch the program in debug mode, setting a breakpoint over the calling of renderCurrentFrame method... Well, it didn’t crash, so I think that your theory is correct: there’s someone in my app using resources that VJIntegralImage release in advance. By the way, I have no idea about what to do to solve this problem 😟

Jul 11, 2006 11:01 AM in response to iSasha

iSasha

Today I started thinking about auto release pools. After a little reading I found that it is sometimes necessary to flush the pool when it gets too large.

I tested my theory by setting up my own pool around your foundFaces method like so:

//(in VJFilter.m)

-(NSMutableArray*) foundFaces:(CIImage*)image
{
[image retain];
NSAutoreleasePool *myPool = [[NSAutoreleasePool alloc] init];
// the rest of your code here...
[myPool release];
[image release];
return faces;
}


This stops the large leak but there's still a smaller one to find.

Martin.

PowerMac G5 1.6Ghz Mac OS X (10.4.6) 4 gig RAM & NVidia 6800 Ultra

Jul 11, 2006 10:20 PM in response to Martin Buck

This stops the large leak but there's still a smaller
one to find.


Correction, this is the same leak and it comes from a failure to release a CGLayerRef in VideoView.m - renderCurrentFrame. However, if I try to release this I cause a crash.

I'm not quite sure when the CGLayeRef should be released but am 100% certain that this is the cause of the leak.

PowerMac G5 1.6Ghz Mac OS X (10.4.6) 4 gig RAM & NVidia 6800 Ultra

Jul 12, 2006 1:55 AM in response to Martin Buck

...but if you comment all the CG*stuff, the program still leaks, instead if you leave uncommented that code AND you comment the line
[detectedFaces addObjectsFromArray:[faceFilter foundFaces:inputImage]];
you don't have the leak anymore. I think that the program crashes after releasing also CGLayeRef because it's been already released with CGContextRelease(drawContext).

My goal is to reach the bitmapdata of an image (that, in my case, is the frame of the video I play). There's another part of my program where I do the same job with these same classes but using input images taken from files, and in that case the program doesn't give any problem, so the leak is surely connected to the way resources are released during playback.
Anyway, as I was saying, I'm trying to access to the bitmapdata. For what I know, there are 2 ways to do that... 3 really:
1) through TIFFRepresentation --> leak
2) (directly from Cocoa drawing guide) "lock focus on the desired object and use the initWithFocusedViewRect: method of NSBitmapImageRep" --> done: same leak
3) using glReadPixels --> I tried it, following always CIVideoDemoGL example in method saveFrameToFile, but I obtain a blue image 🙂 ...

Jul 12, 2006 2:22 AM in response to iSasha

...but if you comment all the CG*stuff, the program
still leaks, instead if you leave uncommented that
code AND you comment the line


If I comment out VJIntegralImage I still have a serious leak and each leak I find ends up in my GeForce driver. What else would be using the GPU? Also if I don't open a movie the method does not run but the leak is still there. Clearly something else is going on.

I searched the web for problems about the TIFFRepresentation method and the only one I found was down to the AutoReleasePool but this is not a real problem because the pool would be cleared out eventually.

PowerMac G5 1.6Ghz Mac OS X (10.4.6) 4 gig RAM & NVidia 6800 Ultra

Jul 12, 2006 11:02 AM in response to iSasha

There is more than one leak and that is what is causing the confusion.

I can prove that your biggest problem is with the CGLayerRef by the following change.

In VideoView -renderCurrentFrame
replace:
CGContextRelease(drawContext);
with:
CGLayerRelease(drawLayer);

This solves the retention problem with the CIImage and stops the massive memory leak. You were releasing a CGContext that belongs to drawLayer. It was not created by you but by the CGLayer. By releasing drawLayer, which you did create, everything is released as it should be.

There are still two (or three) more leaks to track down so your search is not yet complete, and yes you were right, one is with the TIFFRepresentation method.

I need to get back to my own project now. I hope I've been of some help.

PowerMac G5 1.6Ghz Mac OS X (10.4.6) 4 gig RAM & NVidia 6800 Ultra

Jul 13, 2006 4:48 AM in response to Martin Buck

Today I'll check... Anyway, maybe I found an alternative solution: in other words, even if I still can't understand the problem, I lost too much time over that to spend some more time...

So, the idea is to extract bitmap directly from CGContext (or something like that):

NSMutableData* imageData = [NSMutableData data];
CGImageDestinationRef destCG = CGImageDestinationCreateWithData( (CFMutableDataRef)imageData, kUTTypeTIFF, 1, NULL);
CGImageDestinationAddImage(destCG, currentCGImageRef, NULL);
CGImageDestinationFinalize(destCG);
NSBitmapImageRep *bitmap = [[NSBitmapImageRep alloc] initWithData:imageData];
CFRelease(destCG);

...Then I pass the bitmap to my other classes, launching a [bitmap release] at the end of whole work.
It should work... Precisely, there's no more leak, but I'm still not sure it will do what I expect. I found that during last night, today I'll check.

Obviously, thank you very much for your help! Discussing with you was certainly very useful to me!

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

Core Image memory management problem

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.