CGImageSourceCreateThumbnailAtIndex doesnot rotate the image on ios 17

chriskhong wrote:

I tried omitting kCGImageSourceCreateThumbnailWithTransform or kCGImagePropertyOrientation but it still returns an image with wrong orientation. What's bugging me is that it is only an issue since ios 17.

That's what I'm saying. I think you are confusing orientation for rotation. They are not the same.

Do you recommend a more reliable API to resize and get the image with an expected rotation?

Thanks for your help!

You are really into the "dozens" category. It doesn't matter what method you choose. The operation is the same. You take the original image and transform it. If you want to perform both resize and rotation, then you will compose an affine transform that includes both scale and rotation. You might also need a translate because of the way these operations work with respect to the origin of the image.

The important part is that scaling and rotation does not have anything to do with thumbnails and orientation. Give that up now. It'll never work.

I could give you better information if you could explain what you are trying to do at a high level. This function is operating on CGImage. That's fine, but I don't know if that's what you really need. Most people would be working with UIImage, some with NSImage, a few with CIImage. There are lots of CGImage solutions too, but I'm not sure if that is appropriate. Plus you are using C, which is strange these days.

chriskhong wrote:

apparently a pretty practical approach as it is hybrid app both on Android and iOS.

Yeah, that's never a good idea.

So more details in our approach, we take an image using the API captureStillImageAsynchronouslyFromConnection, which returns a CMSampleBufferRef.

We then use it to construct a CGImageSourceRef, and then create a thumbnail from it with a target size and target rotation, which is calculated using device accelerometerData from CMMotionManager.

Is that all you get from Cordova? <sigh>

It works well for us so far, until the release of iOS 17 and it starts to fail when user taking the picture in landscape.

It seems like the API CGImageSourceCreateThumbnailAtIndex not adjusting the target rotation properly (I debugged and see that the target rotation is properly detected).

The problem here is that kCGImagePropertyOrientation is a property that is meant to be read, not written. It's just metadata that the operating system attaches to an image at creation. Since you are bypassing those higher-level APIs, you have to reverse-engineer this logic.

(As an aside, you said "it still works fine on iPadOS 17, can only reproduce on iOS 17". I don't know if this was a typo and you meant iPadOS 16 or you really meant that it works fine on iPad, but not on iPhone. I recommend that you avoid Apple's iPadOS vs iOS nomenclature. That just confuses people. It's all iOS, but there might be platform differences. The iPad camera is landscape. So if you take it while holding it in a portrait orientation, then kCGImagePropertyOrientation becomes important. I don't know about iPhone, but it is probably already portrait and that explains the problem you are experiencing.)

So, you have to find out how each iOS devices works with respect to this metadata and recreate it. A better idea would be to look for a higher-level API that does the work for you.

What you describe is too involved for me to try to reproduce. I'll just have to take a guess. This code looks like a good place to start. It's swift, but it uses those same low-level C APIs. It takes raw data and creates an image from it. Then, it creates a context that you can draw into. It finds out the correct transform and applies that to the context. Then it draws the image into that new context. Finally, it extracts the result image from the context and returns it.

I'm not sure how you need to construct your transform given your low-level source data. That code also needs to flip the image. That's always fun.

It sounds like you actually are doing thumbnails. If so, you might want to stick closer to your original idea. Only pay more attention to what happens on iPhone. The way you are adding that orientation metadata might work for iPad, but iPhone might not use it and expect the image to actually be rotated. So you might need to rotate the original, add the metadata, and then feed that into the thumbnail creation function with just kCGImageSourceCreateThumbnailWithTransform.

You aren't resizing and rotating the image. You are creating a thumbnail and trying to set an orientation. But since your "rotationAngle" conflicts with the settings of specified by kCGImageSourceCreateThumbnailWithTransform, you get unreliable results.

If you really want to make a thumbnail, then omit the kCGImagePropertyOrientation and create your thumbnail.

If you want to rotate and resize and image, then use a more appropriate API. There are several different methods you could choose. None of them have anything to do with thumbnails.