Delivering media on S3/Cloudfront using signed URLs is not working

Question

Level 1

17 points

Delivering media on S3/Cloudfront using signed URLs is not working

Does anyone have any experience serving media files on S3/Cloudfront using signed URLs?

I am a developer who is building a podcast hosting service and my feeds are passing validation tests and also work in the Overcast podcast app. I'm having a really hard time getting them to work with the Apple Podcast Directory and the Podcast app.

The crux of the problem is figuring out how to serve media files that need to be accessed through signed URLs on Cloudfront. Signed URLs allow you to securely distribute private content, see: https://advancedweb.hu/how-s3-signed-urls-work/

Initially, I used the signed URLs directly in my feeds, which worked fine in Overcast, but not in the Apple Podcast app. I think it's because the Apple Podcast Directory was providing cached versions of my feed, and since the links expire after 1 hour, they were no longer valid when the Podcast App tried to download them. This is just my theory because I do not know how to find out what the actual problem was. All I know is that the podcast app could not load the files.

So I changed the feed to deliver URLs to media files that are a 302 redirect to the signed URL on Cloudfront. My theory is that the podcast app will follow the 302 redirects, which will create a new signed URL that has not expired. However, the Apple Podcast Directory will not validate my feeds with these 302 redirect links. I get the error "Can’t download episodes from your feed." Again, Overcast has no problem and the feeds validate fine.

Since the artwork is also using a signed URL and the 302 redirect, I get the error " Podcast artwork must be between 1400 x 1400 and 3000 x 3000 pixels, JPG or PNG, in RGB color space, and hosted on a server that allows HTTP head requests." I assume it's because the redirect link doesn't include the file extension.

I feel like I'm shooting with my eyes closed. I have ideas that I can try, but I don't have any documentation, and I can't dig any deeper with the Apple Podcast Directory to find out exactly why things are failing.

Any help is greatly appreciated! Also, if there are any docs/resources I should know about, please advise. Thank you!

Posted on Apr 14, 2020 8:48 AM

Reply

Answer 1

JP Redtopia Author

Level 1

17 points

Apr 14, 2020 10:40 AM in response to Roger Wilmut1

Thanks for taking a look Roger. The HTTP request you show for my enclosure url looks pretty standard for a 302 redirect. It specifies the 302 status code and the location headers, and a typical http request will follow the location URL.

You did a byte range request on the original URL, not the redirect. I think the byte range request would occur on the final location. Cloudfront/S3 does support byte range requests. Use cURL -L to follow any redirect to reach the eventual endpoint.

A few things could be happening:

A correct media file extension is required
The Apple Podcast Directory validator does not follow 302 redirects
If the Apple Podcast Directory validator does follow 302 redirects, perhaps I need to modify the response headers

I see a huge benefit in using a 302 redirect as opposed to linking directly to the media files because then I can start doing more advanced stuff like logging analytics and serving private podcasts. However, doing a redirect with a .mp3 file extension is a bit more complex.

Reply

Answer 2

Apr 14, 2020 8:59 AM in response to JP Redtopia

You can't have podcasts in the iTunes Store where a login is required. It's not allowed, and in any case there's no mechanism for dealing with it.

However, it used to be the case in the iTunes application on a Mac that you could have a podcast which required a login because the mechanism was there to handle it - it very likely is still the case, though whether it applied in Catalina's Apple Podcasts app I have no idea. Also I don't know anyt4hing about the iOS podcasts app but I would doubt that it has this facility.

So you can give the feed URL to your customers and they may be able to subscribe manually (from the File menu) in iTunes on a Mac or PC - you would need to experiment with this to see if and how it works, but these podcasts can't be in the iTunes Store.

EDIT: further to this, Apple's page at https://help.apple.com/itc/podcasts_connect/#/itc1723472cb

says:

Apple Podcasts supports Basic and Digest Authorization, but doesn’t list password-protected shows.
For private or password-protected RSS feeds, avoid including personally identifiable information such as names or email addresses.

(end quote): whether this means there is any change to what I've said I don't know, but you could ask Podcasts Support. Go to https://itunespartner.apple.com/en/podcasts/overview and click the 'Contact Us' link at the bottom of the page.

Reply

Answer 3

Apr 14, 2020 9:10 AM in response to JP Redtopia

Then I'm afraid it's beyond my knowledge, except that media file URLs must end with the appropriate extension or they won't be recognized.

Reply

Answer 4

JP Redtopia Author

Level 1

17 points

Apr 14, 2020 9:07 AM in response to Roger Wilmut1

To be clear, no login is required to access any of the files. The signed URLs include validation and expiration information to let Cloudfront know whether or not to deliver the content.

For example, here is the enclosure to a 302 redirect link in a feed:

<enclosure url="https://www.musiclessons.com/feeds/media/D2ADB27E-2EB0-48DF-A50192CE92BA442B/" length="14639375" type="audio/mpeg"/>

You'll find that the URL redirects to a signed URL. That is all... just a simple mechanism to give a shelf life to links.

Reply

Answer 5

JP Redtopia Author

Level 1

17 points

Apr 14, 2020 9:56 AM in response to Roger Wilmut1

Thanks Roger... that may be true with the Apple Podcast Directory, though all feed validators and other podcast apps do not have a problem with it. It would be helpful to find documentation on this, or another way to validate a media file URL. It would also be helpful to know exactly how the Apple Podcast directory caches feeds and media URLs.

Reply

Answer 6

Apr 14, 2020 10:22 AM in response to JP Redtopia

I can't point you at specific documentation but it's well established that you must have the media extension. I've seen this problem, and it's solved by providing a URL which does have the extension.

As to the caching, iTunes/Podcasts caches the feeds and the main image, but not the media files. Since the feed contains the URLs of the media files these are also cached.

URLs which redirect to another one can cause problems sometimes, though they can work. People often use them to get usage statistics, as with Google, and I've seen that cause problems. Google Drive does not work witeh podcast media files though of course that isn't your issue): but I don't know how S3 works in this case: I've only used it for simple storage. The media files must be stored on a server which accepts byte-range requests (where only part of the file is requested at any one time) and must also accept head requests.

I tested the URL you gave above for byte range requests in Terminal and the result seemed not to suggest that this was available. This is the actual result, for what it's worth:

curl -I -r 200-300 https://www.musiclessons.com/feeds/media/D2ADB27E-2EB0-48DF-A50192CE92BA442B/
HTTP/1.1 302 Found
Date: Tue, 14 Apr 2020 17:17:21 GMT
Content-Type: text/html;charset=UTF-8
Content-Length: 98
Connection: keep-alive
Set-Cookie: AWSALB=Q0CfqjynZvX2sj1MAxMcdIetIDUiNm5sI5BgbS2KQyD9I1VLNST6PkQCRLtNo/gA/WNldqxgwpinsPoMkTAJ716sZnmDuqeYEmwGTO2RDyyr6vdxzTO3zYpw27nW; Expires=Tue, 21 Apr 2020 17:17:21 GMT; Path=/
Set-Cookie: AWSALBCORS=Q0CfqjynZvX2sj1MAxMcdIetIDUiNm5sI5BgbS2KQyD9I1VLNST6PkQCRLtNo/gA/WNldqxgwpinsPoMkTAJ716sZnmDuqeYEmwGTO2RDyyr6vdxzTO3zYpw27nW; Expires=Tue, 21 Apr 2020 17:17:21 GMT; Path=/; SameSite=None; Secure
Cache-Control: private
Location: https://cdnconf.musiclessons.com/131/D2ADB27E-2EB0-48DF-A50192CE92BA442B/D2ADB27E-2EB0-48DF-A50192CE92BA442B_v0004_lq_48k.mp3?Expires=1586888241&Signature=diA%7Ek6I-wemmP%7E8plTISW7L4HBdBT49GRBw52ph08sTwaOF-sjw2Udgg5-zVX9ldB1wzvdoz5CA7fCxpMhDNjq5c9tFXgsLkf8R4Wxnt1wCPXlYXjgfCmGIu1h1FKeetISd7493jV7FTmFgcbOqTHXYrabGsbO64%7Ef38%7EPv5QG-pN8sAJWTrSr1UKQALOkQOKZSxcZXzCsxNrY9Aybhda5HwNt27W3BDpVQjGpb3MYOOYcuhoZqvxcECgBmOLhkGe682TbcLRcCTQP1ipLT3e%7EFLYVIngpSjsODr8udCQnsZr0xdw7JRoiXHFqmvdUJM1UnGXCACK2sVA0JvQ2zwbw__&Key-Pair-Id=APKAIPP73A5Y5Y474ANQ
Server: Microsoft-IIS/10.0
Set-Cookie: cfid=4c71d17d-bfb7-4d11-b6ac-765cc2ba2727;Path=/;Expires=Mon, 04-May-2020 18:55:25 UTC;HTTPOnly
Set-Cookie: cftoken=0;Path=/;Expires=Mon, 04-May-2020 18:55:25 UTC;HTTPOnly
Set-Cookie: RCMSAPP=;Path=/;Expires=Tue, 14-Apr-2020 17:17:21 UTC
Set-Cookie: MLMBRRMBR=;Path=/;Expires=Tue, 14-Apr-2020 17:17:21 UTC
Set-Cookie: MLMBR=;Path=/;Expires=Tue, 14-Apr-2020 17:17:21 UTC
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET

This is what a succesful test looks like:

imac14:~ roger$ curl -I -r 200-300 https://rfwilmut.net/podcastsenh/enpodcast36.html
HTTP/1.1 206 Partial Content
Date: Tue, 14 Apr 2020 17:20:37 GMT
Server: Apache
Upgrade: h2,h2c
Connection: Upgrade
Last-Modified: Mon, 05 Nov 2018 16:36:50 GMT
ETag: "57a1fa7-11d5-579ed7de98a7b"
Accept-Ranges: bytes
Content-Length: 101
Vary: Accept-Encoding,User-Agent
Content-Range: bytes 200-300/4565
Content-Type: text/html

Note 'content-Length: 101' which is the key line.

Reply

Delivering media on S3/Cloudfront using signed URLs is not working

Similar questions