Chris RhodeniOS 6.0 Causes CDN Overages

Chris Rhoden posted on Wednesday, November 14th, 2012 | iPad, iPhone, Mobile

We received a report from the folks at This American Life of extremely high bills from their CDN for the month of October. It is our belief after researching the problem that this is caused by bugs in the iOS 6 Audio Playback frameworks resulting in files being downloaded multiple times – this could result in dramatic overage charges for both content distributers and data plan customers.

Background

We had seen a pretty intense spike in traffic on 99% Invisible and The Moth (both of which we host) last month, and had a pretty good idea of the when the spike began as a result. At the time, we had chalked the rather extreme increase in bandwidth (seen below) to the release of Apple’s new Podcast app, which featured 99% Invisible prominently at release. We figured that Apple had brought 99% Invisible and The Moth some new subscribers, and were pretty happy once we had battened the hatches a bit.

But when we heard from This American Life that they were seeing an order of magnitude increase in their bandwidth usage, we needed to ensure that there wasn’t a problem with our apps that was causing unusual download behavior. Based on our research, it looks like the issue is iOS 6.

The Behavior

To begin, we wanted to know if there was a way that we could differentiate traffic originating from one of our apps from traffic originating from other apps. Because we are using the AV Foundation framework, it turned out that we couldn’t (the User Agent String is the OS Standard one, not app specific). We were able to see that the version of iOS was 6.0 but not the name of the app playing the audio. However, the Apache logs we looked at suggested something unusual. In the following screenshot, the file being downloaded is 8614457 bytes long.

Click to view full size.

What you can see is that the first 2 bytes of the file (in most cases, this will be ID, as in ID3) are downloaded in one request and then what appears to be the file being downloaded multiple times on iOS 6 and only once on iOS 5. (This appears to be an artifact of the way that Apache logs range requests, and we have reason to believe that the file was not downloaded many complete times, but there are still clearly problems.)

Following this, we set up a proxy so that we could watch requests as they were coming from the app. The player appears to get into a state where it makes multiple requests per second and closes them rapidly. Because the ranges of these requests seem to overlap and the requests themselves each carry some overhead, this causes a single download of an MP3 to use significantly more bandwidth than in iOS 5. In one case, the playback of a single 30MB episode caused the transfer of over 100MB of data.

We saw this behavior start after a period of behaving correctly (in some cases behaving correctly for about 5 minutes before the issue appeared) in both our own apps and the Apple Podcast app. We were able to reproduce the issue with several podcasts in the Podcast app, including podcasts using Limelight and Akamai CDNs. We have been unable to reproduce the issue using iOS 5 or using iOS 6.0.1, but there are still many people using iOS 6.0.0. We believe that this issue, combined with the bug causing the phone to behave as though it is connected to WiFi even when it is not, could account for the significant data overages reported with the release of iOS 6.

The strangest bit of behavior happens when the ranges on these requests reach the end of the file. We were able to consistently see that when the file has completed downloading, it begins downloading again from the beginning of the file and continues for as long as one is streaming the file. This means that, for as long as one is listening to audio being streamed with iOS 6, it is using significant amounts of data. Watch the range headers in this video, which is monitoring the HTTP activity of the stock Podcast app (v1.1.2) on iOS 6.0.0 playing back an episode of This Week in Tech. The file finishes buffering and is completely downloaded at around 0:36.

Conclusion

There appears to be a system-wide problem with the AV Foundation framework in iOS 6.0.0, resulting in significantly higher data costs to iPhone users and internet distributors. Users who have not done so should immediately upgrade iOS 6.0.0 devices to iOS 6.0.1, which we can confirm appears to fix the issue on Wifi. While some carriers are offering concessions to customers who may have been affected by this problem, Apple does not appear to have acknowledged the specific issue. The release notes for iOS 6.0.1 mention a change related to Wifi (likely referring to the problem with devices that reported that they were connected to Wifi while connected to 3G and LTE networks), which may be related to the change which fixed this issue.

Caveats

Our tests did not cover 3g or LTE data, as we relied on connecting to Wifi to perform them. Because of the server logs we have access to, it appears that this issue exists over mobile broadband as well.

26 Comments to iOS 6.0 Causes CDN Overages

Roman Mars
November 14, 2012

Despite all this “research” I stand by my assertion that the episode in question had 373,000 individual downloads from hip, beautiful women with cool eyeglasses.

VPHedderel
November 14, 2012

May I blame the Podcast app directly? I very much dislike it divorced from the iOS Music app and I think it’s the best example of the failure of skeuomorphic design.

Ken Adams
November 14, 2012

have have see the same problem on our servers. We have our own CDN network and at one stage some of our servers have crashed due the shear amount of request. The has been no word form Apple in regards to this issue. Nor there has been any details for the customers that have been affected and getting huge bills.

So its about this time that news will finally will hit the pipes and perhaps some one from apple will come up with explanation how this could have happened and not fixed for so long.

Bill Gearhiser
November 14, 2012

Based on my experience in the computer bidness and my experience with Stitcher on my iPhone, I’d be willing to posit that what you’re seeing is a bug in the Stitcher software. I see Stitcher reset itself in the middle of playing podcasts numerous times a day. Many of those occasions seem to be for the purpose of updating the current podcast list, but many of THOSE include resetting my current podcast to the beginning. Who knows? It might be re-downloading at that time.

Marckus Anderson
November 14, 2012

Another sysadmin here. With our podcasts we used to be able to handle our clients’ requests with a simple 32 bit webserver. Once IOS 6 (first the developer beta, then public stable) started to hit our server it choked. Too many requests from these clients, effectively ddossing our server.

Since that time I have done a whole lot to try and mitigate these issues, investigating what was going on. It is exactly the same as is reported in this article that does a great job in describing it. Well done!

So I tested many different webservers. I don’t know why Apache logs 88MB, I think it is because Apache logs the _requested_ accumulated byte range rather than the actually _transferred_ bytecount. When you try nginx, varnish, thttpd, lighttpd and so on they all log different requests. Also, there is a difference with Linux and FreeBSD where the latter logs the minimum used buffer, which I think normally is 8 kbyte.

So while investigating I became none the wiser on how to mitigate the issue for IOS6 devices. I’ve even come to believe that there is no way at all, at the sending side, to resolve it.

Luckily I did test so many servers that I am at least able to say that lighttpd performed much better than all other webservers. Especially with fam/gamin, which tells lighttpd whether a file has been changed or not. Disk IO went way down while throughput went way up. For all devices except IOS6. :-)

Great article, well written, nice screenshots. Much regards to you, Chris.

Marckus Anderson
November 14, 2012

Two additional comments: when I said “When you try nginx, varnish, thttpd, lighttpd and so on they all log different requests,” I meant different transferred bytecounts, the number right of the 206 response code. With apache the number is quite high. However, lighttpd for example logs 64k and if I remember well, nginx logs 8k.

Secondly; lighttpd performed well indeed, I forgot to say that we serve _only_ static mp3 files of approx. 100MB and no dynamic content, e.g. PHP sites, with this server.

Anonymous
November 14, 2012
chris
November 14, 2012

@Anon – Thanks for the reminder! I filed a bug report but suspect it’s of dubious value since, as mentioned in the article, it’s been resolved in the most recent version of iOS.

chris
November 14, 2012

@Marckus – Yeah I mentioned briefly in the article above that we don’t think that the complete file is being downloaded each time, and that byte value is essentially unhelpful.

chris
November 14, 2012

@Bill – As far as we could tell, because the most recent version of Stitcher appears to be using their own Audio Playback utilities, they were unaffected by this. It’s possible that we would have been able to reproduce if we waited long enough, but I suspect that they are immune from this particular issue.

As you point out, they are likely susceptible to their own issues as a result of this strategy.

chris
November 14, 2012

@Ken – I think many people are getting their bills and scratching their heads around now, since we posted we have heard many similar stories.

chris
November 14, 2012

@VPHedderel – As I mentioned above, this affects all apps using the Apple suggested APIs (including the Podcast app, and many of ours). You may feel free to blame whatever you would like, but the purpose of this post is not to issue blame but diagnose a problem.

Giuseppe Taibi
November 14, 2012

Thank you for bringing visibility to this HUGE problem. This bug is totally real and has already cost me over $45 in overage charges. I know of tons of people, especially in the AT&T network that are reporting unusual battery drains and overheating after upgrading to iOS6. As an Apple developer I have filed a bug but Apple is ignoring it as they very often unfortunately do. This is class action material.

chris
November 14, 2012

@Giuseppe – I want to be clear, there’s no reason to think that this is the sole cause of the bandwidth issues. This, coupled with issues already covered elsewhere, could result in overages. I think it is likely that this played a part, and I think it’s all but certain that for the podcast hosts who have been reporting huge traffic spikes it is also likely that this is at least one of the major contributors.

[...] Discovered by Public Radio Exchange Labs, which has been carrying research on the issue, noted: Because the ranges of these requests seem to overlap and the requests themselves each carry some overhead, this causes a single download of an MP3 to use significantly more bandwidth than in iOS 5. In one case, the playback of a single 30MB episode caused the transfer of over 100MB of data. [...]

DS
November 14, 2012

Hi Chris,

My company hosts tons of media-rich sites on Amazon S3. Our bill jumped from an average of $700 a month to $941 in September, then up to $1900 in October. We’re on pace to hit about $2000 this month. I analyzed our logs and saw a massive jump on 9/25. The graphs you posted above look like they spiked on 9/20. Ours didn’t spike until 9/25 because our clients are churches, and most don’t post their sermon mp3s online until Monday (9/24) or Tuesday (9/25). So most listeners to their podcasts started getting the episodes on 9/25. Hence, our massive spike a few days after yours. Here is our traffic from Aug 1 until Nov 13: http://cl.ly/image/3p160v2h1n26. To get a full historical view, check it out from Jan of last year: http://cl.ly/image/2P1y1e150f1N.

I know it’s not Amazon’s issue, but I’m hoping they’ll hear me out and give us a credit. This extra amount is killer.

I really appreciate you taking the time to research and document this. You saved us a ton of hassle trying to sort this out.

[...] } .pw-widget-inline .ra1-pw-native-twitter { width:80px; } Public Radio Exchange has found and documented a bug in Apple’s iOS 6.0 mobile operating system that apparently causes multiple downloads of podcasts, boosting data usage [...]

What Haveyou
November 15, 2012

What is “CDN”? You use the term repeatedly but never define it.

Mike Adler
November 15, 2012

At WFMU, we also experienced related issues with the large volume of byte range requests from iOS 6.0. Thanks for the detailed write up.

[...] tech team is in the news today for their discovery of an iPhone/iPad iOS 6 bug that is causing large overage charges for podcasters like This American Life and probably for their [...]

JoelSu
November 15, 2012

@What Haveyou
CDN = Content Delivery Network. These are the companies like Akamai and Limelight that stream the content to a listener’s device.

Traced
November 15, 2012

This is specific to iOS6 for audio, but in iOS5 it already existed for video. Same thing, 1 iPad takes enough resources for a whole server.
I’ve filed a bug report for it ages ago and posted about it at apple dev forums, but nobody bothered.

flohei
November 16, 2012

Thanks for the great article.

I’ve been experiencing a similar problem (if not the exact same one) this month, though I’m running 6.0.1. So I wonder if it’s a different problem or if I somehow magically reproduced the iOS 6.0 bug on 6.0.1.

For years now, I have never exceeded my plan limit of 1 GB (except for once, when I moved and had no internet at home for 10 days or so). This month, I did already pay for four additional gigabytes.

Was there anybody contacting you having this issue on 6.0.1?

—f

Alex
November 16, 2012

Is this only audio streams bug? After I watched last Apple’s presentation on my iPhone I also received huge bill. I downloaded more than 16GB, though this presentation on YouTube in 1080p resolution have a bit more than 1 GB size.

Derek
November 18, 2012

Glad to see other people are trying to get to the bottom of this as well!

Chris is right in saying the specific problem discussed in this post is not the sole cause of the data spikes for some customers, but it seems to be one of many that all stem from iOS6. The biggest problem seems to be the leakage of mobile data when connected to wi-fi, at least from the end-user’s perspective. If the download loop was happening on wi-fi, only CDNs would notice and care about the extra bandwidth, but because its also using mobile data, customers are seeing the proof in the $$, too.

Even after the release of 6.0.1 and carrier settings update,I’m still experiencing significant mobile data use when connected to wi-fi. Just this morning (Nov 18), there was 50MB of mobile data use while I was at home, on wi-fi.

My network provider (Fido Mobile, in Canada) has completely dropped the ball on this one. Their tech support seems to be completely unaware of the issue, yet there website forums contain official company posts that acknowledge they’ve looked into the matter, and conclude that “THERE IS NO ISSUE.” I was charged $180 in extra data fees for Oct.

Bob
December 19, 2012

Anyone still having this problem after the release of 6.0.1? I know of one podcast producer that is having similar issues even when 6.0.1 users are trying to access content.

Leave a comment

Support Us!

PRX

Categories & projects

Archives