We received a report from the folks at This American Life of extremely high bills from their CDN for the month of October. It is our belief after researching the problem that this is caused by bugs in the iOS 6 Audio Playback frameworks resulting in files being downloaded multiple times – this could result in dramatic overage charges for both content distributers and data plan customers.
We had seen a pretty intense spike in traffic on 99% Invisible and The Moth (both of which we host) last month, and had a pretty good idea of the when the spike began as a result. At the time, we had chalked the rather extreme increase in bandwidth (seen below) to the release of Apple’s new Podcast app, which featured 99% Invisible prominently at release. We figured that Apple had brought 99% Invisible and The Moth some new subscribers, and were pretty happy once we had battened the hatches a bit.
But when we heard from This American Life that they were seeing an order of magnitude increase in their bandwidth usage, we needed to ensure that there wasn’t a problem with our apps that was causing unusual download behavior. Based on our research, it looks like the issue is iOS 6.
To begin, we wanted to know if there was a way that we could differentiate traffic originating from one of our apps from traffic originating from other apps. Because we are using the AV Foundation framework, it turned out that we couldn’t (the User Agent String is the OS Standard one, not app specific). We were able to see that the version of iOS was 6.0 but not the name of the app playing the audio. However, the Apache logs we looked at suggested something unusual. In the following screenshot, the file being downloaded is 8614457 bytes long.
What you can see is that the first 2 bytes of the file (in most cases, this will be ID, as in ID3) are downloaded in one request and then what appears to be the file being downloaded multiple times on iOS 6 and only once on iOS 5. (This appears to be an artifact of the way that Apache logs range requests, and we have reason to believe that the file was not downloaded many complete times, but there are still clearly problems.)
Following this, we set up a proxy so that we could watch requests as they were coming from the app. The player appears to get into a state where it makes multiple requests per second and closes them rapidly. Because the ranges of these requests seem to overlap and the requests themselves each carry some overhead, this causes a single download of an MP3 to use significantly more bandwidth than in iOS 5. In one case, the playback of a single 30MB episode caused the transfer of over 100MB of data.
We saw this behavior start after a period of behaving correctly (in some cases behaving correctly for about 5 minutes before the issue appeared) in both our own apps and the Apple Podcast app. We were able to reproduce the issue with several podcasts in the Podcast app, including podcasts using Limelight and Akamai CDNs. We have been unable to reproduce the issue using iOS 5 or using iOS 6.0.1, but there are still many people using iOS 6.0.0. We believe that this issue, combined with the bug causing the phone to behave as though it is connected to WiFi even when it is not, could account for the significant data overages reported with the release of iOS 6.
The strangest bit of behavior happens when the ranges on these requests reach the end of the file. We were able to consistently see that when the file has completed downloading, it begins downloading again from the beginning of the file and continues for as long as one is streaming the file. This means that, for as long as one is listening to audio being streamed with iOS 6, it is using significant amounts of data. Watch the range headers in this video, which is monitoring the HTTP activity of the stock Podcast app (v1.1.2) on iOS 6.0.0 playing back an episode of This Week in Tech. The file finishes buffering and is completely downloaded at around 0:36.
There appears to be a system-wide problem with the AV Foundation framework in iOS 6.0.0, resulting in significantly higher data costs to iPhone users and internet distributors. Users who have not done so should immediately upgrade iOS 6.0.0 devices to iOS 6.0.1, which we can confirm appears to fix the issue on Wifi. While some carriers are offering concessions to customers who may have been affected by this problem, Apple does not appear to have acknowledged the specific issue. The release notes for iOS 6.0.1 mention a change related to Wifi (likely referring to the problem with devices that reported that they were connected to Wifi while connected to 3G and LTE networks), which may be related to the change which fixed this issue.
Our tests did not cover 3g or LTE data, as we relied on connecting to Wifi to perform them. Because of the server logs we have access to, it appears that this issue exists over mobile broadband as well.