Chris RhodeniOS 6.0 Causes CDN Overages

Chris Rhoden posted on Wednesday, November 14th, 2012 | iPad, iPhone, Mobile | 26 Comments

We received a report from the folks at This American Life of extremely high bills from their CDN for the month of October. It is our belief after researching the problem that this is caused by bugs in the iOS 6 Audio Playback frameworks resulting in files being downloaded multiple times – this could result in dramatic overage charges for both content distributers and data plan customers.


We had seen a pretty intense spike in traffic on 99% Invisible and The Moth (both of which we host) last month, and had a pretty good idea of the when the spike began as a result. At the time, we had chalked the rather extreme increase in bandwidth (seen below) to the release of Apple’s new Podcast app, which featured 99% Invisible prominently at release. We figured that Apple had brought 99% Invisible and The Moth some new subscribers, and were pretty happy once we had battened the hatches a bit.

But when we heard from This American Life that they were seeing an order of magnitude increase in their bandwidth usage, we needed to ensure that there wasn’t a problem with our apps that was causing unusual download behavior. Based on our research, it looks like the issue is iOS 6.

The Behavior

To begin, we wanted to know if there was a way that we could differentiate traffic originating from one of our apps from traffic originating from other apps. Because we are using the AV Foundation framework, it turned out that we couldn’t (the User Agent String is the OS Standard one, not app specific). We were able to see that the version of iOS was 6.0 but not the name of the app playing the audio. However, the Apache logs we looked at suggested something unusual. In the following screenshot, the file being downloaded is 8614457 bytes long.

Click to view full size.

What you can see is that the first 2 bytes of the file (in most cases, this will be ID, as in ID3) are downloaded in one request and then what appears to be the file being downloaded multiple times on iOS 6 and only once on iOS 5. (This appears to be an artifact of the way that Apache logs range requests, and we have reason to believe that the file was not downloaded many complete times, but there are still clearly problems.)

Following this, we set up a proxy so that we could watch requests as they were coming from the app. The player appears to get into a state where it makes multiple requests per second and closes them rapidly. Because the ranges of these requests seem to overlap and the requests themselves each carry some overhead, this causes a single download of an MP3 to use significantly more bandwidth than in iOS 5. In one case, the playback of a single 30MB episode caused the transfer of over 100MB of data.

We saw this behavior start after a period of behaving correctly (in some cases behaving correctly for about 5 minutes before the issue appeared) in both our own apps and the Apple Podcast app. We were able to reproduce the issue with several podcasts in the Podcast app, including podcasts using Limelight and Akamai CDNs. We have been unable to reproduce the issue using iOS 5 or using iOS 6.0.1, but there are still many people using iOS 6.0.0. We believe that this issue, combined with the bug causing the phone to behave as though it is connected to WiFi even when it is not, could account for the significant data overages reported with the release of iOS 6.

The strangest bit of behavior happens when the ranges on these requests reach the end of the file. We were able to consistently see that when the file has completed downloading, it begins downloading again from the beginning of the file and continues for as long as one is streaming the file. This means that, for as long as one is listening to audio being streamed with iOS 6, it is using significant amounts of data. Watch the range headers in this video, which is monitoring the HTTP activity of the stock Podcast app (v1.1.2) on iOS 6.0.0 playing back an episode of This Week in Tech. The file finishes buffering and is completely downloaded at around 0:36.


There appears to be a system-wide problem with the AV Foundation framework in iOS 6.0.0, resulting in significantly higher data costs to iPhone users and internet distributors. Users who have not done so should immediately upgrade iOS 6.0.0 devices to iOS 6.0.1, which we can confirm appears to fix the issue on Wifi. While some carriers are offering concessions to customers who may have been affected by this problem, Apple does not appear to have acknowledged the specific issue. The release notes for iOS 6.0.1 mention a change related to Wifi (likely referring to the problem with devices that reported that they were connected to Wifi while connected to 3G and LTE networks), which may be related to the change which fixed this issue.


Our tests did not cover 3g or LTE data, as we relied on connecting to Wifi to perform them. Because of the server logs we have access to, it appears that this issue exists over mobile broadband as well.

Daniel GrossExploring The Depths Of KCRW Music Mine: Part One

Daniel Gross posted on Monday, September 19th, 2011 | iPad, KCRW, Mobile | No Comments

KCRW Music Mine launched recently to rave reviews. The free iPad app is the result of a collaboration between KCRWRoundArch and PRX, which led the project and development efforts. Music Mine pushes the limits of the tablet platform to create a truly unique music discovery experience. Daniel Gross interviews Matt MacDonald, PRX’s Director of Project Management, about the app.

What can Music Mine do?

The app is a iPad music player that opens up to a grid of 100 songs from artists that played on KCRW shows in the last week. What the app does is highlight these tracks that have been hand-picked by a KCRW DJ who wanted you to hear it. You might find odd pairings, maybe TV on the Radio right next to the newest Jeff Bridges country song. What they have in common is that these are all great songs that were picked by humans with taste and a point of view, not selected by an algorithm. The app lets you explore each of these artists in greater depth and we use software to add supporting songs, videos, photos, bios and blog posts.

How did it all get started?

We started talking with KCRW in the summer of 2010 about building an iPad app. PRX has been doing iOS development since late 2008 with apps including the Public Radio Player for iPhone and This American Life app for iPhone, iPad and Android. So we knew we could make this project ambitious.

As a public radio station, KCRW produces a ton of content. But we wanted to build an app tightly focused on their music. Anil Dewan at KCRW helped us understand their online music identity. But given the existing competition facing KCRW and listening platforms, we were asking: how the heck do we stand out and differentiate ourselves?

And? What makes Music Mine so different from other apps?

The biggest difference is Music Mine’s human touch. Other music apps like Pandora, Discover, Spotify or Last.fm are designed to provide access to every song and every artist in their huge catalogs. When you’re listening to Wilco, say, those apps use an algorithm to automatically recommend the next 5 or 6 ‘related’ artists. KCRW has a huge and diverse catalog but we wanted to amplify and highlight the work that the DJs do by narrowing what you should be listening to.

The user interactions in the app have an attitude and a point of view like the KCRW DJs. One goal for us was to encourage, musical exploration and delight so some of our interaction decisions basically force you to try out music that you might not be familiar with. We made sure to encourage that by not adding features like search or sorting and filtering tools. We also could have directly presented the underlying ordered list that backs the grid layout but we felt that doing so would have made you less likely to try out new songs and that you might only search and tap those you know.

So you want an app with people behind it – but you still want to take advantage of technology, right? What do you need to do that?

Yep, we definitely want to use technology to help us out here. Our office is in Harvard Square, and the local tech scene here is pretty amazing. One of the companies we’d wanted to work with for a long time was The Echo Nest over in Somerville. When we started dreaming up the Music Mine idea we went straight to them.

The Echo Nest provides APIs that allow us to request a ton of information about artists and songs, found audio, video, relevant blog posts, bios and photos. We knew that merging the KCRW playlists with Echo Nest data sources would give us the ability to create a unique window into the KCRW music universe. Rebecca Nesson here at PRX led the effort to merge the data sources from KCRW and Echo Nest. While each of the songs available in the grid is initially picked by a human, PRX wrote software to help narrow down the thousands of tracks that could be available to the 100 available on the grid.

How did the project evolve as you built and rebuilt the product?

A big question early in the project was: should the primary experience for how people listen in the app be focused on the long, multi-hour DJ sets, or center on playback of individual tracks? PRX and RoundArch interviewed people specifically about what they might want in a music discovery app from KCRW and after hearing from them we decided to focus the experience on track-by-track selection. That decision proved challenging and mid-way through the project there was a point where we really had to consider ditching the track-by-track access and move back to a DJ set focused app.

The sub-views in the app like the artist photo, song, video, bio, blog pages came together pretty quickly and didn’t change much over the project but the home view, the grid, was definitely an area where we iterated on ideas for a long time. Yeah — that took a really long time. We might have had 8 or 10 significant tweaks to the layout and presentation. But I think all the time we spent iterating on the ideas really paid off in the execution.

Alternate DJ set focused home view

Early DJ focused home view

Initial grid comp

Final grid design.

How do you feel about it? Is there anything you’d improve or add?

Well…no. Actually, I’m really happy with it. I just re-read the original product vision and I think we’ve really held true to that over the last ten or so months.

The KCRW iPad Experience project will focus on creating an iPad experience that quickly engages target users by leveraging KCRW’s unique style, taste and hand picked music. In addition, tight conceptual focus and appealing interaction models will aid in driving long-term user engagement.

People that we interviewed about what they might want in an music discovery app said they wanted us to make good music important to them, help them find new good music and improve their awareness. We’ve done that. People wanted us to connect them with great new artists, and help them learn more about music and artists they’re already familiar with. I think we’ve done that too.

What’s next from the PRX team?

I’m really excited about an iPhone project that we’re working on with a very popular public radio show. I feel like we’re developing a unique way to help people interact and connect with the show. We’re pushing the device’s capabilities, coming up with new interactions that can surprise and delight people.

Any last thoughts?

People talk about software as engineering, and it’s partially that, but it’s also a craft. This kind of work is like building a beautiful piece of furniture. When it’s done poorly, you end up with a cheap nothing that you throw away. But when it’s done well, you’ll never see all the work that went into it. Sometimes that means a lack of appreciation for the thought and care that was put in.

Devin Chalmers, Rebecca Nesson and Andrew Kuklewicz are top-notch developers, maybe the best people I’ve ever worked with. But even with the best people working together, it requires time and freedom to explore and do it right. We could have finished a long time ago if we’d done it poorly. Instead, we came up with something that we’re proud of, KCRW is happy with, and it seems so are the people using it.

Part Two

Coming up: Rebecca Nesson and Devin Chalmers will be digging into other more technical aspects of the KCRW Music Mine app, perspective transformations on scroll views, knob-twiddly stuff to tweak behavior and how we had to figure out layouts of a 4×6 grid in both portrait and landscape modes.

Andrew KuklewicziPhone Radio Apps Are Not Banned

Andrew Kuklewicz posted on Saturday, November 27th, 2010 | iPad, iPhone, Mobile | 8 Comments

UPDATE 11/28 4:45 PM:
Lance Venta of Radio Insight is following up on the possibility that AirKast may also have been banned as well. In the AirKast store where 277 apps are listed, the latest new release is dated 10/30. (Coincidentally, their last app was for ‘KNUE’, which is a station I grew up listening to in Tyler, TX.)

This would be a big deal, both because they have so many apps, but also because they have significant commercial partnerships (e.g. ESPN, Triton, Radio One which “primarily targets African-American and urban consumers” in 15 markets with 53 stations, Salem Communications “targeting audiences interested in Christian and family-themed content and conservative values” with 94 stations, Bonneville International with 26 stations, Citadel Broadcasting comprised of “165 FM stations and 58 AM stations”, & NextMedia with 33 stations.

I’ve started compiling a list of other providers possibly hit by this when I started this post; time to go back and see who else might have been affected by the policy. Statements from Apple specify that apps in a station’s own store should continue to be approved, but it will be interesting to watch the white-label app sellers’ stores, and see if they can add more station apps.

For example, Jacobs Media (with 249 apps in its store) has not had a new app in their own store since 11/10 (though as Lance pointed out to me, they have had updates as recently as 11/24). They have probably taken to putting all new apps in individual station stores in accordance with Apple’s policies (which they were previously doing with some station apps anyway).

For customers of white label app providers, in effect the price just went up by $100/year, as they now have to buy their own iOS Developer Account rather then riding on the app creator’s. For DJB Apps, with a price of $199.99 (1 platform, 1 station app), adding another $100 for Apple is not a minor difference.

Original Article:
We’ve been asked by a number of partners and acquaintances to weigh in on the recent controversy started by Jim Barcus’s letter to the editor in Radio Magazine.

The jist of the letter it that Apple rejected 10 of his iPhone apps because they no longer accept single stream/station apps:

On Nov. 10, 2010, we had 10 radio station apps rejected by Apple because Apple says “single station app are the same as a FART app and represent spam in the iTunes store” and Apple “will no longer approve any more radio station apps unless there are hundreds of stations on the same app.”

He does not identify a particular source at Apple (few can or do), or cite an official notification from them (such as the reason(s) that would be included in a rejection), but adds:

We have talked to many Apple reps about this, but they appear to have a script that they all read from saying that a single station app is not an enriching end user experience.

While this particular letter has engendered remarkable response (as it seems intended to do), the tale of a developer rejected is a common one. In fact, Brian Stormont had a similar odd rejection back in April (emphasis is mine):

Today I received the oddest rejection from Apple for a pair of iPhone apps I had submitted for two different radio stations. Apple rejected the apps with the explanation that they were of a similar “Family of radio applications” and “should be packaged in a single application using In App Purchase”.

I replied to Apple explaining how combining the two apps is not an option, but I haven’t yet heard back. I’m hopeful this isn’t the start of a new policy on Apple’s part prohibiting apps for individual radio stations.

UPDATE 4/13/2010: The two apps have now been approved, although I never received any clarification from Apple regarding my inquiry. The apps just suddenly became approved. Best I can assume is the original reviewer had made a mistake.

A point of interest is that this precedes Apple’s 9/9/2010 statement which accompanied the first set of official and public approval guidelines for the app store. This document makes clear that “We have over 250,000 apps in the App Store. We don’t need any more Fart apps. If your app doesn’t do something useful or provide some form of lasting entertainment, it may not be accepted.”, and shortly thereafter a point that perhaps Barcus missed, “If your app is rejected, we have a Review Board that you can appeal to. If you run to the press and trash us, it never helps.”

As with Barcus, Brian Stormont expressed (more skeptically) the fear that Apple is trying to cut down on the sheer number of radio apps, and instead focusing on aggregation apps that play an uncounted number of stations, as iTunes does for streams on OS X, or an actual radio does for broadcast.

If this were to happen, and Apple were to enforce a no single station apps policy, it would be immensely troubling to say the least.

The thing is, you shouldn’t believe it.

The app store right now is flush with single station apps, including many featured in ‘What’s Hot’ (such as ‘WMBR’ and ‘RadioU’), and ‘New and Noteworthy’ (‘TuneLab Radio’ appears as #7).
Surely if this were truly a blanket policy they would be rejecting these apps, not featuring them in such coveted spots.

There must be more to this, some of which came out when Trevor Long of your tech life also spoke with Apple:

I’ve spoken with Apple, as have several developers and the situation, while not crystal clear, is certainly very clear on principal. There is no ‘ban’ on Single station Apps.

My own summary is that App developers submitting identical apps with just a logo/stream change under their own developer accounts are not looked well upon. (and if you take the time to download a few of this chap’s apps, you’ll see how uninspiring they are, and similar too)

However, if a station has an app developed, and submitted under their own name, it should breeze through.

Likewise, if an individual or group create an app that links to a single station stream or show, it might face some hurdles.

From a station owner perspective I think that’s a good thing – but, from Apples point of view, good on them for ‘encouraging’ the development of rich apps with a user experience that is more than ‘just a stream’ – in the end, if it was ‘just a stream’ the industry won’t survive!

Moral of this story for Radio stations – be creative, own the rights, own the application submission.

Confirming this account and adding further detail is a translation of another conversation with Apple representatives:

Translation from German article in Radioszene.de:


Faster than expected, Apple has reacted in a telephone call from german radio organization VPRT, This is a summary of the call:

“Apple will not delete any radio apps or prohibit access of radio apps to Apple Store. Apple suggests that every radio station makes their own developer account (79 Euro) and uploads their apps in there themselves. The apps can be developed by third parties (White label solutions). New and the apps in the future will be handled as so far. Apple confirmed, that several identical apps -which are provided from ONE developer account- will no more be accepted. (Comment: there has been some cases in the near past where a developer has submited hundred identical apps into the store). This rule is also applying to newspapers and other industries. If the app will be uploaded by the radio station this is no problem. Radio stations can also upload more than one app on their own developer account”.

Consistent with these posts, other radio app creators as or more prolific than Barcus have made it known they are not experiencing systematic rejections, nor are aware of any policy changes:

Claiming that Barcus is wrong, Paul Jacobs, VP/GM Jacobs Media/jacAPPS, tells RBR-TVBR that he’s been able to build radio station apps since Nov. 10: “Since that date, we have had apps for radio stations accepted as new as well as upgrades. And we are not alone. I invite you to go to the iTunes App Store and go into the “free” apps portion of the Music section. There you will see examples of dozens of radio stations – domestic, international, and Internet – that have been accepted and/or upgraded since November 10.”

Battle of the screen shots?
Jacobs even sent us a screen shot proving it. There were at least five radio stations — both AM and FM –with apps listed post-Nov. 10. Now, not to make any rash judgments here, but Barcus also sent RBR-TVBR a similar image, indicating the opposite was true.

Neither screen shots has been made public, but I have found results consistent with Paul Jacobs claim in my own perusal of the app store (I’ll leave this to you for homework).

UPDATE 11/27 12:05 PM: RBR.com has posted the screen shots. The one of the entire app store, sent by Jacobs, clearly shows many new radio apps. Barcus sent in list of apps by Jacobs Media (not what Jacobs invited users to look at), who has not published a new app since Nov 10. I have seen many Jacobs apps submitted in individual station accounts, so I tend to believe the Jacob’s claim that they have apps approved since the 10th, just not ones in their seller account.

Barcus’s letter seems aimed at causing uproar, positioning this as a conflict between Apple and the whole of radio. It now seems to be a more specific policy that has been applied to him (and perhaps other, less vocal, sellers).

It is unfortunate that other outlets, such as The Register and other sites added their voices to this uproar, and have yet to post corrections.

UPDATE 11/28: The Register and Martin Bryant from The Next Web have both updated their stories (thanks for the link Martin). The Register received a statement from Apple:

“There are many unique radio apps on the App Store and we look forward to approving many more,” reads a statement from an Apple spokesman sent to The Register. “One developer has attempted to spam the app store with hundreds of variations of essentially the same radio app and that is against our guidelines.”

This makes the 3rd communication from Apple confirming that this is an issue regarding a single developer, and a particular type of activity that Apple has previously described as a reason for app rejection (i.e. ‘spamming’).

As others have found Trevor’s post, they have reported this as a developer’s sour grapes rather than a draconian policy against radio, as Lance Venta did in Radio Insight, or in the corrected piece by Steve Safran in Lost Remote. That’s a wonderful retraction Mr. Safran; gracefully done, and with good humor – it ought to increase your readership (I know it did by at least one).

As the app store has reached a population of hundreds of thousands, Apple has been raising the bar on approvals, and you should expect this to continue as even more flock to the growing iOS platform. The best way to get your own apps approved it to start by providing apps people actually want (rich functionality and crafted interfaces). From there, you need to consider many factors in getting your apps approved, not to mention adopted, but that is the subject of my next post…

Matt MacDonaldUseful OmniGraffle stencils and templates for iPhone and iPad

Matt MacDonald posted on Tuesday, October 19th, 2010 | iPad, iPhone, Mobile, Wireframes | 1 Comment

I’ve been using OmniGraffle Pro for most of my iPhone and iPad wireframe jobs. I love the product, simple and easy to use. One of the things that I dig the most about it is the vibrant stencil and template community at http://graffletopia.com/. There are a number of stencils and templates that you can find out on the internet. The two that I’ve found to be most useful over the past few months are:

They are well thought out and I’ve been able to find and modify just about any component I’ve needed.

Support Us!


Categories & projects