A couple of weeks ago, I participated in a hackfest which was thrown by the folks at Frontline. The goal was to spend a few hours putting together a glimpse of the future of documentary storytelling. Four projects came out of the day, but I would like to talk about the project I worked on.

As far as I could tell, I was the only developer resource on the team I was on, which resulted in a bit more focus on the technical aspects of things than is typical. As such, I can tell you that the documentary I worked with was:

1. Already slated to receive some level of interactive work.
2. About the abuses of our economic system by Wall Street.
3. A Frontline documentary.

There’s really not much else I could say. I did end up watching the trailer a bunch of times, but even that started blurring, as my primary goal was to get something that sort of worked ready for demo.

The point of all of this context is that it seems very clear to me that this could be used in a variety of situations – not just those where a documentary is concerned. Anyway, on to the demo.

The first interaction you will have with the system is a QR code, which you are invited to scan with your phone. This QR code points at a mobile-optimized website.

When the website loads in your mobile browser, the QR code disappears and a video begins playing on your computer. The video on the computer screen is deliberately uncluttered, as the point is to allow you to watch the video in a lean-back manner, with no need to interact with your computer.

In your mobile browser, you are given a big green button which also acts as an indicator for your progress through the film. Pressing the button causes a bookmark to be saved of where you are in the film, including context for the segment you were watching and, in some cases, additional content and research. During our demo, one segment of the film, when bookmarked, provided access to watch additional raw footage.

When the bookmark button is pressed, a small indication is shown on the main screen that the bookmark has been saved.

The goal for the project was to provide a way to augment a viewing experience while in no way compromising the narrative created by the filmmaker. There were a number of additional features which I would have loved to have implemented, such as controls for the onscreen content, and a dashboard showing your bookmarks on the main screen when the video has finished. That having been said, I believe that the goals for the project were shown off well, and am proud of what we walked out with.

Next time, I’ll go into the actual guts of how this system worked, and maybe some other uses for the technology.

SJ and Erhardt divided us into four groups and asked us to winnow the big list of ideas brainstormed the night before down to our top ten, adding any new ones we had on the way.  Once we’d done this we combined with another group and did another round of winnowing.  Then we joined all together and did two rounds of straw polls on the remaining projects to help us self-select into working teams.  I’m not sure I can recall every project, but here’s my best stab:

* Revising the Wikipedia page about OER.

* Designing a Webby- or Oscar-like awards contest for the best OER of the year.

* Designing software to help present a personal gallery/portfolio of OER accomplishments.

* Creating an online course to teach strategic thinking using Poker

* Using git as the basis for a collaborative OER repository

* Creating an open-source web application that works like Free Rice but can be implemented with local standardized test questions and local sponsorships

The open-source Free Rice project was an idea I contributed in the first round.  It’s an idea that my father and I have been interested in for years and it seemed like a good fit for the day.  I was happy to see it progress through the rounds of winnowing.  Ultimately Matthew Battles of Metalab and Jeff Mao of the Maine Department of Education chose to work on it with me.  We spent some time discussing how we could make the project into something that school systems would be able to use and decided that ideally the system would be able to hook into an API provided by the testing company with which a school system contracts.  While Matthew and Jeff worked on ideas about the API and a presentation of the idea, I started to hack.  The time for actual hacking was short: about two and a half hours.  So I was happy to have gotten enough up and running just to do a short demo.  You can see the results of that here:  http://freepencils.herokuapp.com.  Jeff and Matthew did an excellent presentation while I clicked around the app on the big screen behind them.  And lo and behold we were selected as winners of the day along with the Academy Awards for OER project.

The hack day organizers are working on a write-up of the day for the conference blog that will give more information about all the projects.  In preparation for that Jeff wrote up a short text version of our pitch.  Here it is:

Many states (48 including DC and 2 US Territories) have adopted the Common Core State Standards for English Language Arts and Mathematics. This is the first time in the history of US Public Education that we will see common learning standards across so many States. Part of this effort is the development of a shared assessment platform that seeks to provide an open source adaptive assessment system that will support formative and summative assessment for classroom, school, district, and state-level use.

Let’s talk a little history, shall we?

PRX makes mobile apps for public radio programs and stations. When we were asked to make an Android app for This American Life, we found that the Android ecosystem was just a little bit fractured. We built a very large and somewhat messy chunk of code to help us work through the issues of supporting 4 different major versions of an operating system, including handling weird and widely covered bugs and device/os interactions.

But no more! We found that by dropping support for the very very old versions of Android, we were able to lock into a much more stable API. There’s still a whole bunch of work that needs to be done in order to start playing audio in the background properly, though (think foreground notifications, the prepare/play api, and handling audio session changes). So I set to work building something completely from scratch which tackles these problems. We even thought long and hard about what should happen if your audio is interrupted with a phone call (we start back up again when you hang up if the call took less than 5 minutes. Otherwise, we kill the session.)

There are a whole bunch of goals for the player moving forward, including a default player widget and notification with play/pause buttons where they’re supported. For now, we hope that the dramatically simplified API and sensible default behavior will be useful to some people, and we can gain enough traction to make PlayerHater the de-facto way to play background audio on Android.

Check out PlayerHater on GitHub and let us know what you think!

There are several fairly robust open source projects out there to help with the switch. MediaElement.js and jPlayer were the two main contenders, with jPlayer winning out do to internal familiarity. Both work by having a Flash fallback option when the native HTML5/JavaScript handling in a given browser can support the file. jPlayer allows you to define multiple source files to help with browser coverage. Currently our catalog is mp3, which IE8 (and lower), Firefox, and Opera do not support natively. We would need to re-encode our entire catalog to ogg or WebM to provide native support there, which may happen at some point, but for now relying on the Flash fallback in jPlayer is a perfectly acceptable solution.

One major advantage of having a single player UI and having it built in HTML that will be consistent across all platforms and browsers regardless of what’s handling the audio is extensibility. Whereas previously we had slightly different Flash widgets for the main player, the popup window player, and the embeddable player, we now need to only maintain a single HTML file. The player is fluid, so it will resize to fit whatever space it is given. Adding additional features or elements to the player down the road will be an extremely simple process.

The move hasn’t been without issue. jPlayer has a few shortcomings that we’ve had to work around. The most significant being an issue with the way in which the Flash player reports file duration when the file is loading. It often fails to accurately estimate the total duration before the entire file loads. This can cause some unexpected behavior with large files. It was fairly trivial to work around, but it shows that supporting two vastly different technologies at the same time is not an exact science. There are a handful of finicky UI limitations with jPlayer as well. Want a vertical volume bar? Get ready to fork.

Performance as a whole, though, has been good. Even on some of our playlist pages which load up dozens of individual players, results from our testing has on par with or better than FlowPlayer.

We’re very excited about the prospects of having this new player out in the wild. Aside from the obvious benefits of iOS support, and the boost in street cred for having using more HTML5, there are likely some advantages we haven’t even considered yet. PRX hopes to help drive innovation in this space, so expect to see any improvements we make flow back into the community.

Old vs New:

Demo

AndEngine is a free 2D game engine for Android.  It allows the creation of a game activity that I could combine with other Android activities to present content.  (I use Android Fragments via the Android Support V4 library to incorporate traditional Android views into the game environment.)  Although AndEngine is intended for 2D games, a forum thread demonstrated how to do the same perspective trick to the camera we’re doing on the iOS side:

 private void setFrustum(GL10 pGL)
 {
    // set field of view to 60 degrees
   float fov_degrees = 60;
   float fov_radians = fov_degrees / 180 * (float)Math.PI;

   // set aspect ratio and distance of the screen
   float aspect = this.getWidth() / this.getHeight();
   float camZ = this.getHeight()/2 / (float)Math.tan(fov_radians/2);

   // set projection
   GLHelper.setProjectionIdentityMatrix(pGL);
   GLU.gluPerspective(pGL, fov_degrees, aspect, camZ/10, camZ*10);

   // set view
   GLU.gluLookAt(pGL, 0, 0, camZ, 0, 0, 0, 0, 1, 0); // move camera back
   pGL.glScalef(1,-1,1); // reverse y-axis
   pGL.glTranslatef(-CAMERA_WIDTH/2,-CAMERA_HEIGHT/2,0); // origin at top left
}

What’s happening here is that the camera is being pulled back away from the scene and a perspective transform is being applied that causes things in the distance to appear farther away.  I can’t explain it any better than the cryptic m34 transform that is applied to the camera on the iOS side, but the effect is the same.

The only other modification I had to make to AndEngine was to create a 3D sprite class that wraps the provided Sprite class and allows the user to set the z position of sprites as well as their x,y position.  In our app world the user doesn’t interact directly with the scene but rather with an scrolling mechanism that moves the scene “on rails” as the user scrolls.  The effect is beautiful but also somewhat hard to capture in screenshots.  You’ll just have to buy the app when it comes out!

The good news is, the app is shaping up beautifully and AndEngine has really come through for what we needed to do.  But there’s a big remaining issue that I’d like to solve.  AndEngine takes care of all of the touches on the scene and passes them to the sprites.  But it does it based on their x,y coordinates.  Unfortunately, the x,y coordinates it calculates based on the touches of the screen do not correspond to the location of the sprites within the scene because of the perspective transformation based on depth.  Under the covers OpenGL knows where the sprites are because it drew them correctly on the screen, but AndEngine itself does not know.  Additionally, I can only get access to a GL10 instance which does not provide the functions I need to project and unproject coordinates.  For now I’m working around this issue, but theoretically I should be able to do the math to convert 2D screen coordinates into 3D scene coordinates using the ratio of the scene size to the screen size, the position of the camera, the angle of view, and the distance of the object in question from the camera.  So far I haven’t succeeded in doing it, but when I get a few days to step back from the project I’ll turn to it again.  If you think you know how it should be done, please comment!

I wanted to make a quick plug for a session that I’m organizing at the 2012 Integrated Media Association conference next week. The session is titled: Pledge your support for mobile and it will be both an overview of current pledge/donation practices and also a hands on working session. We’ll review existing pledge and donation practices, what works and where people get hung up and then both the session panelists and attendees will attempt to streamline and significantly improve the pledge process for public radio and television stations.

The major themes for the IMA conference this year are:

1. Motivate attendees to take individual ownership for innovation
2. Provide the attendees with knowledge that they can take home and use immediately

I’m really excited about our session panelists and facilitators, I’ve worked with each of them in the past on mobile and tablet projects. Devin Chalmers (KCRW Music Mine iPad, Radiolab iPhone), Dan Saltzman from Effective UI (KCRW Music Mine iPad) and Marine Boudeau from New York Public Radio (WNYC, WQXR, Radiolab iPhone and Android). Each has worked outside of public media and has extensive experience in user centered design and interfaces.

I’m also excited to try out a hybrid session with both a conference panel and then a hands on hackathon approach. My hope is that we all learn some new things, then create something new and all leave with actionable steps. This will be a session where attendees will be encouraged to participate, one where we use our collective strengths to improve a process that is ripe for revamp. If you are attending IMA I’m hoping you can join us.

Our session is on Thursday March 8th starting at 2pm.

I’ve uploaded my in-progress slides for the session in an attempt to entice people to come checkout our session.

If you are too lazy for that link, just call: (877) 815-3013

How does it work? Here is the secret formula:

@benjamenwalker + 40 * ( twilio + heroku + three.js + jplayer) = happy birthday

I worked on this over the course of a day and a half; the tools are out there people to make incredible stuff, and fast, even if you are an aging coder like me. Go make something!

(I also got some design inspiration from @loveandradio, but I lined everything up on the other side of the screen, so it is completely different. Also, I’ve only seen Nick’s head spin around like that when I got a concussion.)

Love it or mock it, Facebook’s on to something with the latest beta features for Open Graph. I’m not saying they are the first to realize there is more to the world than liking, as scrobbling on last.fm has been going on for awhile, and we even worked with Doc Searls to do listen log, which has the added (and hopefully inevitable) twist that you own your own data about what you hear/read/like/hate. Imagine that, not just giving that kind of useful, private information away to some company? But I digress.

Many have seen the writing on the wall that personal analytics (i.e. my latest favorite is RunKeeper) is a natural outcome (or accomplice) of the big data and social networking trends, but Facebook is in a position to take it to 11. Not to mention they have real motivation, with the competition at google+ looking for a differentiator or twenty. I embrace personal analytics as providing a new perspective on one’s self, and a way to know each other better.

In public media, during pledge drives, stations often ask listeners to imagine how many hours they listen in a year, and what value that has to them. Why imagine or guess at that when we can actually answer the question with data?

“With ListenLog data, listeners can make fully informed choices about how they support streams, podcasts and on-demand sources of programming.”

…and that is every bit as true for what Facebook wants to start logging. I don’t know who’s version of media logging will ‘win’, and as we’ve learned from friendster and myspace, winning is not a permanent condition, but the concept is powerful, and the potential effect on our behavior profound. Or at least, I hope it is. I also hope to find a way to have my cake and eat it too; own my data, but share intentionally. Facebook may not be there yet (not extreme understatement), but it can’t be ignored in the meantime.

And now what?

For listeners to public radio, what we listen to is as much a part of our lives as the TV and movies we watch, the music we hear, and the books we read. No one is forcing anyone to log this stuff, but if someone should want to, be it for their own curiosity, what it says about them, or just amusement, we should make sharing this easy, and as complete as other commercial media. In fact, we should do better – folks should be proud to share public media, and lead others to that experience as well.

While I have focused so far on media logging, we have another action besides ‘listen’ that we want to encourage and shout from the pixelated rooftops: ‘support’. I want to see that early and often, whether it is supporting a broadcast, stream, program, episode, network, host or even a program that hasn’t been made yet.

Brass tacks

I can’t write a labs blog without getting to something more technical, they’ll take away my public keys.

Here’s some ideas I think come next, and I hope others in public media think and will do the same. Hopefully we can even agree and work together (it has been known to happen):

1. There are movie, video and TV action and object types already defined. I think these should be adopted by public/community TV, with perhaps some embroidery over time. As they say, “adopt and extend”, not analysis seems like the way to go.
2. Audio is lacking standard definition of simple actions (e.g. listen), though for video/TV there is a built-in ‘watch’ action defined that is a good basis. We also need to define action types for public media (e.g. ‘support’ leaps to mind).
3. We also need to define audio media objects for radio programs, episodes, stories, segments, streams, and podcasts. Probably others have started to do this as well in their facebook apps, but wouldn’t be nice if at least in public media we could agree? Audio is where we need to do work; news stories (e.g. Articles) and video/TV are pretty well covered in the built-in object types.
4. With these standard objects and actions, then we need to make this easy to log. That means adding this to however users listen. For PRX, that means integrating optional Facebook open graph listen logging to our station and program mobile apps and the Public Radio Player, and websites like prx.org.
5. We also need to make it easier for station websites to log this listening activity, just as they do facebook and other kinds of sharing now. Integrating listen logging into the media players via open source drupal and wordpress plugins seems to be the easiest way to do that, since between those two you seen a majority of new work, especially with the drupal work by NPR on Core Publisher, PBS on bento, APM using Drupal for the new Marketplace site, and PRI shows like To The Best Of Our Knowledge developing their new site on drupal with a talented team at Wisconsin Public Radio. And that’s just to name a few. Figure out how to integrate with these drupal distros, and you get a big bang for your buck.
6. IMA launched Public Media Metrics to combine traffic data from all public media websites. This worked because we all have learned to share, and when we all use Google Analytics, it is comparing apples to apples. If we can standardize on the actions and objects used in Facebook’s open graph, not only is it less confusing, but there is a similar opportunity for aggregating social media metrics.

Some commercial and start-up media companies already have a jump on us with integrating these facebook beta features, and that’s fine, but let’s not wait too long to catch up and exceed them. Our capacity or cooperation also means we have a unique chance to combine our efforts across public media for design, implementation and measurement. We have millions of listeners and viewers; we should let them prove it.

What can Music Mine do?

The app is a iPad music player that opens up to a grid of 100 songs from artists that played on KCRW shows in the last week. What the app does is highlight these tracks that have been hand-picked by a KCRW DJ who wanted you to hear it. You might find odd pairings, maybe TV on the Radio right next to the newest Jeff Bridges country song. What they have in common is that these are all great songs that were picked by humans with taste and a point of view, not selected by an algorithm. The app lets you explore each of these artists in greater depth and we use software to add supporting songs, videos, photos, bios and blog posts.

How did it all get started?

We started talking with KCRW in the summer of 2010 about building an iPad app. PRX has been doing iOS development since late 2008 with apps including the Public Radio Player for iPhone and This American Life app for iPhone, iPad and Android. So we knew we could make this project ambitious.

As a public radio station, KCRW produces a ton of content. But we wanted to build an app tightly focused on their music. Anil Dewan at KCRW helped us understand their online music identity. But given the existing competition facing KCRW and listening platforms, we were asking: how the heck do we stand out and differentiate ourselves?

And? What makes Music Mine so different from other apps?

The biggest difference is Music Mine’s human touch. Other music apps like Pandora, Discover, Spotify or Last.fm are designed to provide access to every song and every artist in their huge catalogs. When you’re listening to Wilco, say, those apps use an algorithm to automatically recommend the next 5 or 6 ‘related’ artists. KCRW has a huge and diverse catalog but we wanted to amplify and highlight the work that the DJs do by narrowing what you should be listening to.

The user interactions in the app have an attitude and a point of view like the KCRW DJs. One goal for us was to encourage, musical exploration and delight so some of our interaction decisions basically force you to try out music that you might not be familiar with. We made sure to encourage that by not adding features like search or sorting and filtering tools. We also could have directly presented the underlying ordered list that backs the grid layout but we felt that doing so would have made you less likely to try out new songs and that you might only search and tap those you know.

So you want an app with people behind it – but you still want to take advantage of technology, right? What do you need to do that?

Yep, we definitely want to use technology to help us out here. Our office is in Harvard Square, and the local tech scene here is pretty amazing. One of the companies we’d wanted to work with for a long time was The Echo Nest over in Somerville. When we started dreaming up the Music Mine idea we went straight to them.

The Echo Nest provides APIs that allow us to request a ton of information about artists and songs, found audio, video, relevant blog posts, bios and photos. We knew that merging the KCRW playlists with Echo Nest data sources would give us the ability to create a unique window into the KCRW music universe. Rebecca Nesson here at PRX led the effort to merge the data sources from KCRW and Echo Nest. While each of the songs available in the grid is initially picked by a human, PRX wrote software to help narrow down the thousands of tracks that could be available to the 100 available on the grid.

How did the project evolve as you built and rebuilt the product?

A big question early in the project was: should the primary experience for how people listen in the app be focused on the long, multi-hour DJ sets, or center on playback of individual tracks? PRX and RoundArch interviewed people specifically about what they might want in a music discovery app from KCRW and after hearing from them we decided to focus the experience on track-by-track selection. That decision proved challenging and mid-way through the project there was a point where we really had to consider ditching the track-by-track access and move back to a DJ set focused app.

The sub-views in the app like the artist photo, song, video, bio, blog pages came together pretty quickly and didn’t change much over the project but the home view, the grid, was definitely an area where we iterated on ideas for a long time. Yeah — that took a really long time. We might have had 8 or 10 significant tweaks to the layout and presentation. But I think all the time we spent iterating on the ideas really paid off in the execution.

Alternate DJ set focused home view

Early DJ focused home view

Initial grid comp

Final grid design.

How do you feel about it? Is there anything you’d improve or add?

Well…no. Actually, I’m really happy with it. I just re-read the original product vision and I think we’ve really held true to that over the last ten or so months.

The KCRW iPad Experience project will focus on creating an iPad experience that quickly engages target users by leveraging KCRW’s unique style, taste and hand picked music. In addition, tight conceptual focus and appealing interaction models will aid in driving long-term user engagement.

People that we interviewed about what they might want in an music discovery app said they wanted us to make good music important to them, help them find new good music and improve their awareness. We’ve done that. People wanted us to connect them with great new artists, and help them learn more about music and artists they’re already familiar with. I think we’ve done that too.

What’s next from the PRX team?

I’m really excited about an iPhone project that we’re working on with a very popular public radio show. I feel like we’re developing a unique way to help people interact and connect with the show. We’re pushing the device’s capabilities, coming up with new interactions that can surprise and delight people.

Any last thoughts?

People talk about software as engineering, and it’s partially that, but it’s also a craft. This kind of work is like building a beautiful piece of furniture. When it’s done poorly, you end up with a cheap nothing that you throw away. But when it’s done well, you’ll never see all the work that went into it. Sometimes that means a lack of appreciation for the thought and care that was put in.

Devin Chalmers, Rebecca Nesson and Andrew Kuklewicz are top-notch developers, maybe the best people I’ve ever worked with. But even with the best people working together, it requires time and freedom to explore and do it right. We could have finished a long time ago if we’d done it poorly. Instead, we came up with something that we’re proud of, KCRW is happy with, and it seems so are the people using it.

Part Two

Coming up: Rebecca Nesson and Devin Chalmers will be digging into other more technical aspects of the KCRW Music Mine app, perspective transformations on scroll views, knob-twiddly stuff to tweak behavior and how we had to figure out layouts of a 4×6 grid in both portrait and landscape modes.

Enter Chef.

Chef allows us to develop (in source control) a system which describes both a particular configuration we would like our servers to have, and how to achieve this configuration. The configuration files are implemented in ruby, so the learning curve is primarily around nomenclature (and there’s a whole lot of it to learn). That having been said, most of the new concepts introduced by chef make sense once the system at large, “clicks.” I don’t expect a blog post to explain all of the nuance of an infrastructure configuration framework, but I think I can give enough introduction that one can walk away confident in their ability to do some simple deployments.

Clients and Servers

An Example Chef Deployment

The first thing one needs to learn about Chef is the notion of a server and a client. A chef server holds information about the chef clients¹ in your deployment environment and what configuration they should have in order to be considered set up. It also stores the code you have written so that clients can achieve the expected configuration, but it never executes it. A chef server can be thought of, in simple terms, as a database.

One can install the server themselves, or can opt to use the Opscode platform, which is free for smaller deployments.

Clients pull down information about how they are expected to configure themselves from the chef server, and then execute the code you have written to get to that point. They do this by manually or periodically running chef-client, which reads from the chef server what code it will execute, then downloads and executes it.

Cookbooks and Recipes

The ruby code stored on a chef server is grouped into files called recipes, and those recipes are grouped into directories called cookbooks. Usually, cookbooks are collections of recipes that relate to a specific software package. For instance, one might write a cookbook for the Apache web server that included a recipe for the web server itself, a recipe for mod_ssl, and a recipe for mod_dav.

Many cookbooks (and, by extension, many many recipes), are available on the opscode community website. If you see one you would like to install, you can use the knife cookbook site install <cookbook name> command. Recently, Opscode did a major overhaul and made sure that many of these cookbooks are in working order. Please be aware, however, that many of them were written months or years ago, and may require some tweaking, especially if the software package they work with changes often.

Run Lists and Roles

Another construct to be aware of is the, run list. These are usually recipes, listed in order, that should be executed by the client every time it pulls information down from the chef server. Chef also supports roles, which are run lists in their own right, but can be referenced in other roles and run lists. In our current deployment system, we have a few roles which each contain several recipes, and each Chef client is configured to have only one or two roles in its run list.

Starting a Chef-Managed Instance

The typical workflow for starting a new instance is to spin one up, install chef, register it as a client with your chef server, add things to the client’s run list on the chef server, and then run chef-client to pull down the run list and execute it. Quite a mouthful!

Thankfully, in practice, this can easily be rolled up into one step. Because we are making use of Amazon EC2 at PRX, we installed the knife-ec2 gem and we can run knife ec2 server create -r "run list". This spins up an ec2 instance, bootstraps chef, registers it as a client with the chef server, and sets the run list appropriately. It also runs chef-client automatically when everything is set up, so one should have a new EC2 instance with all of the software one wants installed and configured.

Wrapping Up

We’ve covered the actual steps involved in using a configured chef server to deploy new EC2 instances, but we haven’t touched on how to actually configure the server. For that, I will refer you to the fantastic Chef Repo article on the Opscode Wiki, which will explain the directory structure and basic workings of the repository you will use to interact with the chef server.

I found much of the language involved in learning the chef framework confusing when I was learning it, and I hope that this article serves to make the more fundamental concepts that are needed a little easier to understand.

1. This is not strictly true – chef makes a distinction between clients and nodes; the former being something which interacts with the chef server and the latter being something which is assigned a set of code to execute. In nearly every case, however, the two are interchangeable. Just remember that when you assign something a role or recipe, you are assigning a node, and when you download updated configuration details from the chef server, you are using a client.