Blogs Media Lab

Installing a Lighttpd proxy

By

It had been a slow build, but an incident a few weeks ago made it finally clear: the Walker website was becoming a victim of its own success.  A post on the Teens site contained a picture of the Joker for the then-upcoming Batman movie, and as Halloween approached we found ourselves on the front [...]

It had been a slow build, but an incident a few weeks ago made it finally clear: the Walker website was becoming a victim of its own success.  A post on the Teens site contained a picture of the Joker for the then-upcoming Batman movie, and as Halloween approached we found ourselves on the front page of Google image search as people began looking for costume ideas.  The exponential traffic was crippling our web server:

The biggest problem was simply that Apache is heavy.  It’s resource-intensive, especially when you are running several modules as we were – PHP, proxy, cache, etc.  The teens site is especially difficult since it runs as a combination of a blog (PHP on Apache 2) and .wac pages (mod_perl & Axkit).  Every hit to the Joker post – even if the page was cached – would tie up a number of Apache processes as it served the style sheets, images, and javascript to support the page.  We were reaching our MaxClients setting but unable to raise it without running out of memory for our other more intensive servers (mod_perl and postgres, I’m looking at you…).

As this diagram shows, it’s nothing but Apache servers, and it just wasn’t scaling to meet our current demand.

The approach was two-fold: some quick auditing and re-writing of the worst offending .wac pages’ SQL to speed up the slow pages, and yet another web server in front of everything.  It was a no brainer to pick Lighttpd, or “Lighty”.  It’s written to do one thing – serve static content – and do it extremely fast.  Fortunately it can also proxy requests, so it was a pretty simple matter to reassign some ports and write a few rules to route all requests through Lighty.

The end result is astonishing.  Our server hums along happily under even the most intense traffic we can throw at it (the email blast for the British Television Advertising Awards) and doesn’t even start to complain.  Moving the bulk of the requests to the extremely fast and resource-light server meant we could devote more resources to quickly processing the slow pages (mod_perl).  Between the SQL tuning and the extra resources, the bulk of these pages are now served between 2 and 10(!!!) times faster!

The lesson here, for anyone with an Apache server creaking and groaning under increased traffic, is to stop waiting and install Lighty.  If your site is PHP-based, you can run this as a fast CGI module from Lighty and do away with Apache altogether.  You can also use Lighty to stream (and “scrub”!) flv and mp4 video files.  (I’m using both of these techniques for the new ArtsConnectEd.)

The only caveat: be careful as you look for examples on the web.  Remarkably, it seems there are many confused webmasters who expect to see a performance boost by putting Lighty behind their struggling Apache.  This will not help at all, and in fact will probably make things worse.  Lighty has to be first in the chain to take the load off Apache.

Enjoy the speed!  I know our server is enjoying the breathing room!

Walker Blogs survey results

By

Thanks to everyone who took our Blogs survey over the past couple of weeks. We received a good amount of feedback that we’re in the process of digesting. We also picked a winner for the iPod shuffle, who happens to be a new media developer as well. I hope to post a little more info [...]

Thanks to everyone who took our Blogs survey over the past couple of weeks. We received a good amount of feedback that we’re in the process of digesting. We also picked a winner for the iPod shuffle, who happens to be a new media developer as well. I hope to post a little more info about that soon.

I thought I would share some of the statistics and comments from the survey, in case anyone is interested.

My hunch is that the kind of people who take our survey are those that are already somewhat interested in programs that the Walker offers, and are likely to want to closely follow what we do. The responses to the first question seems to back up that assumption:

How did you find the Walker blogs?

How did you find the Walker blogs?

The number of people who came here from a search engine is significantly lower than what we see looking at our analytics, but not all search engine referred visits are of the same quality as people who come on their own. Also, if you’ve been a blog subscriber for years, do you really remember how you found the blog in the first place? Probably not.

We asked how often people read the blogs. The answer is pretty often, with a good chunk of people using RSS readers:

How often do you read the Walker blogs?

How often do you read the Walker blogs?

And the reasons people read the blogs:

For what reasons do you read the Walker blogs?

For what reasons do you read the Walker blogs?

How many people have left a comment:

Have you ever left a comment on the Walker blogs?

Have you ever left a comment on the Walker blogs?


Our blogs are not the most heavily commented around. Often the style of posting we engage in isn’t the most comment inducing.

Where people live surprised me a little bit:

Where do you live?

Where do you live?

St. Paul seems under-represented, as do the suburbs (which are caught up in the jumble of text in the graph). We did focus the choices for this question on the United States, but we do know that there is an international readership.

We also asked people what other blogs they read, which gave us a ton of responses. Here are the handful that were mentioned the most:

And some of the more eclectic (or just mentioned less):

I liked this question a lot, because it gives a sense of what our visitors are reading and what their interests are. The list is heavily weighted towards museum/art blogs and design related blogs. I also liked the question because it gives me a few more sites to add to my RSS reader. Thanks survey takers!

Finally, we asked for some general feedback and comments from anyone who wanted to share. Here are several:

The Walker blogs are great. I really like hearing from designers at Dwell, a magazine I also read, right on the Walker’s site. I love that dialogue w/ outside designers and artists that’s brought to me via the Walker because of the staff’s professional networks that extend well beyond my own! Keep it up :)

I enjoy reading your blogs but I really feel that they could use more pictures,video and audio

It would be nice to have guest bloggers on every topic, or to have less specific, umbrella blogs covering a not-so-wide range of topics and points of view.

Be less Minnesotan.

(I’m not sure what that means, eh?)

I live in Massachusetts and can’t come to the Walker very often, but I love your blogs because they keep me connected… and they make me want to move to Minneapolis.

I think NMI should blog more

Here’s trying.

Working with iTunes U

By

Several weeks ago, Robin posted about the Walker Channel on iTunes U. I am going to follow up on her initial announcement with a more info about the process of designing an iTunes U Page, the preparation of content, and putting content online. Designing an iTunes U Page There are a number of different designs [...]

Walker Art Center iTunes U page

Walker Art Center iTunes U page.

Several weeks ago, Robin posted about the Walker Channel on iTunes U. I am going to follow up on her initial announcement with a more info about the process of designing an iTunes U Page, the preparation of content, and putting content online.

Designing an iTunes U Page

There are a number of different designs for pages in iTunes U. Some institutions that have been in the store for a while have a three column layout. However, Apple has now standardized on a two column layout for iTunes U pages. There options for customizing a page are limited, but not restrictive. Colors for backgrounds, borders, and text can be changed. An overall header image that is 600px by 300px is used on the top of the main page. The downside of a two column layout is that it does not re-size to a smaller iTunes window as nicely as a the three column layout.

A three column iTunes U page

A three column iTunes U page.

Within the main page, you can create separate content groups. We decided to go with three sections: Featured, Exhibitions, and Topics. Within these sections, you create course pages. Each course can be customized with an icon, description, author/instructor, and links. In each course, there can be multiple tabs for different groupings of content. We’re using “Tracks” for most, which are a mix of video and audio content. A few exhibitions courses also have tabs for Art on Call content.

In order to design our site, I ended up doing some of the initial work right in iTunes. I figured out our color scheme and organizational structure, then took a few screenshots of iTunes. The screenshots were pasted together in photoshop, and I layered the header and course images onto it. Thankfully, the iconography choices were straight forward. The exhibitions use images from an exhibition, either artwork or an installation view. For Subjects or Featured courses, the icons are all similar, just using color, pattern, and language changes, each referencing the different artistic program pages that are already on the Walker web site.

Encoding video for the iPod using h.264

The h.264 codec is both amazing and vexing. It has very high compression, good quality, and is a widely supported standard. Working with h.264, especially for devices, can be complicated. Since the 5th generation, iPods have been able to play h.264 encoded video. They can even play 640×480 video and downscale it to their 320×240 screen, which is great since a 640×480 video will look good on a larger screen too. The real trickery with h.264 comes in with profiles.

Exporting to mp4 in Quicktime Pro. Not iPod compatible.

The MPEG Streamclip settings we use.

The MPEG Streamclip settings we use


Most of the time, if you just export a movie from quicktime using h.264, you use the main profile. However, for a device like the iPod, which doesn’t have a fast processor, Apple specifies that you need to use the low-complexity profile. There technical differences are mostly beyond me, but the low-complexity profile drops some of the more advanced hinting and shape features, but will mean a less processor intensive decode process that the iPod can handle.

Getting video encoded into a low-complexity h.264 profile is not a clear process. Apple’s own QuickTime Pro doesn’t let you encode to low-complexity and have any control over the output. If you want to make a movie compatible for the iPod, you must use the Movie to iPod or Movie to iPhone preset. Both of these presets encode at a very high bitrate, which makes for good quality. However, if you have the scenario we have– long movies of not a lot of action–a high bitrate is both filesize prohibitive and not necessary to maintain quality.

Some time ago, we switched to saving all our channel videos in a mp4 file, using the h.264 codec, thinking that it would make them iTunes compatible. We apparently missed the low-complexity part, and discovered that our videos were, in fact, not iPod compatible. This meant we would need to re-encode our video files to make them more useful in iTunes U. I looked at several different pieces of software to do this, but eventually decided upon MPEG Streamclip.

As I noted above, Quicktime Pro would not work for this. I also looked at Compressor, basically the Pro version of QuickTime Pro. Compressor offers much more customizability than QuickTime Pro in terms of codec configuarion and workflow. Compressor, for some reason, takes an inexplicably long time to encode a iPod compatible mp4. On a high-end Mac Pro, encoding a 640×480 was taking well beyond 8 hours. The output look really good, but given that we had 50 files to convert, it was simply not an option, even when using distributed encoding.

I also looked at FFmpegX and VisualHub (now defunct). Both of them are essentially wrappers around FFmpeg, and they produce good results, are very efficent encoders, and let you adjust every setting (almost to a fault). However, FFmpeg suffers from being written to expect a PC gamma of 2.2, and the resulting videos looked considerably darkened when compared to the original.

In the end, MPEG Streamclip worked the best. It offers the same speed of FFMpeg, much of the same control over settings, and deals with the gamma–outputting the a proper video for the iPod. At a bitrate of about 950kpbs, a typical two hour lecture comes in between 450-500 MB, just below the iTunes U limit of 500 MB.

Putting Content into iTunes U

The processes of editing content and putting tracks into iTunes U straight forward, though frustrating, since it involves a lot of clicking and waiting. iTunes has evolved considerably over time, and certainly letting a huge range of users edit parts of the iTunes Music store was not one of the original design specifications. The process is a bit clunky and Web 1.0-style, but it works. Uploading content is done through a browser, which can be a very finicky, especially with large files. After some trial and error, I figured out that setting Firefox as my default browser and using that for uploading worked better than Safari. Safari will time out the upload after a period of time, whereas Firefox keeps on chugging.

Before files are uploaded, they need to be properly loaded with metadata. iTunes U doesn’t let you edit much on the site (just title and artist) so other fields must be filled in on iTunes on your computer before uploading. When you edit the metadata fields, iTunes commits the changes to the movie files itself. When you upload the movie files, iTunes U will pick up on this and display it. One thing I found a little confusing that artwork is not displayed in the store or when you are previewing a file. Apple says that artwork on movie files is used to display on the iPod, but never in iTunes. This is all covered pretty well in the iTunes U User’s Guide.

Despite the time spent figuring out codecs and monkeying around with uploading, we’re very happy to have our content in another venue and excited to keep adding more.

Hacking the iPod Touch

By

Shelley Bernstein over at the Brooklyn Museum has posted some details on their recent hack of the iPod Touch to use in the gallery. Shelly hasn’t posted all the details on the blog, but if you contact her, she will be happy to link you up to the juice. If you’re looking to do something [...]

Shelley Bernstein over at the Brooklyn Museum has posted some details on their recent hack of the iPod Touch to use in the gallery. Shelly hasn’t posted all the details on the blog, but if you contact her, she will be happy to link you up to the juice. If you’re looking to do something like this in a gallery, it’s a great head-start.

Photo by Shelly Bernstein

Photo by Shelly Bernstein

Using an iPod touch as a video display in a gallery is a really great idea for a number of reasons:

  • iPods are cheap (relatively).
  • The screens are offer a high resolution and an acceptable size.
  • They’re small and unobtrusive, so they have the potential to not irritate curators who dislike too much stuff near the art. 
  • The playback hardware is contained right in the unit, so no need for extra wires to a DVD player or other playback device.
  • They have WiFi, so there is potential for remote administration, updating, and connecting to content on the Net. 
  • If you get a first generation, they’re hackable. The second generation will probably be hackable soon. Thanks to things like Cydia, you can install SSH and all kinds of useful goodies.
  • The interface is simple. Though I’m not sure if my grandma would know how to interact with it. 

The only real downside to the IPod touch is the cord comes out a weird angle, making the mounting and display a little tricky.