New Media Initiatives Blog

The good news: about 8 million music lovers are simultaneously clicking on our email blast for information on Rock the Garden!

The bad news: about 8 million music lovers are subsequently melting our server. I feel like I should drive out there and blow on it to help keep it from overheating, but I’m on my bike today and not up for the extra miles. So…

Dear music lovers,

I’m glad there’s so much interest in the show! I think it’s going to be great. But please, please, click slower!

Love,
Your frantic webmaster

For the installation of Worlds Away, we needed to find a way to show the YouTube videos in the gallery. If you’ve visited the show, you’d see the videos installed in a Rec Room in the center of the gallery, complete with green shag carpeting, faux wood-gain paneling, low ceiling and bean bag chairs. It looks very much like my grandparent’s basement, minus the musty smell. The videos are viewable on a CRT television set on the floor in the corner of the room. I built a custom interface to allow people to control the TV, play the YouTube videos over the internet, at a decent quality, without violating the YouTube terms of service, and being mostly mischief-proof.

If you want to be spared the details, here’s a video:

Or, view it in a browser. This opens a pop-up window, Safari suggested, since that what it was built for. Use the left and right arrow keys, enter and escape to navigate.

When we started the project, we looked into ripping the flash video files from YouTube and creating a DVD. Testing showed the quality wasn’t great, but passable on a standard-def CRT television. The big hang-up about a DVD based approach was that it’s expressly forbidden in the YouTube terms of service:

You agree not to access User Submissions (defined below) or YouTube Content through any technology or means other than the video playback pages of the Website itself, the YouTube Embedable Player, or other explicitly authorized means YouTube may designate.

While it’s one thing to just avoid agreements like this with a wink and a nod for one-off or personal use, it didn’t seem like a good idea to blatantly violate the terms of service for an installation that would be in the gallery for months.

We also discussed setting up a computer in the gallery and simply putting a web page on it thin a custom player and view the videos. We weren’t too happy with that idea either, since it didn’t really fit within the Rec Room concept in the gallery — my grandparents don’t have a computer in the basement. We also discussed putting a big plasma TV on the wall, more like a modern TV, and using a remote or some sort of a pointing device to select videos using a custom YouTube player. Again, this option seemed costly (no spare plasma screens sitting around) and not within the theme of the room.

What we wanted was something kind of like the Apple TV, except limited to the 12 videos we selected for the exhibition. Quick research revealed no way to lock down an Apple TV or limit it to a particular YouTube playlist, so that was out. But what about making a web page that worked in a similar fashion?

It is possible to make Safari work in fullscreen mode using Saft. And there are applications that work with the Apple Remote to set up custom events and keystrokes. A computer with the right video card can also go out to a TV. And a little javascript magic in the browser could pull it all together with fancy fade and scroll effects.

jQuery and Safari fun
I started with the web page. I already knew I wanted to use jQuery for this project, if possible. I’ve been using MooTools for a while now and I wanted to expand my horizons to include a second Javascript library (half kidding). Using a TV screen for a display limits the usable resolution to 640×480, so there is not enough room to display all the entries on screen at once. Instead, I found the jCarousel plugin that I was able to use to scroll the video information across the screen. JCarousel has a lot of hooks for various events, so it made it quite easy to plug in to. The second thing I’d need was something to show the videos on screen once selected. The thickbox plugin is a lightbox clone, and seemed to do the trick. I tied it all together with some custom event hooks for keystrokes and a some logic to keep errant keystrokes from triggering multiple videos, excess scrolling, etc. In the end, I found that using the arrow keys and esc and enter are the best. In Safari, if you use any actual letter keys, it will default to type ahead find functionality and interfere with the scrolling offset.

The interface is designed explicitly for Safari. Since this is a single computer installation under our control, I don’t have to worry about whether or not it works in IE or Firefox. I have no idea if it works in IE (ignorance truly is bliss), but it only half-works in Firefox. Firefox doesn’t want to load the flash player in Thickbox. I’ve experienced problems with Firefox ignoring the z-index on Flash elements in the past, so this is not surprising.

Remote buddy
Now that I had figured out how to deal with the browser, I had to figure out a way to have the remote interface with the browser by sending specific keystrokes for each button. After looking at several programs that do this with the Apple remote, I settled on Remote Buddy. It was the easiest to configure so that it would only work with specific applications. I configured it to send the left arrow key on left arrow press, right on right, enter on play and escape key on menu button press.

One problem with the remote was that our computer was an older mirror-door G4 tower, which does not have an infrared receiver. Thankfully, Remote Buddy supports third-party usb receivers, so I ordered one, and it worked like a charm. Microsoft does build good hardware.

Video out
Unfortunately, we weren’t able to track down a G4 that had S-video or composite out, so I also ordered a newer video card for that. We got an ATI Radeon 9200 which has an excellent control panel for TV-out settings, letting you manipulate all aspects of the size, stretch, overscan, etc.

Net connection
In this configuration, the videos stream in real time from YouTube’s servers. Our galleries have pretty good wireless saturation, and an airport card in this tower allows the machine to connect to the network. I spent some time looking into using Squid for caching the youtube videos, but it appears YouTube recently changed the way videos are embedded, using some sort of a time-based token in the URL, making caching difficult if not impossible. Another option would be to keep the videos loaded in the browser, but in a hidden element. However, since there’s no interface for controlling the YouTube embedded player with javascript, starting and stopping the player is not possible.

Putting it together
After connecting all these things, I cleaned up the interface a bit, changing the typeface to Avenir, choosing colors that worked better on the Television (don’t use white). I configured Safari to start in full-screen kiosk mode, which Saft enables. Safari and Remote Buddy were set to launch on login, and the machine is set to automatically log in. Additionally, I set up a simple shell script that checks each of them to make sure they’re running and if not, relaunches them. With relaunching the applications, it’s important to use the command-line version of Applescript, since that will cleanly deal with backgrounding the process and giving correct screen focus.

Doc Czypinski, who was the lead tech for installing the show, was able to figure out a way to glue a white plastic strip to the back of the remote. He was then able to cable the remotes to the gallery furniture so that they don’t walk away during the show.

Post-mortem
Now that the computer has been in the gallery for about a week, there are two things we’ve learned. First, visitors tend to expect things to be playing automatically on the TV, since that’s the way most media things are in galleries. When they do figure out they need to use the remote, they tend to click the big play button and not cycle through the videos to choose. I’ve attempted to help this by putting arrows on the screen to indicate left and right so that visitors know to use the arrows on the remote to scroll.

The second issue is that the wireless hasn’t been 100% reliable. As anyone who uses Wifi on a regular basis knows, it’s not always up 100% of the time, and the wifi in our gallery goes out once in a while. I’m looking at writing a script that will cycle the Airport card off and on if it looses connectivity, or displays a message on screen if there is extended connectivity loss. A wired connection would certainly be better, but the convenience of Wifi cannot be beaten.

Gallery photo by Cameron Wittig. Arm of Justin Heideman.

One of the cool things we’re doing for the Walker’s upcoming exhibition Picasso and American Art is significantly increasing our iPod audio tour capacity. For the exhibit we were able to get 25 iPod Video’s, and like our normal iPod audio tours, we will be letting visitors use them for free. The same content is also available on Art on Call.

This presents a bit of a challenge however. Up until now we’ve only had four iPod Nano’s to worry about, and plugging a few into a computer or two to charge isn’t that big a deal. Now however we have 25 of them to deal with, and there certainly aren’t enough USB ports to go around. The goal was to find a way to charge most of the iPods, do it in a limited space, and do it for as cheap as possible.

My solution was to buy three USB hubs and use them just for charging. We don’t really need to have them connected to the computer to sync with, we just want the power. This turned out to be harder than I thought. I went through a few USB hubs trying to get the iPods just to charge off the supplied AC adaptor. Each hub I tried didn’t allow this. It would only charge when the hub was connected to a computer via USB. I can’t fathom a reason why they limited it like this, as the power comes from AC on the hub, not from USB. Whether the hub was connected to a computer should not really dictate whether power could be supplied to the device or not. Alas, there was no cost efficient way around this.

So I had no choice, if I wanted to charge via any hub, I had to connect the hub to a computer. Thankfully we did have a computer near where our iPod storage is. Except it only has two open USB ports, not the three I needed. Another stumbling block. But then the thought occured to daisy chain the hubs. In essence, the USB cable that was supposed to go to the computer for each hub would plug into one of the other hubs instead. The last in the chain would then plug into the computer. Basically we could connect all of the iPods to a computer with one USB cord, regardless of how many hubs we had. And that’s what we did, as it worked perfectly:

One interesting feature of this is it allows us to mount all of these iPods at once, as you can see here. This actually makes adding and editing content on all of them a breeze. So in the end, perhaps all of the troubles were a blessing.

Total cost for this: $60. It may not look the prettiest, but sometimes when you’re trying to be frugal, getting something that just works is what counts.

We Make Money Not Art has an interview with Burak Arikan up, who is the lead developer of OpenIO and Pinkie. Prior to reading the interview, I had no idea what either were, but they sound very cool, like most things that come out of the Media Lab:

Pinkie is a network based electronics prototyping board. Pinkie has been designed to easily compose sensors and actuators that reside in different locations. Pinkies are inherently invisible, they hide behind the structures and only serve as facilitators to interface the physical world to the digital network.

Open I/O works like a peer-to-peer file sharing program, rather then sharing media files in your PC, you share data sensed from your physical environment. While Pinkies are organizing the low-level information (e.g., sensing the world), Open I/O is for higher concepts such as managing distributed devices, collaboration, and social networking.

This sounds to be a very interesting project. It seems like the floodgates are starting to open on small hardware devices that are open and easily programmable. Since there are so many of these devices, it seems only natural that they need social networking.

[Now that our main blogger is leaving, we’ve got to start picking up the slack and posting. I promise eventually to not just post about hardware and software bugs, but today that’s what I’ve got…]

Porter continues to be a rockstar hardware-wise, but I’ve been having some trouble with the proxy/caching webserver running on it. Sure, it’s caching, but every so often it would grab a version of page and decide to keep it for 3 days instead of the directed 1 hour. At first I thought it was a one-time deal from switching some cache settings on the server, but it kept happening… Walker staff would make a change to some content, wait, but it would never show up on the live page. Trouble.

The problem was caused by a line stored in the cached HTTP header: Cache-Control: max-age=259200. (that’s three days worth of seconds) (I’m including details so google can pick this up and hopefully save some poor guy a frustrating morning.) After some serious digging it appears the mod_cache module we’re using was taking whatever Cache-Control header was being sent by the browser and saving it in the cached header! In other words, I had configured the server to cache things for a maximum of 1 hour, but all it took to blow that up was a browser (or spider?) sending a request saying it didn’t want anything older than 3 days. Our caching server held on to that “3-days” part and decided the whole page should be valid for that long. Totally. Wrong.

I debated making changes to the mod_cache source and recompiling, but I finally found an easier answer: “CacheIgnoreHeaders Cache-Control". This tells the caching module to ignore the problem lines, and it seems to be golden. I’ll let it run for a while and see…

[In further bad news, the US got creamed 3-0 by the Czech Republic in their World Cup opener. Not unexpected, but it doesn’t bode well for getting past group play…]

I would like to be the first to publicly welcome “Porter” to the Walker’s family of webservers! Porter was, to be honest, long overdue - and to continue the awkward metaphor, it was a difficult birth. Maybe next time a C-Section. (ok, I’m done.)

The problem was obvious to anyone who’d used our website for any significant amount of time in the last year or two: as our technology on the backend increased, as new features and sites were added, the existing server was crawling to a slow and painful death. Frequent reboots (reboots! On Linux! The horror!) were required, and working in the CMS admin system was nearly intolerable. You could literally go get a drink of water while loading certain pages.

The solution was equally obvious: upgrade! But the execution proved quite labor-intensive - lots of tightly integrated bits and pieces that had to be unravelled carefully and put back together to create a semblance of a whole. Really.

My goal was to transition to the new server without any noticeable downtime, and it went as well as I could have hoped. There were some tense moments at the end - there’s really nothing like the feeling of pulling the plug (metaphorically) on an entire institution’s website and crossing your fingers you didn’t miss something when the new one comes up. Then a big sigh of relief when you realize of course you did, but it’s pretty minor, and wow! Look how much faster it runs!

So, welcome, Porter! You make your daddy proud. (ok, now I’m done.)

Good news in New Media - two new web servers arrived yesterday. This will provide a much-needed upgrade to a few of our sites and hopefully give us some room to grow. We can finally separate a few things that should never have been running on the same machine, and merge some things that should.

Of course, I say that like it’s going to be easy - I’m actually a bit nervous about the whole thing, it’s a lot of custom code and applications to port, not to mention our whole staging/production process. So I’ve got my work cut out for me, but it will be worth it in the end. Keep your fingers crossed!

Recently the New Media department at the Walker completed a cubicle shuffle. In the process I stumbled across this great lost peice of technology that fulfills no useful function but still I haven’t been able to part with. It is the Radio LAN.

A precursor to WIFI the Radio LAN extended a 10Base T Network ao the FM radio waves by means of this “backbone” and transmitter (a 50mW transmitter). The instruction manual claims to have had a range of 150-200 feet.

On the other end of the transmission you need one of these cards. A standard PCMCIA card that has some sort of gigantic plastic thing on it to receive the signal. I have it plugged into my 15″ powerbook just so you can get an idea of scale.

The copyright from the instruction book is from 1998 so it seems way ahead of it’s time. Nice to see the Radio LAN company is still in business but it looks like their products have changed into more long range wireless solutions.

Now that I have successfully documented and shared this great step in wireless history I think it’s finally time to get rid of it.

When we last tuned in, it was the day of the public opening and the Walker’s website was down. We join our hero on his way to physically check the troublesome server:

Arriving at Onvoy, the server appeared to be trying to reboot and just needed a keypress. After that it came up cleanly - the drive is journaled via ext3, so it didn’t even have to check the disk. Problem solved? At the time I didn’t know for sure what had caused the original issue - and I’d deleted most of the /var/log/messages (the main system log) that I’d need to diagnose it. (Why? Bad instincts, I guess: The initial assessment showed the /var partition was full - which is enough to hose a system - so I copied most of what I thought I’d need and then emptied the file).

So I was left with a working server (yes!!) but no solid idea about what had caused the drive I/O errors — the portion of the log file I’d retained only showed the symptoms, not the onset error. I decided the best I could do immediately was to just let it run and watch the logs - and figure out how to restore from our backups.

The restore procedure turned out to be very straightforward, and I immediately took steps to build a set of worst-case scenario disaster recovery CDs. (these included base OS installs for all our production servers and a CD containing a fresh install of the recovery utility and master boot record images of the servers)

But watching the logs proved uneventful - even when the server crashed again early Wednesday and the next Sunday morning. (ahhhhhh!) It seemed whatever was happening essentially took the drive completely offline, and hung the entire operating system while it waited for the drive to come back — so the logs stopped being written. No permanent data to diagnose the problem. Also, the machine would not succesfully reboot until it was power cycled - a soft reboot did not work. (what??!)

If I could catch the server as or just after it crashed, I could physically get to it before it locked up completely and check the logs and dmesg output. Maybe that would give me enough information to solve the crashing server. So it was a game of waiting and researching the few clues I had gathered…

Imagine my great distress when I woke up Sunday morning the 17th, the big public opening, to a screen full of alert messages - our web site (and Art on Call) had been down since about 4:30 that morning. I had a terminal window open from the night before, so I quickly tried to restart one of the Apache servers — file not found?!! There were no files in the software folder. No files in the home folders for our websites. Panicked, I checked the logs: full of I/O errors for the drive. Trying to reboot left the machine completely unresponsive. AHHHHH!!!

I knew there were backups being made by the company we’re colocated with - Onvoy - but I’d never had to use them and didn’t quite know where to start. Some quick reconnaissance in our internal wiki told me the drive was a SCSI drive. Crap. On the 17th I knew just enough about SCSI to know I didn’t know enough to run out and buy a new drive on the spot - way too many options to wade through. A call to a local hardware store (General Nanosystems) confirmed my fears - “is it SCA or LVD?” “um…”

The SCSI interface (pr. “scuzzy”) is really quite incredible. Most desktop machines use the IDE interface to connect their hard drives, which is all well and good for their needs, but production-quality servers need something more - something faster, more reliable, better engineered, and self-diagnosing… Enter SCSI drives.

I head out to Onvoy with a pit in my stomach - even if I can get a new drive today, I’m not confident I can learn or find someone who knows how to restore from the backups… Oh, did I mention it’s the biggest day for the Walker since I started working here? The grand public re-opening?

Tune in next time for part two of the saga, in which our hero saves the day — but really only postpones disaster…