Linksys wireless router

Being familiar with the dd-wrt project, I was pretty sure I could build a firewall out of a Linksys router.  We went with the WRT54GL, currently as cheap as $50 on Amazon.  (We bought local so we’d have it sooner, and it was a bit more).

The first step after flashing the firmware with the latest dd-wrt build (v24-sp2) was to take off the antennas and turn off the radio.  The last thing I want for the firewall is to be broadcasting an SSID and allow wireless associations.  This actually requires a startup script on the router, with a line to remove the wireless module so it won’t try to reenable itself:

wl radio off
wl down
rmmod wl

Good start.  Next I needed to bridge the WAN port with the LAN ports, which ended up being a struggle until I found the easy options in the dd-wrt GUI.  First, set the LAN to use a static IP and make sure you can connect via another machine to configure it.  You’ll also need to enable SSH access and remote configuration – but be sure to lock this down once the firewall is running!

Once you have the LAN configured, you need to set the WAN connection type to “disabled”.  This will give you a checkbox to bridge the LAN and WAN:  “Assign WAN port to switch”.  Lastly, under Advanced Routing set the Operating Mode to “Router” so it stops trying to do NAT.  Apply these settings, and you’ll basically have an expensive dumb switch – all traffic shows up on every port, and there’s no logic at all.  We’re halfway there.

Being unfamiliar with iptables (we use OpenBSD and pf for firewalls around here), I was under the impression that iptables rules would work in a bridging environment.  This is not the case: bridged packets don’t reach iptables at all!  The best I could do was block everything (manual restart needed), or otherwise blow up the configuration (manual restart needed) as I tried to mess with the bridge.  This was an incredibly frustrating learning curve as everything I could find made it sound like this was the way to configure a firewall in Linux, but it just wasn’t working.

Note to keep you sane: don’t do any of this testing in the startup scripts or you’ll brick your router, guaranteed.  Do it all from the command line with a known-good startup.  That way it’s a simple (but annoying) power cycle to get things back up.

The trick, it turns out, is a kernel module called ebtables.  Luckily, this is included in the dd-wrt build, but it’s not turned on by default!  Add this to your startup script:

insmod ebtables
insmod ebtable_filter
insmod ebt_ip.o

And, ta-da, all your iptables rules will start impacting packets!  Now it’s just a matter of configuring the firewall rules.  We’re using something like this:  (vlan0 represents the LAN ports, and vlan1 is the WAN port)

# drop everything by default:
iptables -P FORWARD DROP
# clear the old rules:
iptables -F FORWARD
# forward stuff that's established already
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
# let connections out:
iptables -A FORWARD -i vlan0 -m state --state NEW -j ACCEPT

# firewall access rules
iptables -F INPUT
# WAC ips can get to fw:
iptables -A INPUT -p tcp -d 1.2.3.4 -s 4.3.2.1/24 -j ACCEPT
# drop everything else!
iptables -A INPUT -p tcp -d 1.2.3.4 -j DROP

# ... snipped all the actual access rules and packet flood protection ...

The only trick here is the last few lines which limit access to the firewall machine itself.  We can’t use the FORWARD rules since these packets are destined for the internal hardware and not forwarded, but we do need to limit access via the INPUT chain.  In this example the firewall has IP 1.2.3.4 and the network I want to access it from has 4.3.2.x.  That way I can leave the firewall’s remote access turned on and limit it to our network.  (because there’s no terminal access you can’t make it a truly transparent bridge or you’d never be able to change the config!)

I admit I’m a bit nervous posting some of this in case there’s a glaring security hole, but it seems good to me.  Anyone see anything they’d like to warn me about before we get hacked?

And there you have it!  For the cost of a cheap router and some time (not much, since you can just follow these notes!) you have a full-featured bridging firewall running on dedicated hardware.  With a little extra work it would be easy to get VPN running and much more…  I’m hoping for years of service from this little guy!

( Hat tip another DIY firewall solution that I’d really like to try someday. )

The biggest problem was simply that Apache is heavy.  It’s resource-intensive, especially when you are running several modules as we were – PHP, proxy, cache, etc.  The teens site is especially difficult since it runs as a combination of a blog (PHP on Apache 2) and .wac pages (mod_perl & Axkit).  Every hit to the Joker post – even if the page was cached – would tie up a number of Apache processes as it served the style sheets, images, and javascript to support the page.  We were reaching our MaxClients setting but unable to raise it without running out of memory for our other more intensive servers (mod_perl and postgres, I’m looking at you…).

As this diagram shows, it’s nothing but Apache servers, and it just wasn’t scaling to meet our current demand.

The approach was two-fold: some quick auditing and re-writing of the worst offending .wac pages’ SQL to speed up the slow pages, and yet another web server in front of everything.  It was a no brainer to pick Lighttpd, or “Lighty”.  It’s written to do one thing – serve static content – and do it extremely fast.  Fortunately it can also proxy requests, so it was a pretty simple matter to reassign some ports and write a few rules to route all requests through Lighty.

The end result is astonishing.  Our server hums along happily under even the most intense traffic we can throw at it (the email blast for the British Television Advertising Awards) and doesn’t even start to complain.  Moving the bulk of the requests to the extremely fast and resource-light server meant we could devote more resources to quickly processing the slow pages (mod_perl).  Between the SQL tuning and the extra resources, the bulk of these pages are now served between 2 and 10(!!!) times faster!

The lesson here, for anyone with an Apache server creaking and groaning under increased traffic, is to stop waiting and install Lighty.  If your site is PHP-based, you can run this as a fast CGI module from Lighty and do away with Apache altogether.  You can also use Lighty to stream (and “scrub”!) flv and mp4 video files.  (I’m using both of these techniques for the new ArtsConnectEd.)

The only caveat: be careful as you look for examples on the web.  Remarkably, it seems there are many confused webmasters who expect to see a performance boost by putting Lighty behind their struggling Apache.  This will not help at all, and in fact will probably make things worse.  Lighty has to be first in the chain to take the load off Apache.

Enjoy the speed!  I know our server is enjoying the breathing room!

Photo by Shelly Bernstein

Using an iPod touch as a video display in a gallery is a really great idea for a number of reasons:

• iPods are cheap (relatively).
• The screens are offer a high resolution and an acceptable size.
• They’re small and unobtrusive, so they have the potential to not irritate curators who dislike too much stuff near the art. 
• The playback hardware is contained right in the unit, so no need for extra wires to a DVD player or other playback device.
• They have WiFi, so there is potential for remote administration, updating, and connecting to content on the Net. 
• If you get a first generation, they’re hackable. The second generation will probably be hackable soon. Thanks to things like Cydia, you can install SSH and all kinds of useful goodies.
• The interface is simple. Though I’m not sure if my grandma would know how to interact with it. 

The only real downside to the IPod touch is the cord comes out a weird angle, making the mounting and display a little tricky.

The bad news: about 8 million music lovers are subsequently melting our server. I feel like I should drive out there and blow on it to help keep it from overheating, but I’m on my bike today and not up for the extra miles. So…

Dear music lovers,

I’m glad there’s so much interest in the show! I think it’s going to be great. But please, please, click slower!

Love,

Your frantic webmaster

For the installation of Worlds Away, we needed to find a way to show the YouTube videos in the gallery. If you’ve visited the show, you’d see the videos installed in a Rec Room in the center of the gallery, complete with green shag carpeting, faux wood-gain paneling, low ceiling and bean bag chairs. It looks very much like my grandparent’s basement, minus the musty smell. The videos are viewable on a CRT television set on the floor in the corner of the room. I built a custom interface to allow people to control the TV, play the YouTube videos over the internet, at a decent quality, without violating the YouTube terms of service, and being mostly mischief-proof.

If you want to be spared the details, here’s a video:

Or, view it in a browser. This opens a pop-up window, Safari suggested, since that what it was built for. Use the left and right arrow keys, enter and escape to navigate.

When we started the project, we looked into ripping the flash video files from YouTube and creating a DVD. Testing showed the quality wasn’t great, but passable on a standard-def CRT television. The big hang-up about a DVD based approach was that it’s expressly forbidden in the YouTube terms of service:

You agree not to access User Submissions (defined below) or YouTube Content through any technology or means other than the video playback pages of the Website itself, the YouTube Embedable Player, or other explicitly authorized means YouTube may designate.

While it’s one thing to just avoid agreements like this with a wink and a nod for one-off or personal use, it didn’t seem like a good idea to blatantly violate the terms of service for an installation that would be in the gallery for months.

We also discussed setting up a computer in the gallery and simply putting a web page on it thin a custom player and view the videos. We weren’t too happy with that idea either, since it didn’t really fit within the Rec Room concept in the gallery — my grandparents don’t have a computer in the basement. We also discussed putting a big plasma TV on the wall, more like a modern TV, and using a remote or some sort of a pointing device to select videos using a custom YouTube player. Again, this option seemed costly (no spare plasma screens sitting around) and not within the theme of the room.

What we wanted was something kind of like the Apple TV, except limited to the 12 videos we selected for the exhibition. Quick research revealed no way to lock down an Apple TV or limit it to a particular YouTube playlist, so that was out. But what about making a web page that worked in a similar fashion?

It is possible to make Safari work in fullscreen mode using Saft. And there are applications that work with the Apple Remote to set up custom events and keystrokes. A computer with the right video card can also go out to a TV. And a little javascript magic in the browser could pull it all together with fancy fade and scroll effects.

jQuery and Safari fun

I started with the web page. I already knew I wanted to use jQuery for this project, if possible. I’ve been using MooTools for a while now and I wanted to expand my horizons to include a second Javascript library (half kidding). Using a TV screen for a display limits the usable resolution to 640×480, so there is not enough room to display all the entries on screen at once. Instead, I found the jCarousel plugin that I was able to use to scroll the video information across the screen. JCarousel has a lot of hooks for various events, so it made it quite easy to plug in to. The second thing I’d need was something to show the videos on screen once selected. The thickbox plugin is a lightbox clone, and seemed to do the trick. I tied it all together with some custom event hooks for keystrokes and a some logic to keep errant keystrokes from triggering multiple videos, excess scrolling, etc. In the end, I found that using the arrow keys and esc and enter are the best. In Safari, if you use any actual letter keys, it will default to type ahead find functionality and interfere with the scrolling offset.

The interface is designed explicitly for Safari. Since this is a single computer installation under our control, I don’t have to worry about whether or not it works in IE or Firefox. I have no idea if it works in IE (ignorance truly is bliss), but it only half-works in Firefox. Firefox doesn’t want to load the flash player in Thickbox. I’ve experienced problems with Firefox ignoring the z-index on Flash elements in the past, so this is not surprising.

Remote buddy

Now that I had figured out how to deal with the browser, I had to figure out a way to have the remote interface with the browser by sending specific keystrokes for each button. After looking at several programs that do this with the Apple remote, I settled on Remote Buddy. It was the easiest to configure so that it would only work with specific applications. I configured it to send the left arrow key on left arrow press, right on right, enter on play and escape key on menu button press.

One problem with the remote was that our computer was an older mirror-door G4 tower, which does not have an infrared receiver. Thankfully, Remote Buddy supports third-party usb receivers, so I ordered one, and it worked like a charm. Microsoft does build good hardware.

Video out

Unfortunately, we weren’t able to track down a G4 that had S-video or composite out, so I also ordered a newer video card for that. We got an ATI Radeon 9200 which has an excellent control panel for TV-out settings, letting you manipulate all aspects of the size, stretch, overscan, etc.

Net connection

In this configuration, the videos stream in real time from YouTube’s servers. Our galleries have pretty good wireless saturation, and an airport card in this tower allows the machine to connect to the network. I spent some time looking into using Squid for caching the youtube videos, but it appears YouTube recently changed the way videos are embedded, using some sort of a time-based token in the URL, making caching difficult if not impossible. Another option would be to keep the videos loaded in the browser, but in a hidden element. However, since there’s no interface for controlling the YouTube embedded player with javascript, starting and stopping the player is not possible.

Putting it together

After connecting all these things, I cleaned up the interface a bit, changing the typeface to Avenir, choosing colors that worked better on the Television (don’t use white). I configured Safari to start in full-screen kiosk mode, which Saft enables. Safari and Remote Buddy were set to launch on login, and the machine is set to automatically log in. Additionally, I set up a simple shell script that checks each of them to make sure they’re running and if not, relaunches them. With relaunching the applications, it’s important to use the command-line version of Applescript, since that will cleanly deal with backgrounding the process and giving correct screen focus.

Doc Czypinski, who was the lead tech for installing the show, was able to figure out a way to glue a white plastic strip to the back of the remote. He was then able to cable the remotes to the gallery furniture so that they don’t walk away during the show.

Post-mortem

Now that the computer has been in the gallery for about a week, there are two things we’ve learned. First, visitors tend to expect things to be playing automatically on the TV, since that’s the way most media things are in galleries. When they do figure out they need to use the remote, they tend to click the big play button and not cycle through the videos to choose. I’ve attempted to help this by putting arrows on the screen to indicate left and right so that visitors know to use the arrows on the remote to scroll.

The second issue is that the wireless hasn’t been 100% reliable. As anyone who uses Wifi on a regular basis knows, it’s not always up 100% of the time, and the wifi in our gallery goes out once in a while. I’m looking at writing a script that will cycle the Airport card off and on if it looses connectivity, or displays a message on screen if there is extended connectivity loss. A wired connection would certainly be better, but the convenience of Wifi cannot be beaten.

Gallery photo by Cameron Wittig. Arm of Justin Heideman.

This presents a bit of a challenge however. Up until now we’ve only had four iPod Nano’s to worry about, and plugging a few into a computer or two to charge isn’t that big a deal. Now however we have 25 of them to deal with, and there certainly aren’t enough USB ports to go around. The goal was to find a way to charge most of the iPods, do it in a limited space, and do it for as cheap as possible.

My solution was to buy three USB hubs and use them just for charging. We don’t really need to have them connected to the computer to sync with, we just want the power. This turned out to be harder than I thought. I went through a few USB hubs trying to get the iPods just to charge off the supplied AC adaptor. Each hub I tried didn’t allow this. It would only charge when the hub was connected to a computer via USB. I can’t fathom a reason why they limited it like this, as the power comes from AC on the hub, not from USB. Whether the hub was connected to a computer should not really dictate whether power could be supplied to the device or not. Alas, there was no cost efficient way around this.

So I had no choice, if I wanted to charge via any hub, I had to connect the hub to a computer. Thankfully we did have a computer near where our iPod storage is. Except it only has two open USB ports, not the three I needed. Another stumbling block. But then the thought occured to daisy chain the hubs. In essence, the USB cable that was supposed to go to the computer for each hub would plug into one of the other hubs instead. The last in the chain would then plug into the computer. Basically we could connect all of the iPods to a computer with one USB cord, regardless of how many hubs we had. And that’s what we did, as it worked perfectly:

One interesting feature of this is it allows us to mount all of these iPods at once, as you can see here. This actually makes adding and editing content on all of them a breeze. So in the end, perhaps all of the troubles were a blessing.

Total cost for this: $60. It may not look the prettiest, but sometimes when you’re trying to be frugal, getting something that just works is what counts.

Pinkie is a network based electronics prototyping board. Pinkie has been designed to easily compose sensors and actuators that reside in different locations. Pinkies are inherently invisible, they hide behind the structures and only serve as facilitators to interface the physical world to the digital network.

Open I/O works like a peer-to-peer file sharing program, rather then sharing media files in your PC, you share data sensed from your physical environment. While Pinkies are organizing the low-level information (e.g., sensing the world), Open I/O is for higher concepts such as managing distributed devices, collaboration, and social networking.

This sounds to be a very interesting project. It seems like the floodgates are starting to open on small hardware devices that are open and easily programmable. Since there are so many of these devices, it seems only natural that they need social networking.

Porter continues to be a rockstar hardware-wise, but I’ve been having some trouble with the proxy/caching webserver running on it. Sure, it’s caching, but every so often it would grab a version of page and decide to keep it for 3 days instead of the directed 1 hour. At first I thought it was a one-time deal from switching some cache settings on the server, but it kept happening… Walker staff would make a change to some content, wait, but it would never show up on the live page. Trouble.

The problem was caused by a line stored in the cached HTTP header: Cache-Control: max-age=259200. (that’s three days worth of seconds) (I’m including details so google can pick this up and hopefully save some poor guy a frustrating morning.) After some serious digging it appears the mod_cache module we’re using was taking whatever Cache-Control header was being sent by the browser and saving it in the cached header! In other words, I had configured the server to cache things for a maximum of 1 hour, but all it took to blow that up was a browser (or spider?) sending a request saying it didn’t want anything older than 3 days. Our caching server held on to that “3-days” part and decided the whole page should be valid for that long. Totally. Wrong.

I debated making changes to the mod_cache source and recompiling, but I finally found an easier answer: “CacheIgnoreHeaders Cache-Control". This tells the caching module to ignore the problem lines, and it seems to be golden. I’ll let it run for a while and see…

[In further bad news, the US got creamed 3-0 by the Czech Republic in their World Cup opener. Not unexpected, but it doesn't bode well for getting past group play...]

The problem was obvious to anyone who’d used our website for any significant amount of time in the last year or two: as our technology on the backend increased, as new features and sites were added, the existing server was crawling to a slow and painful death. Frequent reboots (reboots! On Linux! The horror!) were required, and working in the CMS admin system was nearly intolerable. You could literally go get a drink of water while loading certain pages.

The solution was equally obvious: upgrade! But the execution proved quite labor-intensive – lots of tightly integrated bits and pieces that had to be unravelled carefully and put back together to create a semblance of a whole. Really.

My goal was to transition to the new server without any noticeable downtime, and it went as well as I could have hoped. There were some tense moments at the end – there’s really nothing like the feeling of pulling the plug (metaphorically) on an entire institution’s website and crossing your fingers you didn’t miss something when the new one comes up. Then a big sigh of relief when you realize of course you did, but it’s pretty minor, and wow! Look how much faster it runs!

So, welcome, Porter! You make your daddy proud. (ok, now I’m done.)

Of course, I say that like it’s going to be easy – I’m actually a bit nervous about the whole thing, it’s a lot of custom code and applications to port, not to mention our whole staging/production process. So I’ve got my work cut out for me, but it will be worth it in the end. Keep your fingers crossed!