Walker Blogs

New Media Initiatives

Continuous Deployment with Fabric

Posted February 22, 2012 at 9:25 am — Filed under:

textured brown fabric
We have been using Fabric to deploy changes to walkerart.org. Fabric is a library that enables a string of commands to be run on multiple servers. Though similar things could be done with shell scripts, we enjoy staying in one language as much as possible. In Fabric, strings are composed and sent to remote servers as commands over an SSH connection. Our Fabric scripts have been evolving over time with the project using the mentality: “If you know you are going to be doing something more than twice, script it!”

With Fabric we can tailor our deployments precisely. We deploy often with one of two commands:
fab production simple_deploy or fab production deploy.
simple_deploy simply pulls new code from the repo and restarts the web server.
deploy does many things, each of which can be executed independently, and is explained below.

The scripts we run go both ways, code goes up to the server and data comes back to the workstation. We have fab sync_with_production, which pulls the database and images. The images arrive locally in a directory specified by an environment variable or the default directory. Conventional naming schemes simplify variables across systems such as the database name. Except for some development settings, our workstation environments are identical to the production environment, which means we can replicate a bug or feature locally and immediately.

We have been collecting all of the commands we normally run on the servers into our fabfile. And then we can group them by calling tasks from other tasks. Our deployment consists of 12 tasks. With this Fabric task, one can deploy to the production or staging server with this one command:
fab production deploy.

This makes it incredibly simple to put code that is written on developer workstations into production in as safe and secure way. Here is our deployment in Fabric:

def deploy():
with cd(env.project):
run('git pull')
get_settings()
install_requirements()
production_templates()
celerybeat('stop')
celeryd('stop')
synccompress()
migrate()
gunicorn('restart')
celeryd('start')
celerybeat('start')

First the “with” blocks put us onto the remote server, into the right directory and within Python’s virtual environment. From there “git_pull” gets the new code which contains the settings files, and “get_settings” copies any new settings into place. The task called “install_requirements” calls on pip to validate our virtual environment’s packages against the setting file called requirements. All third party packages are locked to a version so we aren’t surprised by new “features” that have adverse effects. We use celery to harvest data from other sites so we make sure they are running with fresh config files. The task “syncompress” does our compressing of css and js, “migrate” alters the database per our migration files and gunicorn is the program that is running django.

It takes about 60 seconds for a new version of the website to get into production. From there it takes 0-10 minutes for the memcached values to expire before the public changes are visible. We are deploying continuously so watch closely for updates!

Optimizing page load time

Posted December 14, 2011 at 8:02 am — Filed under:

We launched the new walkerart.org late on December 1, and it’s been a great ride. The month leading up to (and especially the preceding week starting Thanksgiving Day, when I was physically moving servers and virtualizing old machines) was incredibly intense and really brought the best out of our awesome team. I would be remiss if I didn’t start this post by thanking Eric & Chris for their long hours and commitment to the site, Robin for guiding when needed and deflecting everything else so we could do what we do, and Andrew and Emmet for whispering into Eric’s ear and steering the front-end towards the visual delight we ended up with. And obviously Paul and everyone writing for the site, because without content it’s all just bling and glitz.

Gushy thanks out of the way, the launch gave us a chance to notice the site was a little … slow. Ok, a lot, depending on your device and connection, etc. Not the universally fast experience we were hoping for. The previous Walker site packed all the overhead into the page rendering, so with the HTML cached the rest would load in under a second, easy. The new site is heavy even if the HTML is cached. Just plain old heavy: custom fonts, tons of images popping and rotating, javascript widgets willy-nilly, third-party API calls…

Here’s the dirty truth of the homepage when we kicked it out the door December 1:

12/1: 2.6 MB over 164 requests. Load times are pretty subjective depending on a lot of things, but we had good evidence of the page taking at least 4+ seconds from click to being usable — and MUCH longer in some cases. Everyone was clearly willing to cut us some slack with a shiny new site, but once the honeymoon is over we need to be usable every day — and that means fast. This issue pretty quickly moved to the top of our priority list the Monday after launch, December 5.

The first thing to tackle was the size: 2.6 MB is just way too big. Eric noticed our default image scaling routine was somehow not compressing jpgs (I know, duh), so that was an easy first step and made a huge difference in download size.

12/5: 1.9 MB.

On the 6th we discovered (again, duh) lossless jpeg and png compression and immediately applied it to all the static assets on the site, but not yet to the dynamically-generated versions. Down to 1.8 MB. We also set up a fake Content Delivery Network (CDN) to help “parallelize” our image downloads. Modern browsers allow six simultaneous connections to a single domain, so by hosting all our images at www.walkerart.org we were essentially trying to send all our content through one tiny straw. Chris was able to modify our image generator code to spread requests across three new domains: cdn0.walkerart.org, cdn1, etc. This bypasses the geography and fat pipe of a real CDN, but does give the end user a few more straws to suck content through.

Requests per Domain

www.walkerart.org 26
cdn1.walkerart.org 24
cdn0.walkerart.org 24
cdn2.walkerart.org 21
f.fontdeck.com 4
other 7

 

By the 8th we were ready to push out global image optimization and blow away the cache of too-big images we’d generated. I’m kind of astounded I’d never done this on previous sites, considering what an easy change it was and what a difference it made. We’re using jpegoptim and optipng, and it’s fantastic: probably 30% lossless saving on already compressed jpegs and pngs. No-brainer.

12/8: 1.4 MB, almost half of what we launched with.

Next we needed to reduce the number of requests. We pushed into the second weekend with a big effort to optimize the Javascript and CSS. Earlier attempts using minify had blown up and were abandoned. Eric and Chris really stepped up to find a load order that worked and a safe way to combine and compress files without corrupting the contents. Most of the work was done Friday, but we opted to wait for Monday to push it out.

Meanwhile, I spent the weekend pulling work from the client’s browser back to the server where we could cache it site-wide. This doesn’t really impact bytes transferred, but it does remove a remote API call, which could take anywhere from a fraction of a second (unnoticeable) to several seconds in a worst-case scenario (un-usable). This primarily meant writing code to regularly call and cache all of our various Twitter feeds and the main weather widget. These are now served in the cached HTML and it’s negligible in the load time, instead of 200+ ms on average. It all adds up!

 

CSS Sprite for Header and Footer nav images (it has a transparent background, so it’s supposed to look like that):

 

 

So Monday, 12/12, we pushed out our first big changes to address the number of queries. Eric had combined most of the static pngs into a CSS Sprite, the javascript and CSS were reduced to fewer files, and the third party APIs were no longer called in the browser. Really getting there, now.

12/12: 1.37 MB, and 125 requests

Happily (as I was writing this) Eric just pushed out the last (for now) CSS sprite, giving us these final numbers:

12/13: 1.37 MB, and 110 requests! (down 53% and 67% respectively)

This isn’t over, but it’s gotten to the point of markedly diminishing returns. We’re fast enough to be pretty competitive and no longer embarrassing on an iPad, but there are a few more things to pick off around the edges. We’re heavier and slower than most of our museum peers, but lighter and faster than a lot of similar news sites. Which one are we? Depends which stats I want to compare. :)

We used the following tools to help diagnose and prioritize page loading time issues:
http://tools.pingdom.com/fpt/
https://developers.google.com/pagespeed/
http://developer.yahoo.com/yslow/

 

 

 

 

User testing using paper prototypes

Posted May 6, 2010 at 11:24 am — Filed under:

A few years ago I was trying to explain the concept of “fail early, fail often” to someone, and failing.  (see what I did there?  ;-)  They didn’t understand why you just wouldn’t take longer to build it right the first time.

Now that we’re deep in the process of redesigning our website, I am starting to see the real danger in that sort of thinking.  Despite all our best intentions, we’ve fallen into a trap of thrashing back and forth around certain ideas – unable to agree, unwilling to move forward until we “solve it”, and essentially stuck in the same cycle illustrated in this cartoon.

Click for the whole cartoon (scroll down a bit)

To try to help break the recent impasse on site navigation, we’re doing some simple user testing using paper prototypes of several ideas.  These are meant to be rough sketches to essentially pass/fail the “do they get it?” test, but they’re also giving us a ton of valuable little hints into how people see and understand both our website and our navigation.

An example of some paper prototypes for the navigation. (Don't worry, it's just a rough idea and one of many!)

Our basic process so far is to ask people (non-staff) for first impressions of the top nav: does it make sense?  Do they think they know what they’ll get under each button?  Then we show the flyouts and see if it’s what they expected.  Anything missing?  Anything doesn’t meet their expectations?  Finally we ask a few targeted “task” questions, like “where would you look if you wanted information about n work of art you saw in the galleries?”

Even this simple round of testing has revealed some clearly wrong assumptions on our part.  By fixing these things now (failing early) and iterating quickly, we can do more prototypes and get more feedback (failing often).  I’ll try to post updates as we proceed.

PS — Anyone else doing paper prototypes like this?  I think we all know we’re “supposed” to do quick user testing, but honestly this is the first time in years we’ve actually done something like it.

Setting up smartphone emulators for testing mobile websites

Posted April 23, 2010 at 3:33 pm — Filed under:

While developing the Walker’s mobile site, I needed to test the site in a number of browsers to ensure compatibility. If you thought testing a regular website was a pain, mobile is an order of magnitude worse.

Our mobile site is designed to work on modern smartphones. If you’re using a 4 year old Nokia phone with a 120×160 screen, our site does not and will not work for you. If you want to test on older/less-smart phones, PPK has a quick overview post that has some pointers. Even so, getting the current smartphone OS running is no piece of cake. So this post will outline how to get iPhone, Android, WebOS, and, ugh, BlackBerry running in emulation. Note: I left out Windows Mobile, as does 99% of the smartphone buying public.

Let’s knock off some low hanging fruit: iPhone

Getting the iPhone to run in emulation is very easy. First, you have to have a mac. If you’re a web developer, you’re probably working on a mac. You need to get the iPhone developer tools. You’ll have to register for a free Apple Developer account, agreeing to their lengthy and draconian agreement. Once that’s done, you can slurp down the humongous 2.3gb download and install it. Once installed, you’ll have a nice folder named Developer at the root of your drive, and navigate inside it and look for the iPhone Simulator.app. That’s your boy, so launch it and, hooray! You can now test your sites in Mobile Safari.

iPhone Simulator in Apple Developer Tools

The iPhone Simulator is by far the easiest to work with, since it’s a nice pre-packaged app, just like any other. And it is a simulator, not an emulator. The difference being, a simulator just looks and acts like an iPhone, but actually runs native code on your machine. An emulator emulates a different processor, running the whole host OS inside the emulator. The iPhone simulator runs an x68 version of Safari, and just links to the mobile frameworks, compiled in x86, on your local machine. A real, actual iPhone has all the same frameworks, but they’re compiled in ARM code on the phone.

Walker Art Center on the iPhone

Android

In typical google fashion, Android is a bit more confusing, but also more powerful. There are roughly three different flavors of Android out there in the wild: 1.5, 2.0, and 2.1. The browser is slightly different in each, but for most simple sites this should be relatively unimportant.

To get the Android Emulator running, download the Android SDK for your platform. I’m on a Mac, so that’s what I focus on here. You’ll need to have up-to-date java, but if you’re on a Mac, this isn’t a problem. Bonus points to google for being the only one to not require you to sign up as a developer to get the SDK. Once you have the file, unpack it and pull up the Terminal. Navigate to the directory, and then look inside the tools directory. You need to launch the “android” executable:

Very tricky: Launch the android executable.

This will launch the Android SDK and Android AVD Manager:

Android SDK and AVD Manager

The first thing you’ll probably want to do is go to Installed Packages and hit Update All…, just to get everything up-to-date. With that done, move back to Virtual Devices and we’re going to create a New virtual device:

Set up new Android Virtual Device

Name it whatever you want, I’d suggest using Android 2.1 as your target, give it a file size of around 200mb (you don’t need much if you aren’t going to install any apps) and leave everything else as default. Once it’s created, you can simply hit start, wait for it to boot, and you’re now running Android:

Android Emulator Running

Palm WebOS

Palm is suffering as a company right now, and depending on the rumors, is about to be bought by Lenovo, HTC, Microsoft, or Google. Pretty much everyone agrees that WebOS is really cool, so it’s definitely worth testing your mobile site on. WebOS, like the iPhone and Android, use Webkit as it’s browser, so things here are not going to be unexpected. The primary difference is the available fonts.

Running the WebOS emulator is very easy, at least on the Mac. First, you need to download an grab a copy of VirtualBox, and second, you download and install the Palm SDK. Both are linked from this page.

Installing VirtualBox is dead easy, and works just like any other OS X .pkg install process:

Then download and install the Palm WebOS SDK:

When you’re done, look in your /Applications folder for an app named Palm Emulator:

When you launch the emulator, you’ll be asked to choose a screen size (corresponding to either the Pre or the Pixi) and then it will start VirtualBox. It’s a bit more of a cumbersome startup process than the iPhone Simulator, but about on par with Android.

WebOS emulator starting up. It fires up VirtualBox in the background.

WebOS running.

BlackBerry

BlackBerry is the hairiest of all the smartphones in this post. Unless you know the Research In Motion ecosystem, and I don’t, it seems that there are about 300 different versions of BlackBerry, and no easy way to know what version you should test on. From what I can tell, the browser is basically the same on all the more recent phones, so picking one phone and using that should be fairly safe. RIM is working on BlackBerry 6, which is purported to include a WebKit based browser, addressing the sadness their browser causes in web developers everywhere.

The first thing you’re going to need to simulate a BlackBerry is a windows machine. I use VMWare Fusion on my mac, and have several instances of XP, so this is not a problem. The emulator is incredibly slow and clunky, so you’ll want a fairly fast machine or a Virtual Machine with the settings for RAM and CPU cranked up.

There are three basic parts you’ll need to install to get the BlackBerry emulator running: Java EE 5, BlackBerry Smartphone Simulator, and BlackBerry Email and MDS Services Simulator. Let’s start with Java. You need Java Enterprise Edition 5, and you can get that on Sun/Oracle’s Java EE page. I’ve had Java EE 5 and 6 on my windows machine for quite some time, so I’m not actually sure what version BlackBerry requires, but it’s one of them, and they’re both free. Get it, install it, and add one more hunk of junk to your system tray.

Now you need the emulators themselves: To get an emulator, head over to the RIM emulator page and pick a device. I went with the 9630 since that seems fairly popular and it was at the top of the list of devices to chose. I’d grab the latest OS for a generic carrier. You will have to register for a no-cost RIM developer account to download the file.

While you’re there, you’ll also want to grab the MDS (aka Mobile Data Service) emulator. This is what enables the phone to actually talk to the internet. To grab this, click on the “view all BlackBerry Smartphone Simulator downloads” link, and then choose the first item from the list, “BlackBerry Email and MDS Services Simulator Package”. Click through and grab the latest version.

Once the download completes, copy the .EXEs to windows and run them. You’ll walk through the standard windows install process, and when you’re done, you’ll be left with some new menu items. Let’s start the MDS up first, since we’d like a net connection. Here’s where you should find it:

I like to take screenshots of Windows to show how crazy bad it is.

And this is what it looks like starting up:

MDS running. It's a java app.

Now let’s start up the phone emulator itself:

BlackBerry 9630 Emulator

For me, it takes quite a while to start the phone, about a minute. I started off with a smaller VM instance and it was 5+ minutes to launch, so be warned. After it starts, you’ll be left with a screen like this:

You can’t use the mouse to navigate on the screen, which is crazy counter-intuitive for anyone who has used the other three phones mentioned in this post. Instead, you click on the buttons on screen or use your keyboard to navigate. Welcome to 2005. To get to the browser, hit the hangup button, then arrow over to the globe and hit enter. You can hit the little re-wrap/undo button to get to the URL field once the browser launches. Here’s what our site looks like:

Tips and tricks: How to convert ancient real media video into a modern h.264 mp4

Posted February 22, 2010 at 2:13 pm — Filed under:

First of all, I’d like to apologize to all the people on twitter that follow me and had to endure my ranting about the trials and tribulations of converting real media files: I’m sorry.

So let’s say you have a pile of real media video that was recorded sometime earlier in the decade when real video was still relevant, but you realize any sane person these days doesn’t have RealPlayer installed and can’t view it. What you really want is that video to exist in an mp4 so you can stream it to a flash player, or eventually use <video> in html5 (once they work that codec stuff out). If you do a little googling on how to convert real video into h.264 mp4, you’ll find lots of programs and forum posts claiming they know to do it. But it’s mostly programs that don’t actually work and forum posts that are no longer relevant or strewn with blocking issues.

Thankfully, there is a better way, and I will lay it out for you.

Step one: Download the actual media
In our scenario, you have a list of 80 or so real media files that you need to convert. The URLs for those things probably look something like

http://media.walkerart.org/av/Channel/Gowda.ram

. If you were to download that .ram file, you’d notice that it’s about 59 bytes; clearly not enough to be the actual video file. What it is, is a pointer to the streaming location for the file. If you open up that .ram file in a text editor, you’ll see it points to

rtsp://ice.walkerart.org:8080/translocations/media/Gowda.rm

, which is the location for our real media streaming server here at the Walker.The thing we really want is the .rm file, but it can be a little hard to get via rtsp. Since we’re not stream ripping someone else’s content (that would be wrong, dontcha know), we can just log in to the server and based on that file path it’s looking for, grab the .rm via SCP or a file transfer mechanism of our choice. I happened to know that all our .rm files are actually accessible via HTTP, so I just did a little find/replacing in the URLs and built a list with

wget

to download them.

Step two: Convert the real media files to mp4
If you were trying to do this back in the day this would be a major pain. You’d have to use

mencoder

and the longest, most convoluted command-line arguments you’ve ever seen. Thankfully, Real recently came out with an updated version of RealPlayer that has a handy little thing in it called RealPlayer Converter. Sounds too good to be true, right? It is.

For larger files, it only works well on Windows, and it doesn’t give you a lot of options for encoding. The mac version will hang at 95% encoding for most files, and that’s pretty annoying. Save yourself the trouble and use a Windows box. Once you have RealPlayer installed, open up the converter, drag your .rm files in, and set the conversion settings. Depending on what your original sources are, you might need to fiddle with the options. I used the h.246 for iPod and iPhone, because that fit the size (320×240) that my source files were. I cranked down the bitrate to 512kbps and 128kbps, because my source .rm files were about 384kbps and 64kbps to start with. This will give you a .m4v file, which is basically a .mp4 file with a different extension, but should work OK for most stuff.

Queue everything up and let it rip. On a two year old PC, it took about a day to process 48 hours worth of video.

Step three: Check your files
This is the part where your curse a little bit, realizing that in half the video you just encoded, the audio is out of sync with the video. This is a common problem when converting real video, and Real’s own tool doesn’t do a good job of handling it, never mind the fact that if you just play the video in RealPlayer, it plays in sync just fine. If you were to open the .m4v up in QuickTime Pro and look at the movie properties, you’d see something like this:

Notice the problem there? The video and audio tracks have different lengths, the video track being shorter than the audio. There is a way to fix this.

Step four: Synchronize the audio and video
There is a handy mac program that helps you fix just this synchronization issue. It’s called QT Sync. Operation is pretty simple. You open up a video file and it shows you fiddle with the video/audio offset until it is synced up. Here’s a screenshot:

Ideally, proper sync will occur when the number of frames is equal for both the audio and video. In my experience, most of my videos were synced when the video frame count was about 10 short of the audio frames, but your mileage may vary. Some of the videos I worked with would also slowly drift out of sync over time, and unfortunately, there isn’t a way to fix those. Just sync them up the the beginning and rest easy knowing you’ve done what you can.

Step four-and-a-half: Save the video
This is where things get tricky again. How you save the video depends on what you’re going to do with it. If your output target is just for iPods and iPhone, and you’re not going to be streaming it from a streaming server, you have it good. If you are planning on streaming, skip to step five. You can save the video from QT Sync without re-encoding. You’ll just be writing the audio and video streams in a new .mp4 wrapper, this time with a proper delay set up for one of the streams. To save the .mp4, you file > export, and use “Movie to mpeg-4″ as the format. Go into the options, and you want to use “Pass through” as the format for both audio and video, and do not check anything in the streaming tab. Here’s what it looks like:

This will take a moment to write the file, but it won’t re-encode. If you open the resulting mp4 up in QuickTime Pro and look at the properties, you should see something like this:

Note how the video track has a start time 6 seconds later than the audio. This is good and should play in sync. If Rinse and repeat for each of your videos that is out of sync and you’re done.

Step five: Save the video
If you’re reading this, it’s because you want to take your converted video and stream it to a flash player, using something like Adobe Streaming Media Server. If you were to take that synced, fixed up mp4 from step 4.5 and put it on your streaming media server and started streaming, you’d notice that the audio and video were out of sync again. See, Adobe Streaming Media Server doesn’t respect the delay or start time in an .mp4 file. I didn’t test other streaming servers like Wowza, but I’m guessing the suffer from the same issue. It sucks, but I can kind of see it making sense for a streaming server to expect them to be in sync.

Instead, we are stuck fixing the video the hard way. You have the video sync’d up in QT Sync, but instead of saving it as a .mp4 as in step 4.5, save it as a reference movie with a .mov extension. We’re doing this because we’ve got to re-encode the video, again, essentially hard-coding the audio or video delay into the streams, rather than just the .mp4 wrapper.

Step six: Encode the video (again)
So, now you have a bunch of .mov referrence files that are ready to be batch processed. You can use whatever software you like to do this, but I like MPEG Streamclip, which I wrote about a little in this post about iTunes U. It is way faster than Compressor, and it does batch processing really nicely.

You want to use settings that are similar to what your file is already using. I outlined that above, but here’s what the settings screen looks like:

Yes, you’re losing a bit of quality here encoding the video for the second time, but there isn’t a way around it. In looking, I couldn’t notice a difference between the original .rm file, the first version m4v, and the fixed and synced .mp4. There is no doubt some loss, but it is an acceptable trade-off to get a usable video format.

Some thoughts on preserving Internet Art

Posted July 13, 2009 at 2:49 pm — Filed under:

aenWe’re in the process of retiring our last production server running NT and ColdFusion (whew!), and this means we needed to get a few old projects ported to our newer Linux machines.  The main site, http://aen.walkerart.org/, is marginally database-driven: that is, it pulls random links and projects from a database to make the pages different each time you load.  The admin at the time was nice enough to include MDB dump files from the Microsoft Access(!) project database, and the free mdbtools software was able to extract the schema and generate import scripts.  Most of this page works as-is, but I had to tweak the schema by hand.

After the database was ported to MySQL, it was time to convert the ColdFusion to PHP.  (Note: the pages still say .cfm so we don’t break links or search engines – it’s running php on the server)  Luckily the scripts weren’t doing anything terribly complicated, mostly just selects and loops with some “randomness” thrown in.  I added a quick database-abstraction file to handle connections and errors and sanitize input, and things were up and running quickly.

… sort of.  The site is essentially a repository of links to other projects, and was launched in February 2000.  As you might imagine there’s been some serious link rot, and I’m at a bit of loss on how to approach a solution.  Steve Dietz, former New Media curator here at the Walker, has an article discussing this very issue here (ironically mentioning another Walker-commissioned project that’s suffered link rot.  Hmm.).

One strategy Dietz suggests is to update the links by hand as the net evolves.  This seems resource-heavy, even if a link-validating bot could automate the checking — someone would have to curate new links and update the database.  I’m not sure we can make that happen.

It also occurred to me to build a proxy using the wayback machine to try to give the user a view of the internet in early 2000.  There’s no API for pulling pages, but archive.org allows you to build a URL to get the copy of a page closest to a specific date, so it seems possible.  But this is tricky for other reasons – what if the site actually still exists?  Should we go to the live copy or the copy from 2000?  Do we need to pull the header on the url and only go to archive.org if it’s a 404 to 500?  And what if the domain is now owned by a squatter who returns a 200 page of ads?  Also, archive.org respects robots.txt, so a few of our links have apparently never been archived and are gone forever.  Rough.

In the end, the easy part was pulling the code to a new language and server – it works pretty much exactly like it did before, broken links and all.  The hard part is figuring out what to do with the rest of the web…  I do think I’ll try to build that archive.org proxy someday, but for now the fact it’s running on stable hardware is good enough.

Thoughts?  Anyone already built that proxy and want to share?

Build a bridging firewall (cheap!)

Posted June 22, 2009 at 1:07 pm — Filed under:

New Media has a number of development servers located in-house where we get stuff done before releasing it out into the wild.  Until last week these were protected by an aging OpenBSD firewall running packet filter and all was well until midweek when the motherboard failed.  Not having a spare on hand, I was scrambling for a solution.

Linksys wireless router

Linksys wireless router

Being familiar with the dd-wrt project, I was pretty sure I could build a firewall out of a Linksys router.  We went with the WRT54GL, currently as cheap as $50 on Amazon.  (We bought local so we’d have it sooner, and it was a bit more).

The first step after flashing the firmware with the latest dd-wrt build (v24-sp2) was to take off the antennas and turn off the radio.  The last thing I want for the firewall is to be broadcasting an SSID and allow wireless associations.  This actually requires a startup script on the router, with a line to remove the wireless module so it won’t try to reenable itself:

wl radio off
wl down
rmmod wl

Good start.  Next I needed to bridge the WAN port with the LAN ports, which ended up being a struggle until I found the easy options in the dd-wrt GUI.  First, set the LAN to use a static IP and make sure you can connect via another machine to configure it.  You’ll also need to enable SSH access and remote configuration – but be sure to lock this down once the firewall is running!

Once you have the LAN configured, you need to set the WAN connection type to “disabled”.  This will give you a checkbox to bridge the LAN and WAN:  “Assign WAN port to switch”.  Lastly, under Advanced Routing set the Operating Mode to “Router” so it stops trying to do NAT.  Apply these settings, and you’ll basically have an expensive dumb switch – all traffic shows up on every port, and there’s no logic at all.  We’re halfway there.

Being unfamiliar with iptables (we use OpenBSD and pf for firewalls around here), I was under the impression that iptables rules would work in a bridging environment.  This is not the case: bridged packets don’t reach iptables at all!  The best I could do was block everything (manual restart needed), or otherwise blow up the configuration (manual restart needed) as I tried to mess with the bridge.  This was an incredibly frustrating learning curve as everything I could find made it sound like this was the way to configure a firewall in Linux, but it just wasn’t working.

Note to keep you sane: don’t do any of this testing in the startup scripts or you’ll brick your router, guaranteed.  Do it all from the command line with a known-good startup.  That way it’s a simple (but annoying) power cycle to get things back up.

The trick, it turns out, is a kernel module called ebtables.  Luckily, this is included in the dd-wrt build, but it’s not turned on by default!  Add this to your startup script:

insmod ebtables
insmod ebtable_filter
insmod ebt_ip.o

And, ta-da, all your iptables rules will start impacting packets!  Now it’s just a matter of configuring the firewall rules.  We’re using something like this:  (vlan0 represents the LAN ports, and vlan1 is the WAN port)

# drop everything by default:
iptables -P FORWARD DROP
# clear the old rules:
iptables -F FORWARD
# forward stuff that's established already
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
# let connections out:
iptables -A FORWARD -i vlan0 -m state --state NEW -j ACCEPT

# firewall access rules
iptables -F INPUT
# WAC ips can get to fw:
iptables -A INPUT -p tcp -d 1.2.3.4 -s 4.3.2.1/24 -j ACCEPT
# drop everything else!
iptables -A INPUT -p tcp -d 1.2.3.4 -j DROP

# ... snipped all the actual access rules and packet flood protection ...

The only trick here is the last few lines which limit access to the firewall machine itself.  We can’t use the FORWARD rules since these packets are destined for the internal hardware and not forwarded, but we do need to limit access via the INPUT chain.  In this example the firewall has IP 1.2.3.4 and the network I want to access it from has 4.3.2.x.  That way I can leave the firewall’s remote access turned on and limit it to our network.  (because there’s no terminal access you can’t make it a truly transparent bridge or you’d never be able to change the config!)

I admit I’m a bit nervous posting some of this in case there’s a glaring security hole, but it seems good to me.  Anyone see anything they’d like to warn me about before we get hacked?

And there you have it!  For the cost of a cheap router and some time (not much, since you can just follow these notes!) you have a full-featured bridging firewall running on dedicated hardware.  With a little extra work it would be easy to get VPN running and much more…  I’m hoping for years of service from this little guy!

( Hat tip another DIY firewall solution that I’d really like to try someday. )

Walker Channel live streaming with flash video

Posted February 5, 2009 at 6:03 pm — Filed under:

If you’ve tuned into the live streaming events the Walker Channel has carried in the past, you have been forced to use Real Player to watch. Real was great back in the day when the Walker Channel was launched, but in 2009 it is a little dated. Flash streaming is much more convenient, and the VP6 codec flash offers is quite good. 

For tonight’s The Art of the Book panel discussion, we will be using ustream.tv to stream the event, rather than Real Player. No fancy plugins or separate applications required. It is also free, and doesn’t require us to run our own Real Media server. It will also help us decrease the turn-around on getting a recorded event into the Walker Channel, iTunes U and YouTube. None of this is rocket surgery, of course. Other places, like The UpTake, have been using free straming services very effectively, we’re just a little late getting on the bandwagon. 

We’re doing tonight’s lecture is a test of ustream, and we will be working out any kinks. We’ve done some testing already, but haven’t used it in a live setting where anyone other than a handful of people have been watching. 

If you’re watching and run into any problems, let us know. Shoot me an e-mail (click on my name to get the address), hit us on twitter, post here, or join the chatroom on ustream.

Installing a Lighttpd proxy

Posted November 22, 2008 at 12:39 pm — Filed under:

It had been a slow build, but an incident a few weeks ago made it finally clear: the Walker website was becoming a victim of its own success.  A post on the Teens site contained a picture of the Joker for the then-upcoming Batman movie, and as Halloween approached we found ourselves on the front page of Google image search as people began looking for costume ideas.  The exponential traffic was crippling our web server:

The biggest problem was simply that Apache is heavy.  It’s resource-intensive, especially when you are running several modules as we were – PHP, proxy, cache, etc.  The teens site is especially difficult since it runs as a combination of a blog (PHP on Apache 2) and .wac pages (mod_perl & Axkit).  Every hit to the Joker post – even if the page was cached – would tie up a number of Apache processes as it served the style sheets, images, and javascript to support the page.  We were reaching our MaxClients setting but unable to raise it without running out of memory for our other more intensive servers (mod_perl and postgres, I’m looking at you…).

As this diagram shows, it’s nothing but Apache servers, and it just wasn’t scaling to meet our current demand.

The approach was two-fold: some quick auditing and re-writing of the worst offending .wac pages’ SQL to speed up the slow pages, and yet another web server in front of everything.  It was a no brainer to pick Lighttpd, or “Lighty”.  It’s written to do one thing – serve static content – and do it extremely fast.  Fortunately it can also proxy requests, so it was a pretty simple matter to reassign some ports and write a few rules to route all requests through Lighty.

The end result is astonishing.  Our server hums along happily under even the most intense traffic we can throw at it (the email blast for the British Television Advertising Awards) and doesn’t even start to complain.  Moving the bulk of the requests to the extremely fast and resource-light server meant we could devote more resources to quickly processing the slow pages (mod_perl).  Between the SQL tuning and the extra resources, the bulk of these pages are now served between 2 and 10(!!!) times faster!

The lesson here, for anyone with an Apache server creaking and groaning under increased traffic, is to stop waiting and install Lighty.  If your site is PHP-based, you can run this as a fast CGI module from Lighty and do away with Apache altogether.  You can also use Lighty to stream (and “scrub”!) flv and mp4 video files.  (I’m using both of these techniques for the new ArtsConnectEd.)

The only caveat: be careful as you look for examples on the web.  Remarkably, it seems there are many confused webmasters who expect to see a performance boost by putting Lighty behind their struggling Apache.  This will not help at all, and in fact will probably make things worse.  Lighty has to be first in the chain to take the load off Apache.

Enjoy the speed!  I know our server is enjoying the breathing room!

Initial info on Max/MSP 5 emerges

Posted September 28, 2007 at 12:31 pm — Filed under:

David Zicarelli of Cycling ’74 has posted some initial notes on what’s new and different in Max 5. He

For current users, I would describe Max 5 as analogous to Apple’s transition from Mac OS 9 and Mac OS X. At some point, Apple decided that the technological foundation of their operating system was unsustainable, and required a completely new approach. We came to the same conclusion about many aspects of Max, and especially about the graphical interface — often the most complicated and difficult system within any large application software project.

Max was based on the way the Macintosh worked in 1987. Since then, a lot of things have changed about graphical interfaces, file systems, and pretty much everything else. As a result, the assumptions of 1987 were simply too deeply embedded to keep Max going for another 20 years with the same internal codebase. This became increasingly apparent in recent years, as we seemed be doing nothing but patching Max to keep it working with the latest hardware and software.

I’ve wanted to make Max better, but recently most of my work has been the drudgery of making it operate on OS X, or on Windows, or on Intel processors. While I’ve been doing this, I’ve also been accumulating ideas for what I would do once I got over all this kind of work.

Well, that day did come when we finally finished the Intel port of the OS X version, although it took about a year longer than I thought it would. Once I was able to clear that off my desk, I began organizing my thoughts about what Max requires to survive another 20 years.

He mentions the possibility of Max being available on Linux, changes to the way some objects work, and updates to the GUI. In respect to the GUI, my friend Paul puts it best:

I’m excited to see what their new interface changes will entail. I kind of liked it’s raw ugliness, same way I really liked Slashdot’s 1993 ugliness before they shined it up. Its like a really ugly comfortable couch that you know you should get rid of eventually.

(Obviously, Max is for visual thinkers)

Next