New Media Initiatives

New Media has a number of development servers located in-house where we get stuff done before releasing it out into the wild.  Until last week these were protected by an aging OpenBSD firewall running packet filter and all was well until midweek when the motherboard failed.  Not having a spare on hand, I was scrambling for a solution.

Linksys wireless router

Being familiar with the dd-wrt project, I was pretty sure I could build a firewall out of a Linksys router.  We went with the WRT54GL, currently as cheap as $50 on Amazon.  (We bought local so we’d have it sooner, and it was a bit more).

The first step after flashing the firmware with the latest dd-wrt build (v24-sp2) was to take off the antennas and turn off the radio.  The last thing I want for the firewall is to be broadcasting an SSID and allow wireless associations.  This actually requires a startup script on the router, with a line to remove the wireless module so it won’t try to reenable itself:

wl radio off
wl down
rmmod wl

Good start.  Next I needed to bridge the WAN port with the LAN ports, which ended up being a struggle until I found the easy options in the dd-wrt GUI.  First, set the LAN to use a static IP and make sure you can connect via another machine to configure it.  You’ll also need to enable SSH access and remote configuration - but be sure to lock this down once the firewall is running!

Once you have the LAN configured, you need to set the WAN connection type to “disabled”.  This will give you a checkbox to bridge the LAN and WAN:  “Assign WAN port to switch”.  Lastly, under Advanced Routing set the Operating Mode to “Router” so it stops trying to do NAT.  Apply these settings, and you’ll basically have an expensive dumb switch - all traffic shows up on every port, and there’s no logic at all.  We’re halfway there.

Being unfamiliar with iptables (we use OpenBSD and pf for firewalls around here), I was under the impression that iptables rules would work in a bridging environment.  This is not the case: bridged packets don’t reach iptables at all!  The best I could do was block everything (manual restart needed), or otherwise blow up the configuration (manual restart needed) as I tried to mess with the bridge.  This was an incredibly frustrating learning curve as everything I could find made it sound like this was the way to configure a firewall in Linux, but it just wasn’t working.

Note to keep you sane: don’t do any of this testing in the startup scripts or you’ll brick your router, guaranteed.  Do it all from the command line with a known-good startup.  That way it’s a simple (but annoying) power cycle to get things back up.

The trick, it turns out, is a kernel module called ebtables.  Luckily, this is included in the dd-wrt build, but it’s not turned on by default!  Add this to your startup script:

insmod ebtables
insmod ebtable_filter
insmod ebt_ip.o

And, ta-da, all your iptables rules will start impacting packets!  Now it’s just a matter of configuring the firewall rules.  We’re using something like this:  (vlan0 represents the LAN ports, and vlan1 is the WAN port)

# drop everything by default:
iptables -P FORWARD DROP
# clear the old rules:
iptables -F FORWARD
# forward stuff that's established already
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
# let connections out:
iptables -A FORWARD -i vlan0 -m state --state NEW -j ACCEPT

# firewall access rules
iptables -F INPUT
# WAC ips can get to fw:
iptables -A INPUT -p tcp -d 1.2.3.4 -s 4.3.2.1/24 -j ACCEPT
# drop everything else!
iptables -A INPUT -p tcp -d 1.2.3.4 -j DROP

# ... snipped all the actual access rules and packet flood protection ...

The only trick here is the last few lines which limit access to the firewall machine itself.  We can’t use the FORWARD rules since these packets are destined for the internal hardware and not forwarded, but we do need to limit access via the INPUT chain.  In this example the firewall has IP 1.2.3.4 and the network I want to access it from has 4.3.2.x.  That way I can leave the firewall’s remote access turned on and limit it to our network.  (because there’s no terminal access you can’t make it a truly transparent bridge or you’d never be able to change the config!)

I admit I’m a bit nervous posting some of this in case there’s a glaring security hole, but it seems good to me.  Anyone see anything they’d like to warn me about before we get hacked?

And there you have it!  For the cost of a cheap router and some time (not much, since you can just follow these notes!) you have a full-featured bridging firewall running on dedicated hardware.  With a little extra work it would be easy to get VPN running and much more…  I’m hoping for years of service from this little guy!

( Hat tip another DIY firewall solution that I’d really like to try someday. )

… blog about it in May!

Museums and the Web 2009 wrapped up with a challenge to all the inspired delegates: use the energy and ideas generated here to get one thing done in April.  (The idea being that many small steps build momentum, and it’s too easy to ignore the small upgrades we should constantly be pushing out.)

Yesterday I pushed out a few small upgrades to our aging collection site:

You can now limit your search to objects that are On View

What works by Dan Flavin can you come see right now?

OpenSearch capable

Can’t get enough of our collection?  Add it to your browser’s built-in search box!  When you’re on the Collection site, you should be able to pull down your browser’s search field and add “Walker Art Center”.

Developers (Piotr!): you can now use the Walker collection in your Yahoo Pipes tool without having to scrape the results!  Not an API (yet), but a good step.  Check out the XML for ideas.

Bring it all together:

You’re a busy person.  You’d love to come see Chuck Close’s Big Self-Portrait, and you know the Walker’s got it in their collection, but you see it’s not on view.  You don’t have time to check our website every day, so how will you ever know when it goes on display?  Easy:  build a search that finds Big Self-Portrait, then turn on the “On View” flag.  The object disappears (not on view), but you can subscribe to the OpenSearch RSS feed for this query (click the icon).  Now, when Big Self-Portrait is available to see in the galleries, the object will show up in your RSS reader!  (note: I picked this painting randomly.  I make no guarantee about seeing it in the galleries any time soon.  :)

So, baby steps.  Get one things done that opens more doors.

#didonethinginapril (I tag Andrew at the MIA to get one thing done in May!)

Charlie Moad (developer at IMA) kicks off the session with a discussion of cloud computing, the advantages and disadvantages.  One of his most compelling arguments in a non-technical sense is the incredible energy efficiency of these large data centers: their cooling system and power use are at levels we can’t approach in our co-located server rack. Google is approaching a 1.1:1 ratio of cooling to power consumption. They’ve recently documented their cooling and datacenter practices here.

Other advantages Charlie mentioned for using Cloud computing:

• Scalability
• Pay as you go. This is the big benefit. You use what you need when you need it, also helping the efficency.
• No hardware to administer. No downtime. This makes sysadmins very happy.

Some disadvantages are:

• Security. (Not sure on this… don’t recall amazon or google having any big issues with security. This is in the hands of us doing their jobs and setting proper permissions.)
• Portability. AWS and Google App Engine (GAE) are proprietary systems. GAE has more issues in this realm than AWS.

One other thing to note about Google App Engine that Charlie didn’t mention is that GAE is a spec, and from what I’ve heard from various python people, Google very much wants it to be implemented by others. There is already an open source implementation of AppEngine called AppScale. And Joyent has an implementation called ReasonablySmart.

IMA is using Amazon Web Services (AWS) for hosting ArtBabble. A simple breakdown of their usage thus:

• EC2 instances for transcoding video
• S3 and CloudFront for storing video and media files (images/js/etc)
• Wowza streaming server running on EC2 for streaming video

Cloud computing structure for ArtBabble

Charlie had a nice slide I don’t remember being in the paper: a diagram of where these services sit in the cloud (storage vs service) and what the end user’s browser is actually talking to at any time. It sounds like changing the number of wowza instances is still a manual process, but I imagine it could be automated.

The stats are impressive: 40,000 video views since launch 9 days ago, and 3,500 registered users.  They’re cleverly using Google / Yahoo sign-ins to create OpenID accounts, without telling people it involves OpenId.  Uptake is much higher by hiding the technology on this process…  Also impressive is the cost, or lack thereof: they’re able to run ArtBabble for the same cost as their internal website.

Charlie closes by mentioning a few recent advances in Amazon’s hosting that allows essentially pre-paying for a year’s service at a much discounted rate.

I think I’m not the only webmaster in the audience who is thinking “we have to move our sites into the cloud,” but also concerned about finding the time to do so.  This paper and presentation have gone a long way towards answering some questions I haven’t been able to research fully.

Jusitn Heideman also contributed to this post.

It had been a slow build, but an incident a few weeks ago made it finally clear: the Walker website was becoming a victim of its own success.  A post on the Teens site contained a picture of the Joker for the then-upcoming Batman movie, and as Halloween approached we found ourselves on the front page of Google image search as people began looking for costume ideas.  The exponential traffic was crippling our web server:

The biggest problem was simply that Apache is heavy.  It’s resource-intensive, especially when you are running several modules as we were - PHP, proxy, cache, etc.  The teens site is especially difficult since it runs as a combination of a blog (PHP on Apache 2) and .wac pages (mod_perl & Axkit).  Every hit to the Joker post - even if the page was cached - would tie up a number of Apache processes as it served the style sheets, images, and javascript to support the page.  We were reaching our MaxClients setting but unable to raise it without running out of memory for our other more intensive servers (mod_perl and postgres, I’m looking at you…).

As this diagram shows, it’s nothing but Apache servers, and it just wasn’t scaling to meet our current demand.

The approach was two-fold: some quick auditing and re-writing of the worst offending .wac pages’ SQL to speed up the slow pages, and yet another web server in front of everything.  It was a no brainer to pick Lighttpd, or “Lighty”.  It’s written to do one thing - serve static content - and do it extremely fast.  Fortunately it can also proxy requests, so it was a pretty simple matter to reassign some ports and write a few rules to route all requests through Lighty.

The end result is astonishing.  Our server hums along happily under even the most intense traffic we can throw at it (the email blast for the British Television Advertising Awards) and doesn’t even start to complain.  Moving the bulk of the requests to the extremely fast and resource-light server meant we could devote more resources to quickly processing the slow pages (mod_perl).  Between the SQL tuning and the extra resources, the bulk of these pages are now served between 2 and 10(!!!) times faster!

The lesson here, for anyone with an Apache server creaking and groaning under increased traffic, is to stop waiting and install Lighty.  If your site is PHP-based, you can run this as a fast CGI module from Lighty and do away with Apache altogether.  You can also use Lighty to stream (and “scrub”!) flv and mp4 video files.  (I’m using both of these techniques for the new ArtsConnectEd.)

The only caveat: be careful as you look for examples on the web.  Remarkably, it seems there are many confused webmasters who expect to see a performance boost by putting Lighty behind their struggling Apache.  This will not help at all, and in fact will probably make things worse.  Lighty has to be first in the chain to take the load off Apache.

Enjoy the speed!  I know our server is enjoying the breathing room!

It’s Monday morning, and like many of you I’m sitting down to my computer for the day. I’ll check my email (office and GMail), see what’s new in the blogosphere (Google Reader), maybe catch up on the world a bit (Google News), and finally start my day of programming. (searching Google to see if anyone’s already written the code I’m working on ;)

You may notice a trend: Google. What would my world be like without Google? It’s an interesting thought experiment.

… except yesterday morning, it was a reality. Details are still sketchy (it may have just been Comcast?) but for an hour Sunday morning, I couldn’t reach a single Google domain. No GMail, no RSS, no search. And, in a creepy side effect I hadn’t anticipated, MANY sites were either slow, broken, or eternally loading as my browser tried in vain to pull the Google Analytics or Adsense code for the page. In short, the Internet was Broken.

It was amazing to watch myself try to remember how to search without Google. Maybe… Yahoo? No, I’ll try Ask.com. Turns out the results are terrible, at least for programming-related searches. A friend called looking for directions to a farmers’ market, and it took me a bit to remember the world beyond Google Maps. Doesn’t… Microsoft do maps? Hmm, who else… If only I could search!!

Twitter was still up, and via TwitScoop.com I could see a big spike in Google-related tweets, but I have yet to see any official word on this. Clearly big companies have downtime (Amazon was down for a few hours recently), but still… It may be time to rethink my utter dependency on “The Google”.

Time to return the favor to the blogosphere and the mplayer-users mailing list, where most of this information was painstakingly discovered. I’m working on a phase of the new ArtsConnectEd site where we’re trying to automate the importing of all of our various media types into two standard, embeddable formats: mp3 and flv. The source media is in everything from Real Audio and Real Video to different flavors of quicktime (mp4, etc), old mp3 codecs, some avi, and even wav files. We also need to generate thumbnails for all the video, and properly detect the edge cases where we have an audio-only quicktime file that wants to be a video but clearly should actually be an mp3 audio file.

The solution is ffmpeg and mplayer / mencoder, and the amazing thing is it’s almost as easy as just throwing a file at it and telling it what format you want. The developers of these tools have done a truly amazing job.

Things get much trickier when you introduce Real Media into the mix. An unfortunate number of our Channel videos are in this format, so it’s something we have to solve. After many attempts and failed encodings - including one hair-pulling episode where it turned out the audio was actually out of sync in the original file and not in the transformed version - I believe I have a “good enough for now” command line formula for converting Real Media to flv. (ffmpeg actually can’t handle .rm files, so this is all done in mencoder)

mencoder realmediafile.rm -ni -o flvoutput.flv -oac mp3lame -lameopts abr:br=56 -srate 22050 -ovc lavc -lavcopts vcodec=flv:vbitrate=300:mbd=2:mv0:trell:v4mv:cbp:last_pred=3 -fps 30.000 -ofps 24 -mc 1 -of lavf -lavfopts i_certify_that_my_video_stream_does_not_use_b_frames

Some of that is over my head and I just copied it (last_pred=3??) but the real key for this process seems to be knowing and accurately setting the input stream’s framerate using -fps xx.xx. Without that, the real demuxer will sometimes guess wrong and get out of sync, resulting in an unpleasant hung process as it comes across the next data chunk. I only noticed this happening with RV40 on multirate files, RV30 seemed solid. You can pull the info about the stream, including framerate and original dimensions, using mplayer:

mplayer -identify -frames 0 -vc null -vo null -ao null realmediafile.rm

Even cooler, in the cases where the .rm file isn’t local, both of those commands work by feeding them a stream like rtsp://server/realmediafile.rm!

With any luck, someone will find this page and save themselves a day of frustration. Or someone will find this page and notice I’m doing something wrong and correct me!

Some great conversation happening in the comments of my writeup of the Search session at MW2008, and it made me remember something I wanted to bring up at the conference but forgot. Namely, the concept of “master metadata”, or the idea that there’s one authoritative version of the metadata describing an object.

This came up for me in the session the MFA and MIT did on sharing their data for a new subsite: they mentioned the data was being “augmented” on the final site, and that someday they’d be interested in getting this extra information back into their main repository.

The problem’s immediately obvious: with all of the proposed sharing and opening up of our data, presumably to allow others to weigh in on it and add their voice, there are often situations where institutions would like to have some of this new data. For instance, we’re building a new version of ArtsConnectEd and intend to allow museum educators to variously tag, comment on, and draw relationships between objects. This will almost certainly be “good data”, stuff that would be valuable to integrate in our internal collection database.

The question is, how? Once your data is available for sharing, and someone actually builds something good with it and enhances it, is there a way to get that new data back into the source? Is there / should there be a way to tag metadata as “original source” or “augmented”? Should we be asking anyone harvesting our data to push back their changes for us to audit and possibly include?

Anyone solved this? Seb, are you getting info back from Flickr Commons you can then add to your internal database? Phil / Jenna, any thoughts on how to get that extra data back?

This session has been great for me, as this is very much where my head is at right now with ArtsConnectEd… My live notes follow:

Brian Kelly chairs a session on Search, announcing that with the smaller size both speakers are willing to make this a bit more workshop-like. Terry Makewell starts by introducing his project: 9 partners making up the National Museums Online Learning Project. He goes over some of the goals of the project, and the current state of things, and the realization that some sort of federated search was needed to span the partners’ collections.

How to do the federated search? Multi-institution project meant different technical teams, different technologies, and limited resources in some cases. See the paper for more details, but the two technologies they considered most carefully are OAI/PMH and Opensearch.

OAI, the path we’re going down with ArtsConnectEd, uses a central repository and runs the searches there. Opensearch spools the searches out to each institution and then re-orders them locally and returns the result.

Opensearch fit the project requirements and timeline most efficiently, so that was their choice. He discusses their prototyping effort: scraping search results to generate the RSS for Opensearch. They now have a single page with a configuration file they can drop on each partners’ website and it will “just work”. Potential caveats: what if the search result page changes? Also the Opensearch can only be as fast as the response from the slowest partner.

He shows the working prototype, and I’m excited to see they’ve got thumbnails where available - their scraper must be fairly robust for each partner.

Lessons learned: federated search doesn’t have to be expensive or complicated, and it can work with small and large museums equally well. Their method pushes the work offsite, requiring minimal or no effort on the museum’s part.

(Note to self: end slide show with a kitten and you’re in.)

Q&A - Scalibility issues come up, they’re aware of them coming. Asked if they considered Google Co-Op: yes, but quickly found that Google was unable to deeply crawl many of the partners’ collections due to dynamic urls. Lots of twitter traffic in this session too.

Very interesting debate for me to hear on OAI vs. Opensearch. Many institutions moving towards OAI, but the scope of implementing it is a barrier for most. My feeling that OAI gives more searchable fields is somewhat refuted by the idea that the average user has no interest or knowledge of these fields (culture, era, etc)…

(Mike Ellis shows off by building a co-op search during the session.)

Johan Mhlenfeldt Jensen from the Museum of Copenhagen, Denmark, speaks next on his paper. Trying to catch up, I was distracted for the beginning.

The example he’s showing now exposes some fields for filtering, rather than just keywords. Interesting. Another example showing map-based searching, says it’s immensely popular. Easy to make for photographic collections since the address is known, much harder for other sorts of objects sometimes.

Interesting discussion on “advanced search” - he says studies show it’s minimally used, Google has changed everything. People just want a single field. Hmm… Are we wasting time and overbuilding if we have anything more advanced than a single field?? This is the question I’m banging against as I listen to these speakers.

He asks “is the best the enemy of the good?” Good question. Do we wait forever getting it right? Clearly, no, but how far do we go.

They both have good input on the question I ask about overbuilding: move the advanced search behind the scenes and make it more semantic. Still need the metadata, but don’t ask users to know about it. Also need a way to drill down after search: start with simple search, and then apply filters.

Very good comment on positioning: where and at what point in the process do you expose filters and result counts?

Brian summarizes the importance of getting static URIs for resources: then Google will “just work”…

(Note to self: implement Opensearch for the Walker and ACE)

Standing room only at David Greenfield’s morning session. I’m looking forward to this session now that my head’s full of Web 2.0 / Museum ideas thanks to Nina’s recent presentation. David starts by tipping his hat to Shelley at Brooklyn, Gail at V&A, and a few others - this conference, he says, is full of people tackling different parts of the Web 2.0 creature.

Quickly goes into some theory: Howard Gardner’s idea of multiple intelligences, how one measure does not predict another. Roger Schank’s notion of “narrative and intelligence”, how learning is connected to the experience and situation of the learning. Seymour Papert: communities of learners, and how group learning is powerful: everyone of all levels in the same classes, and a hierarchy of learning. Finally, Ken Robinson: “schools are killing creativity”, based on an Industrial Revolution mindset which no longer works in today’s society. I’m intrigued by the theory, but I’m more intrigued by how he’s going to tie this to Web 2.0 tools.

(Seriously, people are sitting in the isle and standing across the back of the room. It’s packed.)

David moves to a list of common problems museums face with technology and learning, and mentions a few solutions such as partnerships.

He’s got a wiki set up to support the session: www.redberry.pbwiki.com.

Finishes with some examples from the Living Museum and their Web 2.0 efforts: blogging, Facebook, Youtube.

Q&A - Audience giving examples of Web 2.0 efforts in their institutions. The issue of in-house programming staff comes up again - I think people get it, but it’s still hard to convince their institutions that this is so important.

Fiona Cameron

Object Orientated Democracies: Contradictions, challenges, and opportunities

Fiona starts by defining “Networked Objects” - collections now operate in a global flow of greater resources online. Collections information is becoming fluid. The meaning is created and re-created in various ways, especially due to influences in popular culture.

Here, controversy is seen as a positive element: objects take on a new role as mediator, rather than simply cultural symbols.

Example of interpretting a Palestinian dress - different readings of the same object depending on reader’s perspective. Placing these objects in an open wiki was seen as highly problematic as “public” meets “museum culture”.

She shows an incredible map: “complexifying collections interfaces” showing an overview of the various spheres of influence on collections and objects. I love this: translation of the object is ongoing, not fixed.

These maps are blowing my mind! I need to find this paper online and pour over it more to really grasp what she’s saying here: “the meaning and significance of objects can take many forms.” She describes four influences: local knowledge, expert communities, experiental, and the museum voice. All combine to create jointly-generated knowledge of an object.

Emphasizing the legitimacy of other types of knowledge, and embracing complexity in collections.

Peter Samis

Who’s responsible for Saying what we See?

Exhibition of the work of Olafur Eliasson, and online component. Peter starts by talking about the idea of a “phenomenon maker” - doesn’t exist without people experiencing it, participating. Objects don’t have a meaning that the museum could convey, it required people to describe how they were experiencing it. Allowed visitors to describe the work on a new blog: they came, they reacted, they wrote. However, as we’ve seen in similar projects, most people were lurking, wanting to read others’ comments but not add their own.

Who knew Peter spoke French? :)

Their online component allowed them to guage interest in objects: those that seemed to “require” a comment. People needed to talk about them - positive AND negative.

What’s the value of comments such as these? He asks this tantalizing question, and leaves it for the Q&A at the end.

Traces the evolution of museum blogs: from institutional to more public participation, and theorizes on the final step of merging this interaction. References Nina Simon’s museum social interaction hierarchy.

Aaron Cope, flickr.com

The API as Curator

About artists and institutions “opening up” - not giving everything away, but allowing sharing and seeing what people build. It’s really about “plumbing, and making plumbing not scary.” If you’re talking about the web, eventually you’re going to have to talk about computer programming.

EXCELLENT pitch for bringing the programming in-house for museums! Not everyone needs to be a programmer, but we need it: for the “plumbing”.

Threadless: t-shirt company online where users generate and vote on designs to be made into shirts. It’s essentially printmaking, in a new form.

Now into APIs and the importance of exposing the data. (Flickr commons) Story of Dan Catt creating GeoCommons using flickr APIs to get geocoding data into flickr tags (basically a hack) and mash it into a google map (using THEIR API). Flickr hired him and made it real, but he was able to built it in the first place because the “parts” were exposed. (That sounds dirty. But it’s important.)

Museums need FAST release cycles to keep up with this stuff. Small steps towards awesome: this required having programmers ON STAFF to build APIs and build off of other APIs.

Aaron: great slides!

Q&A:

how to attract in-house staff? Teach programming at more levels.