A few weeks ago, Google announced support for analytics inside mobile apps and some cursory support for mobile sites:

Google Analytics now tracks mobile websites and mobile apps so you can better measure your mobile marketing efforts. If you’re optimizing content for mobile users and have created a mobile website, Google Analytics can track traffic to your mobile website from all web-enabled devices, whether or not the device runs JavaScript. This is made possible by adding a server side code snippet to your mobile website which will become available to all accounts in the coming weeks (download snippet instructions). We will be supporting PHP, Perl, JSP and ASPX sites in this release. Of course, you can still track visits to your regular website coming from high-end, Javascript enabled phones.

And that is the extent of the documentation you will find anywhere on Google on how to run analytics without javascript. The code included is handy if you happen to run one of their platforms, but the Walker’s mobile site runs on the python side of AppEngine, so their code doesn’t do us much good. Thankfully, since they provide us with the source, we can without too much trouble, translate the php or perl into python and make it AppEngine friendly.

How it works

Regular Google Analytics works by serving some javascript and a small 1px x 1px gif file to your site from Google. The gif lets Google learn many things from the HTTP request your browser makes, such as your browser, OS, where you came from, your rough geo location, etc. The javascript lets them learn all kinds of nifty things about your screen, flash versions, event that fire, etc. And Google tracks you through a site by setting some cookies on that gif they serve you.

To use GA without javascript, we can still do most of that, and we do it by generating our own gif file and passing some information back to Google through our server. That is, we generate a gif, assign and track our own cookie, and then gather that information as you move through the site, and use a HTTP request with the appropriate query strings and pass it back to Google, which they then compile and treat as regular old analytics.

The Code

To make this work in appeinge, we create a  URL in our webapp that we’ll serve the gif from. I’m using “/ga/”:

def main():
application = webapp.WSGIApplication(
[('/', home.MainHandler),
# edited out extra lines here
('/ga/', ga.GaHandler),
],
debug=False)
wsgiref.handlers.CGIHandler().run(application)

And here’s the big handler for /ga/. I based it mostly off the php and some of the perl (click to expand the full code):

from google.appengine.ext import webapp
from google.appengine.api import urlfetch
import re, hashlib, random, time, datetime, cgi, urllib, uuid

# google analytics stuff
VERSION = "4.4sh"
COOKIE_NAME = "__utmmobile"

# The path the cookie will be available to, edit this to use a different cookie path.
COOKIE_PATH = "/"

# Two years in seconds.
COOKIE_USER_PERSISTENCE = 63072000

GIF_DATA = [
      chr(0x47), chr(0x49), chr(0x46), chr(0x38), chr(0x39), chr(0x61),
      chr(0x01), chr(0x00), chr(0x01), chr(0x00), chr(0x80), chr(0xff),
      chr(0x00), chr(0xff), chr(0xff), chr(0xff), chr(0x00), chr(0x00),
      chr(0x00), chr(0x2c), chr(0x00), chr(0x00), chr(0x00), chr(0x00),
      chr(0x01), chr(0x00), chr(0x01), chr(0x00), chr(0x00), chr(0x02),
      chr(0x02), chr(0x44), chr(0x01), chr(0x00), chr(0x3b)
  ]

class GaHandler(webapp.RequestHandler):
	def getIP(self,remoteAddress):
	  	if remoteAddress == '' or remoteAddress == None:
			return ''

		#Capture the first three octects of the IP address and replace the forth
		#with 0, e.g. 124.455.3.123 becomes 124.455.3.0
		res = re.findall(r'\d+\.\d+\.\d+\.', remoteAddress)
		if res:
			return res[0] + "0"
		else:
			return ""

	def getVisitorId(self, guid, account, userAgent, cookie):
		#If there is a value in the cookie, don't change it.
		if type(cookie).__name__ != 'NoneType': # or len(cookie)!=0:
			return cookie

		message = ""

		if type(guid).__name__ != 'NoneType': # or len(guid)!=0:
			#Create the visitor id using the guid.
			message = guid + account
		else:
			#otherwise this is a new user, create a new random id.
			message = userAgent + uuid.uuid1(self.getRandomNumber()).__str__()

		m = hashlib.md5()
		m.update(message)
		md5String = m.hexdigest()

		return str("0x" + md5String[0:16])

	def getRandomNumber(self):
		return random.randrange(0, 0x7fffffff)

	def sendRequestToGoogleAnalytics(self,utmUrl):
		'''
		Make a tracking request to Google Analytics from this server.
		Copies the headers from the original request to the new one.
		If request containg utmdebug parameter, exceptions encountered
		communicating with Google Analytics are thown.
		'''
		headers = {
			"user_agent": self.request.headers.get('user_agent'),
			"Accepts-Language": self.request.headers.get('http_accept_language'),
			}
		if len(self.request.get("utmdebug"))!=0:
			data = urlfetch.fetch(utmUrl, headers=headers)
		else:
			try:
				data = urlfetch.fetch(utmUrl, headers=headers)
			except:
				pass

	def get(self):
		'''
		Track a page view, updates all the cookies and campaign tracker,
		makes a server side request to Google Analytics and writes the transparent
		gif byte data to the response.
		'''
	  	timeStamp = time.time()

		domainName = self.request.headers.get('host')
		domainName = domainName.partition(':')[0]

		if len(domainName) == 0:
			domainName = "m.walkerart.org";

		#Get the referrer from the utmr parameter, this is the referrer to the
		#page that contains the tracking pixel, not the referrer for tracking
		#pixel.
		documentReferer = self.request.get("utmr")

		if len(documentReferer) == 0 or documentReferer != "0":
			documentReferer = "-"
		else:
			documentReferer = urllib.unquote_plus(documentReferer)

		documentPath = self.request.get("utmp")
		if len(documentPath)==0:
			documentPath = ""
		else:
			documentPath = urllib.unquote_plus(documentPath)

		account = self.request.get("utmac")
		userAgent = self.request.headers.get("user_agent")
		if len(userAgent)==0:
			userAgent = ""

		#Try and get visitor cookie from the request.
		cookie = self.request.cookies.get(COOKIE_NAME)

		visitorId = str(self.getVisitorId(self.request.headers.get("HTTP_X_DCMGUID"), account, userAgent, cookie))

		#Always try and add the cookie to the response.
		d = datetime.datetime.fromtimestamp(timeStamp + COOKIE_USER_PERSISTENCE)
		expireDate = d.strftime('%a,%d-%b-%Y %H:%M:%S GMT')

		self.response.headers.add_header('Set-Cookie', COOKIE_NAME+'='+visitorId +'; path='+COOKIE_PATH+'; expires='+expireDate+';' )
	  	utmGifLocation = "http://www.google-analytics.com/__utm.gif"

		myIP = self.getIP(self.request.remote_addr)

		#Construct the gif hit url.
		utmUrl = utmGifLocation + "?" + "utmwv=" + VERSION  + \
			"&utmn=" + str(self.getRandomNumber()) + \
			"&utmhn=" + urllib.pathname2url(domainName) + \
			"&utmr=" + urllib.pathname2url(documentReferer) + \
			"&utmp=" + urllib.pathname2url(documentPath) + \
			"&utmac=" + account + \
			"&utmcc=__utma%3D999.999.999.999.999.1%3B" + \
			"&utmvid=" + str(visitorId) + \
			"&utmip=" + str(myIP)

		# we dont send requests when we're developing
		if domainName != 'localhost':
			self.sendRequestToGoogleAnalytics(utmUrl)

		#If the debug parameter is on, add a header to the response that contains
		#the url that was used to contact Google Analytics.
		if len(self.request.get("utmdebug")) != 0:
			self.response.headers.add_header("X-GA-MOBILE-URL" , utmUrl)

	  	#Finally write the gif data to the response.
		self.response.headers.add_header('Content-Type', 'image/gif' )
		self.response.headers.add_header('Cache-Control', 'private, no-cache, no-cache=Set-Cookie, proxy-revalidate' )
		self.response.headers.add_header('Pragma', 'no-cache' )
		self.response.headers.add_header('Expires', 'Wed, 17 Sep 1975 21:32:10 GMT' )
		self.response.out.write(''.join(GIF_DATA))

So now we know what to do with our requests at /ga/ when we get them, we just need to make the proper requests to that URL in the first place. So we need to generate the URL we’re going to have the visitor’s browser request in the first place. With normal django, we would be able to use template_context to automatically insert it into the page’s template values. But, since AppEngine doesn’t use that, we have our own helper functions to do that, which I showed some of in my last post. Here’s the updated helper functions, with the GoogleAnalyticsGetImageUrl function included:

import settings

def googleAnalyticsGetImageUrl(request):
	url = ""
	url += '/ga/' + "?"
	url += "utmac=" + settings.GA_ACCOUNT
	url += "&utmn=" + str(random.randrange(0, 0x7fffffff))

	referer = request.referrer
	query = urllib.urlencode(request.GET) #$_SERVER["QUERY_STRING"];
	path = request.path #$_SERVER["REQUEST_URI"];

	if len(referer) == 0:
		referer = "-"

	url += "&utmr=" + urllib.pathname2url(referer)

	if len(path)!=0:
		url += "&utmp=" + urllib.pathname2url(path)

	url += "&guid=ON";

	return {'gaImgUrl':url}

def getTempalteValues(request):
	myDict = {}
	myDict.update(ua_test(request))
	myDict.update(googleAnalyticsGetImageUrl(request))
	return myDict

Assuming we use getTemplateValues to set up our inital template_values dict, we should have a variable named ‘gaImgUrl’ in our page. To use it, all we need to do is put this at the bottom of every page on the site:

<img src="{{ gaImgUrl }}" alt="analytics" />

My settings file contains the GA_ACCOUNT variable, but replaces the standard GA-XXXXXX-X setup with MO-XXXXXX-X. I’m assuming the MO- tells google that it’s a mobile so accept the proxied requests.

One thing to keep in mind with this technique is that you cannot cache your rendered templates. The image you server will necessarily have a different query string every time, and if you cached it, you would ruin your analytics. Instead, you should cache nearly everything from your view functions, except the gaImgUrl variable.

Reviewing our current site to inform the mobile site

The web framework we use for most of our site has the ability, with some small changes, to load different versions of a page based on a visitor’s User Agent (what browser they’re using). This would mean we could detect if a visitor was running IE on a Desktop or Mobile Safari on an iPhone, and serve each of them two different versions of a page. This is how a lot of mobile sites are done.

This is not the approach we went with for our mobile site, because it violates two of the primary rules (in my mind) of making a mobile website:

1. Make it simple.
2. Give people the stuff they’re looking for on their phones right away.

Our site is complicated: we have pages for different disciplines, a calendar with years of archives, and many specialty sites. Rule #1, violated. To address #2, I took a look at our web analytics to figure out what most people come to our site looking for. This won’t surprise anyone, but it’s hours, admission, directions, and what’s happening today at the Walker:

Top Walker Pages as noted by Google Analytics

So it seems pretty clear those things should be up front. One of the other things you might want to access on a mobile is Art on Call. While Art on Call is designed primarily around dial-in access, there is also a website, but it isn’t optimized for the small screen of a smartphone. We have WiFi in most spaces within our building, so accessing Art on Call via an web interface and streaming audio via HTTP rather than POTS is a distinct possibility that I wanted to enable.

Prototyping with Google AppEngine

I decided to develop a quick prototype using Google AppEngine, thinking I’d end up using GAE in the end, too. Because this was a 20% time project, I had the freedom to do it using the technology I was interested in. AppEngine has the advantage of being something that isn’t hosted on our server, so there was no need to configure any complicated server stuff. In my mind, AppEngine is perfect for a mobile site because of the low bandwidth requirements for a mobile site. Google doesn’t provide a ton for free, but if your pages are only 20K each, you can fit quite a bit within the quotas they do give provide. AppEngine’s primary language is also python, a big plus, since python is the best programming language.

In about two days I built a proof of concept mobile site that displayed the big-ticket pages (hours, admission,etc.) and had a simple interface for Art on Call. Using iUi as a front-end framework was really, really useful here, because it meant that the amount of HTML/CSS/JS I had to code was super minimal, and I didn’t have to design anything.

I showed the prototype to Robin and she enthusiastically gave me the green light to work on it full-time.

Designing a mobile website

A strategy I saw when looking at mobile sites was to actually have two mobile sites: one for the A-grade phones (iPhone, Nokia S60, Android, Pre) and one for the B- and C-grade phones (Blackberry and Windows Mobile). The main difference between the two is the use of javascript and some more advanced layout. Depending on what version of Blackberry you look at, they have a pretty lousy HTML/CSS implementation, and horrendous or no javascript support.

To work around this, our mobile site does not use any javascript on most pages and tries to keep the HTML/CSS pretty simple. We don’t do any fancy animations to load between pages like iUi or jQtouch do: even on an iPhone those animations are slow. If you make your pages small enough, they should load fast enough and a transition is not necessary.

Designing mobile pages is fun. The size and interface methods for the device force you to re-think how to people interact and what’s important. They’re also fun because they’re blissfully simple. Each page on our mobile site is usually just a headline, image, paragraph or two, and some links. Laying out and styling that content is not rocket surgery.

Initially, when I did my design mockups in Photoshop, I wanted to use a numpad on the site for Art on Call, much like the iPhone features for making a phone call. There’s no easy input for doing this, but I thought it wouldn’t be too hard to create one with a little javascript (for those that had it). Unfortunately, due to the way touchscreen phones handle click/touch events in the browser, there’s a delay between when you touch and when the click event fires in javascript. This meant that it was possible to touch/type the number much faster than the javascript events fired. No go.

Instead, the latest versions of WebKit provide with a HTML5 input field with a type of “number”. On iPhone OS 3.1 and better, it will bring up the keypad already switched to the numeric keys. It does not do this on iPhone OS prior to 3.1. I’m not sure how Android and Pre handle it.

Mocked up Art on Call code input.

Implimented Art on Call code input.

Comparing smartphones

Here’s a few screenshots of the site on various phones:

Palm Pre

Android 1.5

Blackberry 9630

Not pictured is Windows Mobile, because it looks really bad.

A future post may cover getting all of these emulators up and running, because it’s not as straight easy as it should be. Working with the blackberry emulator is especially painful.

How our mobile site works

The basic methodology for our mobile site is to pull the data, via either RSS or XML from our main website, parse it, cache it, and re-template it for mobile visitors. Nearly all of the pages on our site are available via XML if you know how to look. Parsing XML into usable data is a computationally expensive task, so caching is very important. Thankfully, AppEngine provides easy access to memcache, so we can memcache the XML fetches and the parsing as much as possible. Here’s our simple but effective URL parse/cache helper function:

from google.appengine.api import urlfetch
from xml.dom import minidom
from google.appengine.api import memcache

def parse(url,timeout=3600):
	memKey = hash(url)
	r = memcache.get('fetch_%s' % memKey)
	if r == None:
		r = urlfetch.fetch(url)
		memcache.add(key="fetch_%s" % memKey, value=r, time=timeout)
	if r.status_code == 200:
		dom = memcache.get('dom_%s' % memKey)
		if dom == None:
			dom = minidom.parseString(r.content)
			memcache.add(key="dom_%s" % memKey, value=dom, time=timeout)
		return dom
	else:
		return False

Google AppEngine does not impose much of a structure for your web app. Similar to Django’s urls.py, you link regular expressions for URLS to class-based handlers. You can’t pass any variables beyond what’s in the URL or the WebOb to the request handler. Each handler will call a different method, depending if it’s a GET, POST, DELETE, http request. If you’re coming from the django world like me, this is not much of a big deal at first, but it gets tedious pretty fast. If I had it to do over again, I’d probably use app-engine-patch from the outset, and thus be able to use all the normal django goodies like middleware, template context, and way more configurable urls.

Within each handler, we also cache the generated data where possible. That is, after our get handler has run, we cache all the values that we pass to our template that won’t change over time. Here’s an example of the classes that handle the visit section of our mobile site:

from google.appengine.ext import webapp
from google.appengine.ext.webapp import template
from google.appengine.api import memcache
from xml.dom import minidom
from google.appengine.api import memcache
from utils import feeds, parse, template_context, text
import settings

class VisitDetailHandler(webapp.RequestHandler):
	def get(self):
		url = self.request.get("s") + "?style=xml"
		template_values = template_context.getTempalteValues(self.request)
		path = settings.TEMPLATE_DIR + 'info.html'
		memKey = hash(url)

		r = memcache.get('visit_%s' % memKey)
		if r and not settings.DEBUG:
			template_values.update(r)
			self.response.out.write(template.render(path, template_values))
		else:
			dom = parse.parse(url)
			records = dom.getElementsByTagName("record")
			contents = []
			for rec in records:
				title = text.clean_utf8(rec.getElementsByTagName('title')[0].childNodes[0].nodeValue)
				body = text.clean_utf8(rec.getElementsByTagName('body')[0].childNodes[0].nodeValue)
				contents.append({'title':title,'body':body})

			back = {'href':'/visit/#top', 'text':'Visiting'}
			cacheableTemplateValues = { "contents": contents,'back':back }
			memcache.add(key='visit_%s' % memKey, value={ "contents": contents,'back':back }, time=7200)
			template_values.update(cacheableTemplateValues)
			self.response.out.write(template.render(path, template_values))

Dealing with parsing XML via the standard DOM methods is a great way to test your tolerance for pain. I would use libxml and xpath, AppEngine doesn’t provide those libraries in their python environment.

Because the only part of Django’s template system that AppEngine uses is the template language, and nothing else, we have to roll our own helper functions for context. Meaning, if we want to pass a bunch variables by default to our templates, something easy in django, we have to do it a little differently with GAE. I set up a function called getTemplateValues, which we pass the WebOb request, and it ferrets out and organizes info we need for the templates, passing it back as a dict.

def ua_test(request):
	uastring = request.headers.get('user_agent')
	uaDict = {}
	if "Mobile" in uastring and "Safari" in uastring:
		uaDict['isIphone'] = True
	if 'BlackBerry' in uastring:
		uaDict['isBlackBerry'] = True
	return uaDict

def getTempalteValues(request):
	myDict = {}
	myDict.update(ua_test(request))
	myDict.update(googleAnalyticsGetImageUrl(request))
	return myDict

In my next post, I’ll talk about how to track visitors on a mobile site using google analytics, without using javascript.

This project was at least three years in the making, with the last two of those being the technical work of research, design, and development.  In this series of posts I’d like to present some of the decisions we struggled with and the process we went through in developing the new site.  I’ll start with the Art Finder, followed by a post on the Art Collector and presentations, and finish with a post about some of the more technical aspects including the data and harvesting technologies we’re using.

Art Finder

The Art Finder is the guts of the site, a portal into our thousands and thousands of objects, text records, and more.  I don’t think it’s an exaggeration to say designing and building this component was the biggest challenge we faced in the entire process.  We’ve redesigned the interface many times, often significantly, and are still not certain it’s right.  We’ve changed the underlying technology from a SQL / Lucene hybrid to a straight-up Solr search engine.  We’ve debated (endlessly) what fields to include, and what subset of our data to present in those fields.  We’ve gone back and forth over tab titles, and even whether to use tabs.  A rocky road, to say the least.

The big idea

What if we could start with everything and narrow it down from there?  Offer the user the entire collection and let them whittle away at it until they found what they wanted?

It’s all browse.  Keyword is just another filter.

To me this is the big breakthrough of the ArtsConnectEd interface.  We don’t hide the content behind a search box, or only show filters after you try a keyword.  We don’t have a separate page for “Advanced Search”, but we offer the same power through filters.  There is still a keyword field for those who know exactly what they’re looking for, but we get to use our metadata in a more powerful way than simple text.  That is, since know the difference between the word “painting” appearing in the description and something that is a painting, we can present that to the user through filters.

How we Got here

We wanted many ways for the user to explore the collection, with the idea we might hopefully mimic some of the serendipity of exploring a gallery.  The tech committee felt early on that we’d need, in addition to a robust search, some way to freely browse.  Our initial attempt was to split the Art Finder into a Browse interface (left) and a Search interface (right).

After forcing users to choose a content type to browse (Object, Text, etc), we exposed facets (fields) to allow filtering, e.g. by Medium or Style.  These facets were hidden by default in the Search interface, where instead you started with a keyword and content type as tabs — but could then click to reveal the same browse filters!  The more we played with these two ideas, the more we realized they were essentially the same thing, the only difference being a confusing first step and then having to learn two interfaces.  The real power of the site was in combining them, committing fully to Browse, and adding the keyword search as a filter.

Lastly, as we harvested more of our collections we realized pushing filters to the front offered a better way to drill down when many of our records are not text-heavy and thus less findable via keyword search.  In many ways browse leveled the playing field of our objects between those with healthy wall labels and those with more sparse metadata.

What works

(In my humble opinion!)  A good browse has to do a few things:

• Be fast. Studies have shown that slow search (or browse) results derail a user’s chain of thought and makes it difficult to complete tasks.  We went one step further and did away with the “Go” button for everything but keyword – making a change to a pulldown automatically updates your result set.  (It’s not instant, but it’s fast enough the action feels connected to the results)
• Reduce complex fields to an intuitive subset. We have a huge range of unique strings for the Medium field, but we’ve broadly grouped them to present a reasonable-sized pulldown.  Likewise for the Culture pulldown.  (We manually reduce the terms for Medium, and have a automated Bayesian filter for the Culture field)
• Have good breadcrumbs. Users need to know what options are in effect and be able to backtrack easily.
• Avoid dead ends. With many interfaces it’s entirely too easy to browse yourself into an empty set.  By showing numbers next to our filter choices, we can help users avoid these “dead ends”.
• Expose variety. Type “Jasper Johns” in the artist field, and check out the Medium pulldown: it shows the bulk of his work is in Prints, but we also have a few sculptures, some mixed media, etc.  A nice way to see the variety of an artist’s work at-a-glance.
• Autocomplete complicated fields. If a search box is targeted to a field (like our Artist box), it needs to autocomplete.  Leaving a field like this open to free text is asking for frustration as people get 0 results for “Claes Oldenberg“. (Auto-suggest “did you mean” should also work!)
• Have lots of sort options. One of my favorite features of the new Art Finder is the ability to sort by size.  Super cool.  (check out the Scale tab in the detail view for more fun!)

I’m biased after this project, but I’m fairly convinced combining faceted browsing with keyword search is absolutely the way to go for collection search.  It gives the best of both worlds, powerful but still intuitive.

What could be better

… but is it really intuitive?  People seem to still be looking for a big inviting search box to start with.  The interface is crowded, and the number of options looks intimidating.  We’ve ended up avoiding using the words “Search” and “Browse” because they were loaded and causing confusion.  We’ve tried many versions of the tab bar to try to clarify what filters apply globally (e.g. Institution) and which only effect that tab (Works of Art have an Artist, for instance), but I don’t believe we’ve solved it.

I think the two components of the interface that give us the most trouble and confusion are actually the “Has Image” checkbox and the “Reset All” button.  These are consistently missed by people in testing, and we have tried almost everything we can think of.  Oh, and the back button.  The back button is “broken” in dynamic search like this.

Also, while I really like the look of the tiles in the results panel, we’ve had to heavily overload the rollover data to show fields we can sort by since there’s no more room in the tiles.  We also intended to create alternative result formats, such as text bars, etc, which could show highlights on matching keywords, but this item was pushed back for other features.

We’ve defaulted to sorting alphabetically by title when a user first reaches the page, and I’m no longer sure this is best.  As we’ve populated the collections in ArtsConnectEd we’ve ended up with a bunch of works that have numbers for titles, make the alpha sort less obvious.

You tell me!  Give the site a spin and post a comment – what works, and what could be better?

Resources:

• Designing for Faceted Search (http://www.uie.com/articles/faceted_search/)
• Faceted Search: Designing Your Content, Navigation, and User Interface (http://www.uie.com/events/virtual_seminars/facets/FacetedSearchVS35Handout.pdf)
• Faceted Search (http://en.wikipedia.org/wiki/Faceted_search)
• Best Practices for Designing Faceted Search Filters (http://www.uxmatters.com/mt/archives/2009/09/best-practices-for-designing-faceted-search-filters.php)
• V&A Collections (beta) (http://www.vam.ac.uk/cis-online/search/?q=blue&commit=Search&category%5B%5D=5&narrow=1&offset=0&slug=0)
  • Their facets aren’t as up front as I’d like (you have to start with a keyword), but they’re done really well once they show up.
  • You can also cheat and leave keyword blank to get a full browse and go right to the facets…  Maybe start here?
• MOMA Collections (http://www.moma.org/collection/search.php)
  • Nice presentation of facets, but I wish two things: show me a number next to all constraints, not just artists, and let me add a keyword.  (I got a dead end looking for on-view film from the 20s or 2000s)  I also like that it’s a true browse – leaving everything at “All” seems to give me the whole collection.

]]> http://blogs.walkerart.org/newmedia/2009/09/22/behind-the-scenes-of-artsconnected-art-finder/feed/ 10 http://blogs.walkerart.org/newmedia/2009/09/17/web-and-video-standards-roundup/ http://blogs.walkerart.org/newmedia/2009/09/17/web-and-video-standards-roundup/#comments Thu, 17 Sep 2009 19:47:32 +0000 Justin Heideman http://blogs.walkerart.org/newmedia/?p=886

• Eric at Adapted Studio put together this sweet little demo of HTML5 and Canvas in action, in the form of the Game of life. Source code is included, too, if you want to learn a few nifty things.
• Color me surprised, but Microsoft is actually purporting to work together on at least some of the HTML5 spec. This could be good. Using <video> would be much easier if everyone would do it. But there still is the nasty issue of codecs, which is even more thorny than W3C specs.
• This is from about a year ago, but John Resig (of jQuery fame) posted a very nice tutorial for Learning Advanced Javascript. It clears up a lot of confusion about seemingly advanced techniques.
• Also worth perusing is Mark Pilgrim’s Gentle introduction to video formatting. If you’re a video geek, you might know some of this, but there’s detail that might fill in some gaps. The slides are also slightly amusing. I had no idea the .mkv format came from a bunch of guys in Russia that decided to opensource it.

HTML 5 image form here.

In fact a few months ago the New Media department decided to drop support for IE6 on all future websites we create. The last website we built with full IE6 support was the new ArtsConnectEd, mainly because teachers tend to have little say in what browsers they can use on school computers. However, moving forward we’re phasing out support for IE6. It simply costs us too much time and resources for the dwindling number of users it has on our sites (currently under 10%, which is down 45% from last year and falling fast). We’re not alone, many other sites are doing this as well.

However calling for the killing of IE6 ignores a bit of history as well as new problems to come. There was a time not so long ago when all web developers wanted to be using IE6. The goal back then was to kill off IE5. You see, IE5 had an incorrect box model. Padding and margins were included in a boxes width and height instead of adding to it like in standards compliant browsers.

This caused all sorts of layout errors, and meant hacks (like the Simplified Box Model Hack) had to be used to get content to align correctly. These hacks were so widely used that Apple was going to allow them to be used in the first version of Safari until I convinced Dave Hyatt (lead Safari dev) to take out support for it. IE6 fixed this bug and everyone was happy (for a while anyway).

Going back further, IE5, even with its broken box model, was at one time the browser of choice back when IE4 was killing Javascript programmers because it didn’t support document.getElementById(). IE4 only supported the proprietary document.all leading to a horrible fracturing of Javascript, whereas IE5 added in the JS standard we still use today. Before people embraced IE5, cross platform JS on the web was almost non-existent, a fact I attempted to rectify by building my Assembler site in 1999.

The reason I bring this up is because we have a history of this behavior with regards to IE. We yearn for the more modern versions, only to end up hating those same versions later on. This will not change with the death of IE6. Soon, it will be IE7 that we are trashing, and then IE8 will be the bane of our existence.

This only becomes more clear as we move to HTML5. IE8 doesn’t support it, nor does it support any CSS3. While IE8 does support many of the older standards it had been ignoring for so long, having just recently been released it is already out of date. All of the other browsers do support these advanced web technologies, but IE is the lone browser to ignore them. Once again IE is two steps behind where the web is going, and severely limits our ability to push web technology forward to everyone for many years to come.

So while we celebrate the death of IE6, let us not forget that there will be a new thorn in our side to take its place in short order. IE7, you’re next.

Museums and the Web 2009 wrapped up with a challenge to all the inspired delegates: use the energy and ideas generated here to get one thing done in April.  (The idea being that many small steps build momentum, and it’s too easy to ignore the small upgrades we should constantly be pushing out.)

Yesterday I pushed out a few small upgrades to our aging collection site:

You can now limit your search to objects that are On View

What works by Dan Flavin can you come see right now?

OpenSearch capable

Can’t get enough of our collection?  Add it to your browser’s built-in search box!  When you’re on the Collection site, you should be able to pull down your browser’s search field and add “Walker Art Center”.

Developers (Piotr!): you can now use the Walker collection in your Yahoo Pipes tool without having to scrape the results!  Not an API (yet), but a good step.  Check out the XML for ideas.

Bring it all together:

You’re a busy person.  You’d love to come see Chuck Close’s Big Self-Portrait, and you know the Walker’s got it in their collection, but you see it’s not on view.  You don’t have time to check our website every day, so how will you ever know when it goes on display?  Easy:  build a search that finds Big Self-Portrait, then turn on the “On View” flag.  The object disappears (not on view), but you can subscribe to the OpenSearch RSS feed for this query (click the icon).  Now, when Big Self-Portrait is available to see in the galleries, the object will show up in your RSS reader!  (note: I picked this painting randomly.  I make no guarantee about seeing it in the galleries any time soon.  :)

So, baby steps.  Get one things done that opens more doors.

#didonethinginapril (I tag Andrew at the MIA to get one thing done in May!)

Homepage

The first change you’ll notice when visiting the site is that the home page got an overhaul. The rotators for New Artwork and Featured Collections were changed to display images to the full-size of their boxes and they animate smoother. This means sometimes cropping work, but we think it’s a trade-off worth making.

Articles are also displayed with a three-tier hierarchy, allowing the site to call recent writing out more prominently, even though we feature six instead of 10. The sidebar on the homepage has also been reorganized, bringing the mnartists.org blog to the top and adding links to the Facebook and Twitter profiles for mnartists.org.

The revised homepage.

Revised article page

Articles

Articles got some attention in several ways. First, we changed the way images are displayed by adding a larger expanding gallery at the top of each article, rather than having small images thumbnails listed down the left side. On the back end for editors, we also added an enhanced editor (tiny mce) to allow for richer control over formatting and even embedding other media.

Social media Sharing
Across many areas of the site, you’ll now see a link to Share this article/artwork/collection/event. Using much of the same code we developed for the Walker Calendar, sharing is now easier on mnartists.org. We connect with Facebook, Twitter, Myspace, Delicious, Google Bookmarks, and Yahoo Bookmarks, as well as rolling in email links in a few places.

The new sharing links.

Search

The one change that will probably make everyone cry tears of joy is the search results refinement. We’ve heard lots of complaints about the search, not wholly unfounded. The search actually works pretty good, but the simple search weights everything more or less equally. If you search for someone’s name, hoping to just get their artist page, it will be in the results, but there might be other things that rank higher.

The revised search result page lets you change your simple search into an advanced search, using tabs above the results to select the type of resource you want to search for. This is very similar to what google does with their search results refinements (web, images, video, maps, etc.).

Old style search results

New search results with refinements.

Artist Pages

Artist pages also got an overhaul with two big changes. First, images for each artwork will display at a new, larger size, about 519px tall and/or 520px wide. For artworks with more than one image associated, a gallery rotating gallery will cycle through the images. Previously, if an artwork had more than one image associated, only the first would show up, and the rest would be listed in the “Related Media” list.

Old artist homepage.

Revised artist page.

Secondly, we changed the way Related Media works. Now, it is simply “Media List” lists every type of media associated with an artwork. More importantly, for non-image media, such as video and audio, we embed the media in the actual page. So if you upload a quicktime file, the quicktime embed code will be generated and put right into the page. MP3 audio files will be played with the jwplayer flash player, making audio on the site a lot more nifty. We’re using the excellent jquery.media plugin to do this.

This approach to handling media isn’t without some issues, but given the variety of media already on the site and our resources to work on it, this is the best solution. We are looking at making more substantial changes to this in the future, but this is a good incremental improvement.

Artwork with video before changes.

Artwork with video after changes. (Two video files attached)

The image size and media enhancements have also been applied to the collections area of the site.

Editing text

Another change we made a month ago was adding a visual editor to various form fields on the site. Prior to the change, users could only enter a very limited selection of markup to entries, [b] for bold, [i] for italic, and [a] for a link. We’ve eliminated that and replaced it with the new editor (tiny_mce), which allows for bold, italic, underline, unordered lists, and links. While seemingly simple, it was actually quite a challenge to deal with both the legacy code and the new formatting. The text actually goes through several transformations between the editor, the database, and being displayed again. Keeping everything consistent is a non-trivial pile of regular expressions.

The new visual editor.

One thing that we will have to keep an eye on is users pasting in text from Microsoft Word. Word tends to shove a bunch of garbage pseudo-html into the clipboard, and when pasting, it can be difficult to filter out. The editor has a button to Paste from word (with the blue W) that helps.

Any issues?

If you notice any problems with the site, please let our community manager or myself know. Bugs may crop up, and we do fix things.

There are a number of pre-made DHTML widgets out there that are easy to use, but don’t provide quite the user experience we would like to have. Furthermore, they don’t share the content as cleanly and aren’t event specific. So we made our own functionality. It isn’t rocket surgery, but some notes on what we did may prove useful for others.

Sharing to calendars
Because this is an online calendar, sharing events to other calendars are very important. We already have an iCal feed for our calendar, and it is already set up to share specific events rather than all events, we just hadn’t been using that feature. The new sharing widget does so, simply by passing the proper event ID to our iCal page. A user can download this .ics file and it should work in Outlook, Sunbird, or iCal.

We also added sharing to google calendar. Google has an event publisher guide that documents how to share events to gCal. Compared to most other sharing solutions, gCal is more complicated. The main thing we had trouble with was formatting the date properly. Google prefers the date in what it calls “UTC format”, but I cannot find documentation anywhere showing what google uses is actually UTC format. What UTC format actually appears to be is the ISO 8601 time formatted without any punctuation. This is very similar to what the iCal format uses internally. Thankfully, since we were already calculating this for the iCal format, Nate was able to easily pass me this info for each event. With that, it is simply a matter of putting together the various components of the URL:

gcalURL = 'http://www.google.com/calendar/event?action=TEMPLATE&text='+encodeURIComponent(share_eventTitle);
gcalURL += '&dates='+startDate+"/"+endDate;
gcalURL += "&sprop=website:"+encodeURIComponent(location.href)+"&sprop=name:Walker%20Art%20Center";
gcalURL += "&details="+encodeURIComponent(share_eventDesc);
gcalURL += "&trp=true"
if (theLocation){
	gcalURL += "&location="+encodeURIComponent(theLocation);
}
window.open(gcalURL);
return false;

Sharing to social media sites
MySpace and Facebook both have specifications for how to share events to each of them, documented here and here, respectively. For Facebook, it is important to modify your page to include the meta tags it requests. When you share to Facebook, it doesn’t pass a description or image via the URL. Rather, Facebook scrapes the referred page to ascertain the description and images. Using the Meta tags gives much better results than whatever Facebook’s scraper comes up with. Most of the time, if you rely on their scraper, it will come up with some chrome images from your site rather than actual content images.

MySpace doesn’t scrape the page like facebook, so it’s important to construct a friendly description, with an image, if you can. We put a linked image along with the first few sentences of text for the description we pass to MySpace. Here is the format MySpace uses:

http://www.myspace.com/index.cfm?fuseaction=postto&u=YOURURL&t=YOURTITLE&c=YOURDESCRIPTION&l=3


We also “share” to twitter. There isn’t really sharing per se, on twitter, but you can pre-assemble a tweet for someone. Simply pass someone to http://twitter.com/home/?status=YOURTWEET. We assmble the event title, a reply to our twitter account, and a twitter-friendly shortened URL. Like this: “2008 British Television Advertising Awards @walkerartcenter – http://bit.ly/3c60xK”.

To get the friendly URL, we’re using bit.ly, one of the many URL shortening services available. However, bit.ly has a handy, well documented, API that does JSONP, allowing us to get around cross-site scripting issues.

Sharing to bookmarking sites
Sharing to Bookmarking sites like delicious (formerly del.icio.us, we miss the old URL) is quite simple. These are the formats for Delicious, Google Bookmarks and Yahoo Bookmarks, respectively:

http://delicious.com/save?jump=yes&url=YOURURL&title=YOURTITLE
http://www.google.com/bookmarks/mark?op=edit&bkmk=YOURURL&title=YOURTITLE
http://bookmarks.yahoo.com/toolbar/savebm?u=YOURURL&t=YOURTITLE&opener=bm&ei=UTF-8

Yahoo Bookmarks like to be opened in a pop-up window, but it isn’t strictly necessary. Always remember to urlencode text that is being passed into the URL, since there are many reserved characters in the URL. Javascript provides the encodeURIComponent function to do this.

The biggest problem was simply that Apache is heavy.  It’s resource-intensive, especially when you are running several modules as we were – PHP, proxy, cache, etc.  The teens site is especially difficult since it runs as a combination of a blog (PHP on Apache 2) and .wac pages (mod_perl & Axkit).  Every hit to the Joker post – even if the page was cached – would tie up a number of Apache processes as it served the style sheets, images, and javascript to support the page.  We were reaching our MaxClients setting but unable to raise it without running out of memory for our other more intensive servers (mod_perl and postgres, I’m looking at you…).

As this diagram shows, it’s nothing but Apache servers, and it just wasn’t scaling to meet our current demand.

The approach was two-fold: some quick auditing and re-writing of the worst offending .wac pages’ SQL to speed up the slow pages, and yet another web server in front of everything.  It was a no brainer to pick Lighttpd, or “Lighty”.  It’s written to do one thing – serve static content – and do it extremely fast.  Fortunately it can also proxy requests, so it was a pretty simple matter to reassign some ports and write a few rules to route all requests through Lighty.

The end result is astonishing.  Our server hums along happily under even the most intense traffic we can throw at it (the email blast for the British Television Advertising Awards) and doesn’t even start to complain.  Moving the bulk of the requests to the extremely fast and resource-light server meant we could devote more resources to quickly processing the slow pages (mod_perl).  Between the SQL tuning and the extra resources, the bulk of these pages are now served between 2 and 10(!!!) times faster!

The lesson here, for anyone with an Apache server creaking and groaning under increased traffic, is to stop waiting and install Lighty.  If your site is PHP-based, you can run this as a fast CGI module from Lighty and do away with Apache altogether.  You can also use Lighty to stream (and “scrub”!) flv and mp4 video files.  (I’m using both of these techniques for the new ArtsConnectEd.)

The only caveat: be careful as you look for examples on the web.  Remarkably, it seems there are many confused webmasters who expect to see a performance boost by putting Lighty behind their struggling Apache.  This will not help at all, and in fact will probably make things worse.  Lighty has to be first in the chain to take the load off Apache.

Enjoy the speed!  I know our server is enjoying the breathing room!

Faced with adding another blog (more on that in a couple weeks), we knew that we couldn’t continue to grow our haphazard system any more. We bit the bullet and switched to WordPress MU (WPMU), which solves many of the problems over standalone WordPress. I’ll enumerate the benefits:

• Shared users between all blogs
• Centralized location for files
• Can activate plug-ins across all blogs
• Easy to add new blogs
• Allows us to set up caching

In the past, we had resisted going to WPMU because it wasn’t keeping pace with standalone WordPress in terms of updates. At the same time, we hadn’t updated in a while, and were still using WordPress 2.3.3, when 2.5 and 2.6 added quite a bit, especially better image management. Around the 2.5.2 release of WordPress, the MU trunk was finally in close sync with standalone. No more excuses.

How MU Works

WPMU is very similar to regular WordPress. In fact, there are really only a handful of files that are any different than regular WordPress. The functions that WPMU adds are mostly to deal with the mutiple blog IDs, creating and delting blogs, and dealing with the more complicated permissions. Two of the more handy function it provides are switch_to_blog( $new_blog ) and restore_current_blog(), which let you switch to a different blog, use the proper Loop queries, and then go back to the original blog. This makes it a lot easier to deal with getting information from multiple blogs, an otherwise complicated procedure.

If you look at the database model, WPMU keeps centralized tables for users and blog meta information. Each new blog created gets a set of it’s own tables.

Upgrading

Upgrading to WPMU is not a simple process. In fact, there’s no specific procedure or easy script for it. The main problem is that if you have multiple blogs, your user IDs and post IDs are going to collide. That is, on the New Media blog, my user ID might be 7, but on OffCenter, it is 23, and user ID 7 is someone else. This prevents us from doing a straight SQL import. We’d need a script that could interact with all our existing databases, figure out the conflicts, and then create a new centralized list of users (more on this in a minute).

Additionally, importing posts wouldn’t work either, because with a new users table, all the user IDs are likely to be different. Thankfully, WordPress does provide functionality to export posts on a per-user basis, and then when you import them, assign them to new users. I did a little testing and this worked well enough, but I hacked wordpress and wrote a small script to automate this and dump all the WordPress Export files (WXR) to the filesystem. I was able to run this script and grab all the export files.

As for importing the users, I dumped the wp_users and wp_usermeta table from each database, renamed them, and brought them all into a single database. I wrote a script that iterated through all the databases, matching users based on their login name. Thankfully, we only had two users that shared the same login names, so resolving that conflict was easy. I would collect all the information about a user spread across all the blogs, and then add it correctly to a new wp_users and wp_usermeta table. The trick was to know ahead of time what the new blog IDs in WPMU would be, then re-map the existing wp_user_level and wp_capabilities info to wp_3_user_level, and wp_capabilities, for instance.

Cursed Text Encoding

Another problem we ran into was incompatabilites with text encoding. Many of our entries had invalid characters in them. For compatability reasons with AxKit (which powers most of the Walker site) we’ve been using ISO-8859-1 encoding, which has a fairly limited range of characters. WordPress doesn’t do such a good job of forcing non-standard characters to be transformed into HTML entities, so anytime there was a proper apostrophe, curly quote or emdash, a freaky character would make it’s debut. Despite this, the posts continuted to work, the database threw a couple of warnings, but it continued to work.

The problem came with importing the exported WXR files. WordPress’ importer is very strict, and it would hang on any of these invalid characters. After trying multiple things to filter them out in the export or the database, I ended up using a Text Factory in BBEdit to replace them. Additionally, BBEdit has a wonderful command to “Zap Gremlins”, or take out weird characters in text. A few minutes of batch processing later, and the import files were clean.

Importing the cleaned WXR files (267 of them) was a tedious process that had to be done by hand, but allowed us to catch any errors and make sure the author assignment worked correctly.

VHOST vs. Sub-directory

WPMU can work in two different URL configurations. One is sub-domain based: joe.mydomain.com, steve.mydomain.com, etc. The other is sub-directory based: blogs.mydomain.com/steve, blogs.mydomain.com/joe. You have to decide what configuration to use when you install WPMU. We use a sub-directory based system for our blogs, but a couple other sites, teens.walkerart.org and air.walkerart.org, are also blogs, but use a sub-domain.

This would seem to leave us at somethinig of an impasse. All of my research confirmed there was no way to have both a sub-domain and sub-directory based URL structure. Several people on the WPMU forums were looking for a solution, but none was to be found.

However, there is a way, and it’s a thorougly good hack. Rather than try to force WPMU to work with both, we just let WordPress deal with sub-directory based URLs. Instead, we use a mod_rewrite to proxy the site to the sub-domain URL (your server must have mod_rewrite AND mod_proxy installed). The theme for the site has to be hacked to output all the links as the sub-domain URLs, even though it’s being served out of a sub-directory configuration. The teens site is a good example. If you visit blogs.walkerart.org/teens/, the site works just fine. Here’s the mod_rewrite that would go in the httpd.conf or .htaccess:

RewriteEngine On
RewriteRule ^(.*)$ http://blogs.walkerart.org/teens$1 [P]

All of the links will take you to a teens.walkerart.org URL. To re-write the URLs, you must replace all the get_permalink() function calls with a custom function that returns the sub-domain URL. And if you use any other functions, such as wp_get_pages or wp_list_cats() to output any links, you must again replace the sub-directory URL with the sub-domain URL. Here’s an example of the useful code:

define('BASE_URL','blogs.walkerart.org/teens/'); //original sub-directory URL
define('REWRITE_URL',"teens.walkerart.org/"); // new sub-domain URL
echo str_replace(BASE_URL,REWRITE_URL, wp_list_pages('hierarchical=0&title_li=&echo=0') );

//takes a given post or page ID and returns the permalink for a teens.walkerart.org permalink
//this is used for the teens site that's run on wordpress mu in a subdirectory configuration,
//but uses mod_rewrite and mod_proxy to relay URLs to a different domain
function get_teens_permalink($id){
    $perma = get_permalink($id);
    return str_replace(BASE_URL,REWRITE_URL,$perma);
}

Other useful bits

Aside from the sub-directory/sub-domain problems, there were not any other huge issues with our upgrade. A few little things in our themes had to be changed. Author pages needed to be made WPMU aware. Styles needed to be added for the galleries and captions that WP 2.5 and 2.6 added.

We also replaced blogs.walkerart.org with a page powered by WPMU, rather than the old axkit-based aggregation that was there previously. This is a custom theme that queries all the blogs and displays the current aggregation. The aggregated RSS feeds we offer was something we also needed to keep, but WPMU doesn’t offer a built-in way to do that. Thankfully, there is a plugin, WPMU-Sitewide-Feed, that takes care of it. The plugin is no longer maintained, but there is a version available that I was able to get to work with some slight modifications.

Another plug-in we’re now using is Comment Timeout. This plug-in is pretty simple. It closes comments on old and inactive posts to cut down on spam. It is a little smart, in that it won’t close comments on a “popular” post that has recent activity. In our old setup, we had been using the wonderful Spam Karma plugin to take care of spam. It worked really well, but it’s no longer under development and is not WPMU compatible. Instead, we’ve switched to using Defenseio, which is a more free and potentially smarter competitor to Akismet. Our filter is still being trained, but I have high hopes.