Went to this on Monday, it seems to have been a bit of an activity trigger, so thought I’d add – OK, bit of a copy-and-paste cheat for this one – it’s from an internal post I made for other people’s interest at work – internal references removed.

Earlier this week I went to http://www.meetup.com/es-london/calendar/14829629/ – a highly interesting & informative event about search technologies at The Guardian, a long-running UK newspaper at the forefront of digital publishing.

I’m expecting that the presentations given will be published online, in the interim here are some points you may find interesting:

• Initially various in-house and customised search technologies in the mid-90s, has had to grow massively in scope and content
• Now consolidating on Apache Solr – this is powerful enough to run out-of-the-box, yet scale to serve millions of documents across all their properties – they are even migrating 3rd-party applications
• To ensure the search is scalable, their search schema is kept simple – most searches are by freetext, and curated tags. They then use search features such as faceted search & related items to do the heavy lifting
• Expanding into Linked Data and Reference IDs – http://www.guardian.co.uk/open-platform/blog/linked-data-open-platform for more
• Even given they are in the content publishing business, published content is basically a frontend to a bunch of search queries – ’search as a platform’ – http://www.flax.co.uk/blog/2010/10/19/when-search-isnt-just-search-at-the-guardian/
• They are constantly adding content via a system of automated indexing/replication, 24 hours a day. They have optimised their publishing system for continual, incremental updates rather than large pushes
• They have also opened the technology, and the content – to the world – http://www.guardian.co.uk/open-platform describes their Open Platform
• That link is a bit of a kid-in-a-candy-store one (this must be how our users feel when we do a Data Upload ) – some sections I found useful are http://www.guardian.co.uk/open-platform/faq , http://www.guardian.co.uk/news/datablog , http://www.guardian.co.uk/open-platform/content-api-content-search-reference-guide , and…
• http://explorer.content.guardianapis.com/ to explore the content itself – http://content.guardianapis.com/search?q=spending+review&order-by=newest&format=json
  - http://explorer.content.guardianapis.com/#/search?tag=sport%2Fboxing%2Csport%2Fchess&order-by=newest&format=json
• Their strategy appears to be, as a content organization, to invest in the platform as a core resource, and then expose it for use by partners – “Our vision is to weave the Guardian into the fabric of the Internet, to become ‘of’ the web rather than ‘on’ the web.” They are already seeing large benefits with their commercial partners from this approach – http://blog.rodger-brown.com/2010/10/guardian-open-platform-it-or-innovation.html

Lastly, http://www.currybet.net/cbet_blog/2010/10/search-at-the-guardian-event.php is a summary from one of the speakers, with some good thoughts about ‘broken search’.

Recently At Work, we upgraded the platform running our next-generation music products.

One of the new features is having parts of a product be publically viewable to the world at large (not to mention search engines) – mostly the home page & browsing.

So here they are. Needless to say, not only am I very proud to be part of what we’re doing, I’m also pleased that I can now easily show these to people

Click on the link/image to browse each product. If you encounter a login page when seeing certain content, then you/your organization needs to be a paying customer…

Classical Scores Library

400,000 pages of the most important classical scores and manuscripts.

World Music

50,000 tracks that delivers the sounds of all regions from every continent. (This is the product with albums like “English Drinking Songs”, credits like “Unknown Camel Driver”, etc.)

African American Music Reference

50,000 pages that offers the first comprehensive coverage of many forms of black American musical expression.
American Song

50,000 tracks that allows people to hear and feel the music from America’s past.

Classical Music Reference Library (‘Baker’)

More than 30,000 pages of essential reference materials, spanning the entire history of Western classical music.

Garland Encyclopedia of World Music Online

The first comprehensive online resource devoted to music research of all the world’s peoples.

For our new flat, our employers have been asked to give a reference.

Here’s mine. Paul likes to brighten up people’s days, so instead of a boring form letter:

Off to my dad’s for the weekend before he & sue go to Australia/New Zealand for a month to see two of my sisters.

Am coming to dread the friday evening commute to Paddington, very crowded (one of the times I realise the Stevenage commute is a relatively good deal)and then waiting for an hour before they can be arsed to put some proper trains on. (read in cynical voice.)

This morning, pidgeons were keeping off the snow by walking along the rail tracks.

The snow was at its best phase-you can walk on streets without sliding but the snow is still over grass, etc. It is lovely and mottled.

One of those crappy free papers reminded me of how much in awe of London I was as a child. I’ve gotten so cynical of media.

My phone seems to automatically pick up dictionary words from texts, web posts, etc. So now it frequently suggests spellings which are themselves misslept. Awesome!

My sense of smell seems to be at a +3 modifier due to diet: can smell ‘dry’ food like sushi 20m away through a crowd on t’other side of a glass partition.

On the tube up, some suit guy had an artistically faded note in marker pen on his hand. It read “VOTE RENAL”. We’ll never know.

Hey! we’re looking for a few good men.. well one good man.

My company is hiring, looking for a Senior Software Developer, between 30k-40k, described at http://snipurl.com/php_dev_wanted . We’re advertising on Monster, development groups, etc., but if you know anyone suitable, well, just pass this post on…

Worked at home today. Best way to avoid temptation is by using the laptop and not the main PC and avoid ‘getting up to do something’ – recreate the office atmosphere at home!. Strange how ‘easily’ the day moves past.

Went out for walk in snow, it was lovely!

well, that’s the first time i’ve been unable to get on a train AT ALL at Stevenage. Often packed but never overfull. Fucking football.

I’m migrating my website from PHP to Ruby on Rails. Last night, moved the PHP source code form my old hosts to RailsPlayground. Whilst good value (they do Rails, and have about 10 times the bandwidth/storage/features as my old host for less money) their site/support forum seems to have very dodgy availability. So was largely on my own for figuring out how to host PHP as I’m not that experienced in remote barebones hosting.

Turns out Rails routing applies across the whole site, not just in the railsapp directory, so PHP has to end in .php, and it has to go into the public folder, which in hindsight makes sense (most PHP apps live in the public area of a site).

Here’s the roadmap:

• I have until the nameservers update to arrange for pretty URLs like vodex.net/goalpost to point to dirty ones like vodex.net/goalpost/index.php – don’t want to mess with visitors/search engines. This is a temporary kludge, so I’ll probably just alter Apache conf. a bit. I only have one Rails app live at present (Universe on Rails) and it’s low profile so it’s not important as I’ll…
• Install a Ruby CMS/blog like Typo, and use it instead of the custom PHP setup I used to have.
• Develop something like my current Wordpress Livejournal importer for Typo, using the LiveJournal Ruby module, thus standing on shoulders of giants (LJ’s XML-RPC API seems a little counter-intuitive).
• Move on from there.

In related news, work upon a mockup document management system at the University proceeds apace, partially due to productivity gains from said Ruby on Rails. It’s true – Rails encapsulates best practice out of the box, leading to much better overall progress despite the learning curve. It’s all about the framework – Stuff that would have taken me months in PHP (even with PEAR) has taken just days in Rails. It’s also giving me experience in tools I didn’t make use of when developing GoalPost – Subversion version control, test-driven development, and similar agile methodologies.

At present I’m changing the view from an initial makeshift scaffold layout involving fieldsets to a much solider layout. Changing layout in rails is a snap, much easier than, say, in PHP, where you ended up with HTML-producing code inside logic classes *cough*.  Good workflow here – determine a view/layout (separate from the concerns of the logic creating it), then produce the current design in Dreamweaver or Nvu, say, and then componentize it into layouts. Much cleaner.

My word! Even though I’ve been working on it for nearly six months, seeing this conference programme that I’m delivering a presentation in, is a warming sight indeed. If you’re interested, here is the information sheet!