How ARTHR & ERNIE work: Backbone.js, Rails, Cocoa, and more.

We thought it might be interesting to take you, the dear reader, on a little tour through ARTHR II and how it all works. This post is unashamedly technical, so if all gets a little bit too much, don’t worry: like the Underground at rush hour, there will be another post along shortly.

I will start at the beginning, for the beginning is a good place to start. ARTHR II (henceforth known as ARTHR – the Automated Real-time Text Handling Resource), is our online newspaper layout tool. It provides a drag-and-drop WYSIWYG style interface to make a great looking newspaper quickly and easily. And it’s entirely browser based, with no plugins required, and nothing to download.

It looks a bit like this:

(I know I’ve used this video twice before, but dammit, I’m going to use it again. If you don’t have the time to poke around with ARTHR, it should give you an idea of what you can do with it. If you do have time to have a go, head over here and try it out!)

Overview

There are three main components, all of which integrate tightly together.

ARTHR/ERNIE

  • The frontend Javascript application: the entire user interface, some layout intelligence, and more.
  • The backend Ruby on Rails application: handles storage, persistence, integration with the main Newspaper Club site, background jobs, etc.
  • ERNIE, a Mac OS X/Cocoa rendering server: takes a document layout and converts it into JPG previews or print-ready PDFs

The Frontend

Much of the logic and intelligence in ARTHR runs in the browser, putting it as close to the user as possible to make it as fast as possible.

The frontend is mostly a Backbone.js application, written in classical, Crockford-compliant, Javascript. We’ve been very happy with Backbone – it provides just enough structure to support building things quickly, but without forcing you to go ‘all in’ to their way of doing things. There’s plenty of room to pick your own templating language, or drop down to your own AJAX requests, without feeling like you’re going against the grain. It cleanly separates the presentation layer (HTML/CSS) from the logic, leading to good quality, accessible HTML. And last, but most certainly not least, there’s no magic going on: the entire source code is imminently readable, and we’ve regularly walked through it, by eye or in the debugger, to check we understand the behaviour of something.

We’ve tried to write it as you would a well-structured Object Orientated GUI application, and took much from Soundcloud’s blog post about building Next Soundcloud.

With so much of the app’s complexity running in the user’s browser, it becomes much harder to track down tricky bugs. To help with this, we wrote our own logging engine, with a server-side component. Browser exceptions are automatically sent over, along with a stracktrace if available, and contextual information like the git revision in use, and the IP address. For persistently tricky browser/OS combinations, we can flip a user’s account into remote debug mode, sending us live log statements, effectively letting us tail a user’s actions and get good diagnostic information.

The Backend

The backend of ARTHR is quite a slim component, mostly handling persistence, storage of uploaded images, validation checks, and providing a conduit for the frontend to communicate with ERNIE, the rendering engine.

It’s a Rails 3.2 application, running a Postgres database, with Sidekiq for background jobs. We’re making good use of the hstore type to be able to store arbitrary key/value pairs for different content types.

All that sits in front of nginx, running the brilliant nginx-push-stream-module. This lets us handle long-running EventSource/Server-Sent Event connections while still using our standard Unicorn single-threaded Rails server. We just make an HTTP POST request containing the message payload to a specific channel, and it arrives at any clients listening on that channel shortly after. It’s simple, it just works, and keeps the number of server components to a minimum.

ERNIE

When we developed the first version of ARTHR, a few years ago, we used InDesign Server to provide layout, high quality typography with PDF and JPG output. It served us fairly well, but it’s difficult to integrate tightly with, and it’s often very slow. We needed to speed it up by a couple of orders of magnitude, and have much finer information about layout and text flows. It simply wasn’t possible to do that with IDS.

We needed something to let us layout a document, and quickly convert to both JPG (for previews) and PDF. To make things a little more complex, it needs to support CMYK, and integration with a high-quality typography library for things like ligatures, hyphenation and justification.

After working through most of the options, we ended up at Quartz, part of Cocoa’s CoreGraphics stack in OS X. It’s fast – really fast – because it’s designed to run at 60fps on the desktop. Because Quartz maps very closely to the PDF format rendering to a PDF context is fast too. And it natively supports CMYK throughout, something that Skia and Cairo, two of the main open-source rasterisation engines don’t. For typography there’s the Core Text Layout System, a collection of classes for handling typography and typesetting.

Using all this we built ERNIE, the Expertly Rendered Newspaper Internet Engine. It’s an OS X 10.8 application which exposes an HTTP RPC interface, receiving requests for content size calculations, or document rendering, and returning arrays of JPGs or PDF files.

ERNIE logs

We’re really pleased with the performance. It typically takes ~50ms to render a simple, text heavy document, up to ~700ms for a longer, more graphics heavy document. It’s often 100-200x faster than comparable documents in InDesign Server.

The PDFs Quartz produces have a few oddities, so we post-process them in Ghostscript to improve compatibility with our presses and ensure they’re PDF/X-3:2002 compatible.

Putting it all together

Let’s walk through adding a new bit of text to your document, as an illustration of how this all pieces together.

The user types into the text box and hits save. The frontend immediately sends the new content to the backend for storage. During that request, the backend takes the new content and performs a test rendering against ERNIE, calculating how the text will flow, where the paragraph breaks are, the number of lines per paragraph, etc.

When the request returns, the frontend uses the calculated flow to perform a prediction of where the text will display in the document, and renders the new piece of text in the blueprint. It’s then ready to be moved around the document by the user.

The preview images are now out of date, as they don’t include the piece of text we just added, so the frontend fades them out, and requests a rebuild of them. The backend returns a unique ID for the rebuild request, which the frontend uses to listen for changes in the state of the rebuild, using an EventSource request connected to the nginx-push-stream-module mentioned earlier.

The rebuild request is picked up by one of the background workers which takes the new document structure, including the new text, and passes it through to ERNIE. ERNIE processes the document layout and returns an array of JPG files. These are written to disk by the background worker, and the build is marked as completed. The frontend, listening for changes to the build status, is immediately notified, and requests those JPG previews from the server. As they load, it fades them in and they become visible to the user.

For most users this all happens in less than half a second, end to end. And if the user moves the piece of text around, it all happens again. Of course, the user doesn’t have to wait for the preview images before continuing to edit their document – it’ll keep them up to date as they go.

The End

Hopefully that’s given you an idea of how an application like ARTHR works, all the way through the stack. We’re very happy with the technologies we’ve chosen, and we’re very grateful to everyone who has contributed to all the open source projects we’re building upon.

We’re keen to chat to folks working on similar online publishing tools, and always open to sharing what we’ve learnt. If that’s you: get in touch!

Posted by Tom | Comments (3)

Filed under: engineering

← Previously: Next:

3 Comments

  1. Sun, 10 Feb 2013 at 4:16 pm
    nickf Permalink

    Hey, great writeup! Very sensible reasoning about your front end — it’s definitely one of the things I like about Backbone that for a lot of things it basically just shrugs and lets you sort it out yourself. Bit slower to get a new project bootstrapped, but you ultimately get a flexible framework which suits your needs. Glad to hear the SoundCloud blog post was helpful for you, too.

    Cheers,
    Nick

  2. Sun, 10 Feb 2013 at 8:01 pm
    Infovore » Links for February 10th Permalink

    [...] How ARTHR & ERNIE work: Backbone.js, Rails, Cocoa, and more. | Newspaper Club Newspaper Club is a great product – but I'm really glad Tom's written about the technical underpinnings of the latest version of the code, because it's super impressive. They threw out InDesign and replaced it with their own renderer, written in Cocoa; they have a gorgeous, rich Javascript client that's a joy to use; and they have a development team of 2. TWO. Brilliant work, folks. (tags: newspaperclub engineering code programming architecture geniuses ) [...]

  3. Wed, 13 Feb 2013 at 11:17 am
    Amit Permalink

    This is just brilliant, Tom!

    You’ve quietly constructed an entire web-to-print system without InDesign, while dealing with something as complex as layout in columns. I especially like the ability to keep working while previews eventually make their way to the UI. The bit before the actual rendering preview – the immediate predicted flow in a red box – after someone moves text around, is inspired. I always wondered how you might crack that one – since the lets-wait-and-see approach with InDesign was always a drag.

    Could you possibly apply your stack to an arbitrary .INDD/.IDML document which had marked up layers or tags for “editable bits”? I know you want to move away from InDesign server, but there’s great possibilities if you could serve up a pre-designed template – made by an awesome designer for example – and expose a UI to just insert bits of text where it’s possible/allowed to do so. No idea what the end-product might be, but it does mean that print designers would be able to create re-usable things that developers can’t.

    Amit

RSS feed for comments on this post

Sorry, the comments are closed right now.