We thought it might be interesting to take you, the dear reader, on a little tour through ARTHR II and how it all works. This post is unashamedly technical, so if all gets a little bit too much, don’t worry: like the Underground at rush hour, there will be another post along shortly.
I will start at the beginning, for the beginning is a good place to start. ARTHR II (henceforth known as ARTHR – the Automated Real-time Text Handling Resource), is our online newspaper layout tool. It provides a drag-and-drop WYSIWYG style interface to make a great looking newspaper quickly and easily. And it’s entirely browser based, with no plugins required, and nothing to download.
It looks a bit like this:
(I know I’ve used this video twice before, but dammit, I’m going to use it again. If you don’t have the time to poke around with ARTHR, it should give you an idea of what you can do with it. If you do have time to have a go, head over here and try it out!)
There are three main components, all of which integrate tightly together.
- The backend Ruby on Rails application: handles storage, persistence, integration with the main Newspaper Club site, background jobs, etc.
- ERNIE, a Mac OS X/Cocoa rendering server: takes a document layout and converts it into JPG previews or print-ready PDFs
Much of the logic and intelligence in ARTHR runs in the browser, putting it as close to the user as possible to make it as fast as possible.
We’ve tried to write it as you would a well-structured Object Orientated GUI application, and took much from Soundcloud’s blog post about building Next Soundcloud.
With so much of the app’s complexity running in the user’s browser, it becomes much harder to track down tricky bugs. To help with this, we wrote our own logging engine, with a server-side component. Browser exceptions are automatically sent over, along with a stracktrace if available, and contextual information like the git revision in use, and the IP address. For persistently tricky browser/OS combinations, we can flip a user’s account into remote debug mode, sending us live log statements, effectively letting us tail a user’s actions and get good diagnostic information.
The backend of ARTHR is quite a slim component, mostly handling persistence, storage of uploaded images, validation checks, and providing a conduit for the frontend to communicate with ERNIE, the rendering engine.
It’s a Rails 3.2 application, running a Postgres database, with Sidekiq for background jobs. We’re making good use of the hstore type to be able to store arbitrary key/value pairs for different content types.
All that sits in front of nginx, running the brilliant nginx-push-stream-module. This lets us handle long-running EventSource/Server-Sent Event connections while still using our standard Unicorn single-threaded Rails server. We just make an HTTP
POST request containing the message payload to a specific channel, and it arrives at any clients listening on that channel shortly after. It’s simple, it just works, and keeps the number of server components to a minimum.
When we developed the first version of ARTHR, a few years ago, we used InDesign Server to provide layout, high quality typography with PDF and JPG output. It served us fairly well, but it’s difficult to integrate tightly with, and it’s often very slow. We needed to speed it up by a couple of orders of magnitude, and have much finer information about layout and text flows. It simply wasn’t possible to do that with IDS.
We needed something to let us layout a document, and quickly convert to both JPG (for previews) and PDF. To make things a little more complex, it needs to support CMYK, and integration with a high-quality typography library for things like ligatures, hyphenation and justification.
After working through most of the options, we ended up at Quartz, part of Cocoa’s CoreGraphics stack in OS X. It’s fast – really fast – because it’s designed to run at 60fps on the desktop. Because Quartz maps very closely to the PDF format rendering to a PDF context is fast too. And it natively supports CMYK throughout, something that Skia and Cairo, two of the main open-source rasterisation engines don’t. For typography there’s the Core Text Layout System, a collection of classes for handling typography and typesetting.
Using all this we built ERNIE, the Expertly Rendered Newspaper Internet Engine. It’s an OS X 10.8 application which exposes an HTTP RPC interface, receiving requests for content size calculations, or document rendering, and returning arrays of JPGs or PDF files.
We’re really pleased with the performance. It typically takes ~50ms to render a simple, text heavy document, up to ~700ms for a longer, more graphics heavy document. It’s often 100-200x faster than comparable documents in InDesign Server.
The PDFs Quartz produces have a few oddities, so we post-process them in Ghostscript to improve compatibility with our presses and ensure they’re PDF/X-3:2002 compatible.
Putting it all together
Let’s walk through adding a new bit of text to your document, as an illustration of how this all pieces together.
The user types into the text box and hits save. The frontend immediately sends the new content to the backend for storage. During that request, the backend takes the new content and performs a test rendering against ERNIE, calculating how the text will flow, where the paragraph breaks are, the number of lines per paragraph, etc.
When the request returns, the frontend uses the calculated flow to perform a prediction of where the text will display in the document, and renders the new piece of text in the blueprint. It’s then ready to be moved around the document by the user.
The preview images are now out of date, as they don’t include the piece of text we just added, so the frontend fades them out, and requests a rebuild of them. The backend returns a unique ID for the rebuild request, which the frontend uses to listen for changes in the state of the rebuild, using an EventSource request connected to the nginx-push-stream-module mentioned earlier.
The rebuild request is picked up by one of the background workers which takes the new document structure, including the new text, and passes it through to ERNIE. ERNIE processes the document layout and returns an array of JPG files. These are written to disk by the background worker, and the build is marked as completed. The frontend, listening for changes to the build status, is immediately notified, and requests those JPG previews from the server. As they load, it fades them in and they become visible to the user.
For most users this all happens in less than half a second, end to end. And if the user moves the piece of text around, it all happens again. Of course, the user doesn’t have to wait for the preview images before continuing to edit their document – it’ll keep them up to date as they go.
Hopefully that’s given you an idea of how an application like ARTHR works, all the way through the stack. We’re very happy with the technologies we’ve chosen, and we’re very grateful to everyone who has contributed to all the open source projects we’re building upon.
We’re keen to chat to folks working on similar online publishing tools, and always open to sharing what we’ve learnt. If that’s you: get in touch!