Today we’re very pleased to announce the full release of the new and improved version of our online newspaper making tool, ARTHR. It’s been in testing for a while, but it’s now the default tool if you go into your dashboard and press “Make a Newspaper“. Hurrah!
As part of this, we’re shutting down the old version of ARTHR, now known as ARTHR Classic. This post details how the shutdown will work, and if you’re still using it, important dates to finish your work by.
As of today you can still make a paper in ARTHR Classic, but it’s not recommended unless you have a good reason. If you do, we’d like to hear about it, so we can see how the new version of ARTHR might meet your needs.
Barring any problems, on 18th March, we’ll stop people being able to make new newspapers in ARTHR Classic. If you’ve already started one before then, it’ll still be editable.
Then, on 1st April, we’ll stop people being able to edit newspapers in ARTHR Classic, and the tool will disappear from the site. On that date your newspaper will be converted to a PDF on our site, and you’ll be able to download the contents as a PDF, InDesign document or a ZIP file containing the stories and pictures.
If you’re working on a paper in ARTHR Classic you’ll need to finish and order it before then. Unfortunately it’s not possible to automatically move your paper from ARTHR Classic to the new version of ARTHR as the structure of the document is significantly different.
If any of that causes you a problem, please let us know and we’ll do our best to help out.
We thought it might be interesting to take you, the dear reader, on a little tour through ARTHR II and how it all works. This post is unashamedly technical, so if all gets a little bit too much, don’t worry: like the Underground at rush hour, there will be another post along shortly.
I will start at the beginning, for the beginning is a good place to start. ARTHR II (henceforth known as ARTHR – the Automated Real-time Text Handling Resource), is our online newspaper layout tool. It provides a drag-and-drop WYSIWYG style interface to make a great looking newspaper quickly and easily. And it’s entirely browser based, with no plugins required, and nothing to download.
It looks a bit like this:
(I know I’ve used this video twice before, but dammit, I’m going to use it again. If you don’t have the time to poke around with ARTHR, it should give you an idea of what you can do with it. If you do have time to have a go, head over here and try it out!)
There are three main components, all of which integrate tightly together.
The backend Ruby on Rails application: handles storage, persistence, integration with the main Newspaper Club site, background jobs, etc.
ERNIE, a Mac OS X/Cocoa rendering server: takes a document layout and converts it into JPG previews or print-ready PDFs
Much of the logic and intelligence in ARTHR runs in the browser, putting it as close to the user as possible to make it as fast as possible.
With so much of the app’s complexity running in the user’s browser, it becomes much harder to track down tricky bugs. To help with this, we wrote our own logging engine, with a server-side component. Browser exceptions are automatically sent over, along with a stracktrace if available, and contextual information like the git revision in use, and the IP address. For persistently tricky browser/OS combinations, we can flip a user’s account into remote debug mode, sending us live log statements, effectively letting us tail a user’s actions and get good diagnostic information.
The backend of ARTHR is quite a slim component, mostly handling persistence, storage of uploaded images, validation checks, and providing a conduit for the frontend to communicate with ERNIE, the rendering engine.
It’s a Rails 3.2 application, running a Postgres database, with Sidekiq for background jobs. We’re making good use of the hstore type to be able to store arbitrary key/value pairs for different content types.
All that sits in front of nginx, running the brilliant nginx-push-stream-module. This lets us handle long-running EventSource/Server-Sent Event connections while still using our standard Unicorn single-threaded Rails server. We just make an HTTP POST request containing the message payload to a specific channel, and it arrives at any clients listening on that channel shortly after. It’s simple, it just works, and keeps the number of server components to a minimum.
When we developed the first version of ARTHR, a few years ago, we used InDesign Server to provide layout, high quality typography with PDF and JPG output. It served us fairly well, but it’s difficult to integrate tightly with, and it’s often very slow. We needed to speed it up by a couple of orders of magnitude, and have much finer information about layout and text flows. It simply wasn’t possible to do that with IDS.
We needed something to let us layout a document, and quickly convert to both JPG (for previews) and PDF. To make things a little more complex, it needs to support CMYK, and integration with a high-quality typography library for things like ligatures, hyphenation and justification.
After working through most of the options, we ended up at Quartz, part of Cocoa’s CoreGraphics stack in OS X. It’s fast – really fast – because it’s designed to run at 60fps on the desktop. Because Quartz maps very closely to the PDF format rendering to a PDF context is fast too. And it natively supports CMYK throughout, something that Skia and Cairo, two of the main open-source rasterisation engines don’t. For typography there’s the Core Text Layout System, a collection of classes for handling typography and typesetting.
Using all this we built ERNIE, the Expertly Rendered Newspaper Internet Engine. It’s an OS X 10.8 application which exposes an HTTP RPC interface, receiving requests for content size calculations, or document rendering, and returning arrays of JPGs or PDF files.
We’re really pleased with the performance. It typically takes ~50ms to render a simple, text heavy document, up to ~700ms for a longer, more graphics heavy document. It’s often 100-200x faster than comparable documents in InDesign Server.
The PDFs Quartz produces have a few oddities, so we post-process them in Ghostscript to improve compatibility with our presses and ensure they’re PDF/X-3:2002 compatible.
Putting it all together
Let’s walk through adding a new bit of text to your document, as an illustration of how this all pieces together.
The user types into the text box and hits save. The frontend immediately sends the new content to the backend for storage. During that request, the backend takes the new content and performs a test rendering against ERNIE, calculating how the text will flow, where the paragraph breaks are, the number of lines per paragraph, etc.
When the request returns, the frontend uses the calculated flow to perform a prediction of where the text will display in the document, and renders the new piece of text in the blueprint. It’s then ready to be moved around the document by the user.
The preview images are now out of date, as they don’t include the piece of text we just added, so the frontend fades them out, and requests a rebuild of them. The backend returns a unique ID for the rebuild request, which the frontend uses to listen for changes in the state of the rebuild, using an EventSource request connected to the nginx-push-stream-module mentioned earlier.
The rebuild request is picked up by one of the background workers which takes the new document structure, including the new text, and passes it through to ERNIE. ERNIE processes the document layout and returns an array of JPG files. These are written to disk by the background worker, and the build is marked as completed. The frontend, listening for changes to the build status, is immediately notified, and requests those JPG previews from the server. As they load, it fades them in and they become visible to the user.
For most users this all happens in less than half a second, end to end. And if the user moves the piece of text around, it all happens again. Of course, the user doesn’t have to wait for the preview images before continuing to edit their document – it’ll keep them up to date as they go.
Hopefully that’s given you an idea of how an application like ARTHR works, all the way through the stack. We’re very happy with the technologies we’ve chosen, and we’re very grateful to everyone who has contributed to all the open source projects we’re building upon.
We’re keen to chat to folks working on similar online publishing tools, and always open to sharing what we’ve learnt. If that’s you: get in touch!
Things are proceeding apace in the twin Newspaper Club offices of London and Glasgow. It’s shaping up to be a busy year already, with a sortie of new products and services being planned and built as I type, right this minute.
We released some improvements to our Dashboard pages yesterday, including a sneaky little link to try out the new version of ARTHR. Go on, click it.
ARTHR II (internal codename: Print Harder) will soon be our main tool, and we’ll be turning off ARTHR I shortly after. Now’s your opportunity to get ahead of the game and let us know what you think.
New features include:
Drag things around with your mouse!
Style your text with all sorts of headers and formatting and stuff like that!
Choose from a number of typefaces and sizes!
Superfast performance using our proprietary Expertly Rendered Newspaper Internet Engine (ERNIE)!
Automatically pull a webpage into your paper!
And much, much, more!
If you’re making a paper in the old version of ARTHR, don’t worry – we’ll give you plenty of notice before turning it off, and all your work will be available to download in a number of handy formats.
Phew, it’s been a busy few months in the Engineering Dept., but the last fortnight has been especially full on.
Last week we released the new version of ARTHR to a few customers who had used the old one, to see what they thought of it.
We received lots of very useful feedback and discovered lots of bugs too, both little and large. We’re ploughing through all of that at the moment, releasing tweaks and fixes a few times a day. Thank you everyone who helped out.
The first 90% of the project is always great: you’re breaking new ground, inventing something new, and everything is exciting and good.
The second 90% of the project gets a bit trickier: now you have to make difficult compromises about how to prioritise your time, and the realities of putting your creation into the hands of other people hit home.
We’re well into the third and final 90%. Tracking down frustrating browser bugs, trying to fix the problem with bad connections from corporate networks, wrangling the produced PDFs to be prepress ready.
The good news is that every couple of days I surface from wrestling an especially gnarly PDF problem, or tweaking the server configuration, and I go and have a proper play with it. And I think it’s still good.
For a little flavour here’s a spin through that I just recorded. (Play spot the bug!)
We’ve not been as talkative as we used to be about running the business. That’s mostly because there’s a lot more business than there used to be – both customers, and all the stuff to support that: profit and loss reports, management accounts, Skype calls, meetings, planning.
But that’s no excuse. We’re into the really difficult stage of running a business. We’re still small, but there’s real money, real jobs and real pressures. But, this is also where the meaty stuff is. We’ve learnt a lot, and we’re learning even more. Those thing seem worth sharing, and if not for you, then for our own sake. We’re going to try harder.
So, what’s going on?
For a start, we’ve had two brilliant new people join us. Rosemary is up in Glasgow working full-time with Anne, Emily and Silje to manage all operations, customer service, and well… everything. This is brilliant, because a) Rosemary is brilliant, and b) Anne can go on holiday now.
And Mike is down in London, working with me, to develop the site and future products and services. This is brilliant, because a) Mike is brilliant, and b) I can go on holiday now. He’s sat to the right of me, looking very serious, staring at the Backbone.js documentation.
We’ve been joined part-time over the summer by Jase, who is helping us with some design work. He’s sat opposite me, doing some typing wearing a great pair of red headphones.
Behind me is a whiteboard. It says: “ARTHR II: PRINT HARDER”, with lots of post-it notes underneath.
I’m really worried about talking about stuff before it’s real. Talk is easy, and shipping is hard. But dammit, I’m really excited about this. So, if you promise not to hold us to it, we’re completely overhauling ARTHR, using everything we’ve learnt about how people have used it over the last couple of years. When there’s something to show, you’ll see it here first.
MD and Customer Service have gone for lunch, so it’s time for Engineering to get on the blog and crank out a badly worded and ill-informed press release full of crimes against verbs. Hold onto your browser.
Newspaper Club (London, United Kingdom & Glasgow, Glorious Independent Republic of Scotland), are pleased to announce the release of the next generation of PDF uploading.
This significant advancement will make it even easier to leverage the Newspaper Club printing platform, and seamlessly synergize your credit card with our state of the art Web-Site.
From today, all PDF files uploaded to Newspaper Club will be scanned by the unique PDF management solution before ordering. Valued customers will immediately receive a report detailing common issues, with helpful hints on how to resolve them going forward.
Common issues, such as fonts not embedded correctly, or the use of spot colours, will be highlighted in a simple report, indicating the page or the area in which they occur, allowing the customer to take action to resolve them, or to proceed regardless.
Anne Ward, Managing Director at Newspaper Club, noted, “Newspaper Club is a world leader in the uploading and ordering of PDFs for newspaper printing purposes, and this advance will cement our position at the top of the table.”
“The 1700 page PDF file specification has almost killed me, so this better be worth it”, mumbled Tom Taylor, Head of Engineering. “But I ask customers with any feedback about this valuable service to send it in, allowing us to action their comments.”
Newspaper Club’s commitment to and sponsorship of the innovative, open source, Ruby based, pdf-preflight gem will enable others to leverage their effort and contribute onward improvements.
About Newspaper Club:
Newspaper Club helps anyone make and print their own newspapers. Since 2009, Newspaper Club has printed more than a million newspapers, and in 2010, was the recipient of a Design Museum Designs of the Year award. Learn more at www.newspaperclub.com.
There’s a nice little scroller, and you can click through to the paper on the Newspaper Club site. The medium size is designed to fit neatly into a typical blog post, and there’s a slightly larger option too.
By default anyone can embed a paper, but if you want to turn it off, just untick the option in the sharing settings.
It turns out writing the code to do this is a bit tricker than we thought, so if you notice the design looking a bit wonky on your site, please let us know.
I had dinner with some friends a while ago and we were chatting about what Newspaper Club was up to. The conversation went something like this:
Him: “What kind of things do you print then?”
Me: “Oh well, all sorts of stuff really.”
Him: “Like local newspapers and things like that?”
Me: “Yes, some of those, but really just anything that people want to put on newsprint: comics, portfolios, wedding papers, wrapping paper.”
Him: “Wrapping paper?”
But for our customers, apart from our blog posts, there’s no way of seeing the full range of stuff that other people are printing. We want to surface more of these fantastic papers; to give people a space to show off what they’ve made, and why and how they did it, beyond the reach of the printed paper.
So today we’re beginning that. There’s now an option on each newspaper in your dashboard to share it:
There’s space to write a few words about it, to choose how much of it you want to share, and to tag it with a few keywords. You’ll end up with a page like this one of prettymaps from my profile, that you can share with anyone:
When we’ve got a few more newspapers shared, we’ll open the Newsagent: a portion of the site to allow anyone explore all the shared newspapers, searching by tag or description to find papers they might be interested in. And we’ll be featuring papers and publications that we love on the front page and throughout the site.
But that’s a post for another day. For now, give it a go, and let us know if you have any feedback.
We’re looking for beta testers for a new bit of the Newspaper Club site. If you’ve made a newspaper with us, either in ARTHR or uploaded as a PDF, and are interested in promoting it, sharing it, or just having a page for it on our site, then we’d love to hear from you.
Drop us an email to email@example.com, with the subject line ‘Beta Testing’ and the email address of your account. If you’ve got some of the details of the paper you made, that’d be great too. We can’t guarantee we’ll get back to everyone, but we’ll try our best.
The rest of you: the Newspaper Club Newsagent is coming soon.
Much of our inspiration for starting Newspaper Club is down to Aaron Straup-Cope, and his Papernet projects. Aaron’s thoughts on how paper and printed media can be a natural part of the web and the “network” encouraged us to play around, led to experiments with newspapers, and lo: Newspaper Club was born.
A lot of what we do at Newspaper Club is about making it easy as possible to access newsprint: to lower the barrier of entry and let people make whatever they want. We’ve built ARTHR, our layout tool, we’ve written lots of help pages, and we try and be helpful on email and the phone, all to try and make it as easy as possible.
But we want to make it easier for machines too. Because why should humans have all the fun? Machines should be able to make good looking newspapers too.
So, today we’re launching the alpha release of the Newspaper Club API. The API provides programatic access to ARTHR, letting you write tools and apps that generate newspapers from any content you have.
We’ve written some API documentation which should help you along the way. We know it needs fleshing out in places, but if you’ve got any suggestions for what we’re not explaining very well, please let us know.
As a demonstration, we’ve built a tool nicknamed The Telepaper, that turns a Readability Reading List into a newspaper with just a couple of clicks.
As luck and timing would have it, The Engineering Dept. entered this in Readability’s API contest and are very pleased that it won third place! Hurrah! (Thank you Readability folks.)
The Telepaper is a very simple Ruby Sinatra application that glues the Newspaper Club and Readability APIs together. All the source code is available on the Newspaper Club Github account, and you can try the application for yourself if you’ve got a Readability account.
If you’d like to have a go with the API yourself, you’ll need an API key from us. First take a look at the documentation, then drop us an email containing your Newspaper Club account’s email address, an OAuth callback URL, and a brief description of whatever you’re playing around with, if you know. We’ll have a web interface for this as we tidy things up and enter the beta stage.
We’ll be honest: this is going to be a bit of a learning curve for us. Running an API is hard. And mapping the concepts of print layout to an API is hard. So it’s likely that we won’t have got it right first time. Do let us know your thoughts.