Tom, our co-founder and Head of Engineering, has been down the server mines prepping our systems for a big move over to AWS. He’s leaving us at the end of the month so this is going to be one of his last big technical outings at Newspaper Club. You can read a bit about the clever bits here.
By the time The Great Server Move of 2015 is complete we’ll be able to use words like ‘high availability,’ ‘failover,’ and ‘nuclear bomb proof’ and not just be saying it to sound impressive. Unless it all goes horribly wrong.
Before a recent trip to Berlin I wanted to experiment with printing a newspaper to use as photo filters.
A single copy of a digital tabloid with a shape, pattern or colour on each page.
The pages then work as filters. To do this I expose each photo twice: the first exposure with my camera pointed at the thing I want to photograph and the second exposure pointing at the newspaper. This can be done with any SLR/DSLR or a smartphone (may require an app).
The newspaper is ideal for a portable set of filters– light, easy to fold into a pocket. Plus the texture of the newsprint adds to the charm of the overall effect.
It’s easier to shoot the subject first and the newspaper second. Anything white on the newspaper page will be obscured in the final image, with detail appearing in the dark areas.
I tried to remain experimental with these photos, not anticipating the effect the filter would have on the image and enjoying unexpected results.
As an experiment it worked well. On the next paper I’m going to keep the patterns to a minimum and use simpler shapes and colours.
We’re happy to announce that we launched some new product pages yesterday. We hope they’ll make it easier to figure out which type of paper will best suit your project, with a handy tool to calculate how much your newspaper run will cost and when it will be delivered. We’ve also added links to some brilliant examples of each size and style, to show off the many ways our customers have made the most of the formats. Let us know what you think!
Regular readers will know that Newspaper Club is split between two offices: London and Glasgow. Glasgow is now the HQ, where we do all the operations, logistics, customer service, and much more. And London is where we develop the products and services, writing code, drawing pictures, and so on.
We have a Campfire chat, which we natter in all day long, but for a while I’ve been trying to find other ways of joining the offices together that are less direct and a bit more, well, silly.
Our first go at this is the awfully titled ‘Project Looking Glass’. It’s a two way video screen that sits in the corner of each office, letting us wave at the other office, and them to wave back. It looks a bit like this:
It’s always on, it doesn’t need to dial up or sign in, and if the network drops, it should recover as soon as it can.
We’ve only had it running for a couple of days, so it’s a bit of an experiment for us. So far it seems quite fun, with some good drawings appearing on whiteboards. But that might fade, and it might end up being a bit weird, in which case we’ll turn it off and try something else.
But we also know that we’re going to have to get better at remote working, as we grow and as our business get more complicated. It feels like we can learn a lot from trying things like this out while we’re small enough, and those things will be useful as more people join us, in our offices and out.
We’ll keep you posted on how it’s working out for us, and if you end up making something similar for your offices, we’d love to hear how it’s going.
The Engineering Dept have been hard at work on ARTHR (pictured above), and we’ve made a few improvements we’d like to tell you about.
First of all, you no longer need to make an account to get going with ARTHR. Just head over to the site, start making a newspaper, and when you’re ready, sign up and we’ll save your paper into your new account.
We encourage you to make an account as soon as possible after you’ve decided whether ARTHR is for you: not because we’re greedy for your details, but in case you move computers or change browser, and we can’t match you up with your newspaper any more – we don’t want you to lose your paper!
There’s also a lovely new homepage for ARTHR, so tell all your friends.
But that’s not all! We’ve also:
Revamped the controls at the top of the screen, based on your feedback, making it clearer how to add and remove pages from your newspaper.
Added a feature to be able to add an image to your newspaper directly from a URL. You can even give us a Flickr or Instagram photo page and we’ll do the hard work to fetch the image out of it.
Made adding a new piece of content smarter, so it doesn’t overlap anything already on the page, picking the next suitable space in your newspaper.
And finally, a little reminder that the best paper in the Newsagent each month wins a £100 voucher. The Engineering Dept will even throw in a Twix if it’s made in ARTHR.
This is my kettle. There are many like it, but this one is mine. Kettles form a core part of Newspaper Club operations, and it’s important to pick one that has a number of key features to ensure a quality cup of tea. Below I will detail the factors that contribute to this kettle’s daily success, from a systems engineering point of view.
Uptime is important with critical pieces of infrastructure. Things that move are things that break. This kettle has few moving parts – no electric heating element, or on/off switch. It’s heated by applying energy (ignited gas) to the base and waiting. This gives me the confidence that I can trust it in daily operations for many decades.
However, if it was to fail, (say the handle fell off), it’ll still be partially usable (with an oven glove) until a repair can be made at a convenient time.
It’s not the fastest kettle, but by varying the gas throughput I can vary the time it takes to boil. If I dial it back a bit, it takes just a little less time to boil than my porridge takes to cook, so the tea is brewed just as the porridge is done.
Given a typical UK energy mix, heating things with gas is more efficient than heating things with electricity. However, as more of the UK energy mix (hopefully) shifts to renewable sources over the next few decades, gas will have a relatively larger carbon impact.
Today we’re very pleased to announce the full release of the new and improved version of our online newspaper making tool, ARTHR. It’s been in testing for a while, but it’s now the default tool if you go into your dashboard and press “Make a Newspaper“. Hurrah!
As part of this, we’re shutting down the old version of ARTHR, now known as ARTHR Classic. This post details how the shutdown will work, and if you’re still using it, important dates to finish your work by.
As of today you can still make a paper in ARTHR Classic, but it’s not recommended unless you have a good reason. If you do, we’d like to hear about it, so we can see how the new version of ARTHR might meet your needs.
Barring any problems, on 18th March, we’ll stop people being able to make new newspapers in ARTHR Classic. If you’ve already started one before then, it’ll still be editable.
Then, on 1st April, we’ll stop people being able to edit newspapers in ARTHR Classic, and the tool will disappear from the site. On that date your newspaper will be converted to a PDF on our site, and you’ll be able to download the contents as a PDF, InDesign document or a ZIP file containing the stories and pictures.
If you’re working on a paper in ARTHR Classic you’ll need to finish and order it before then. Unfortunately it’s not possible to automatically move your paper from ARTHR Classic to the new version of ARTHR as the structure of the document is significantly different.
If any of that causes you a problem, please let us know and we’ll do our best to help out.
We thought it might be interesting to take you, the dear reader, on a little tour through ARTHR II and how it all works. This post is unashamedly technical, so if all gets a little bit too much, don’t worry: like the Underground at rush hour, there will be another post along shortly.
I will start at the beginning, for the beginning is a good place to start. ARTHR II (henceforth known as ARTHR – the Automated Real-time Text Handling Resource), is our online newspaper layout tool. It provides a drag-and-drop WYSIWYG style interface to make a great looking newspaper quickly and easily. And it’s entirely browser based, with no plugins required, and nothing to download.
It looks a bit like this:
(I know I’ve used this video twice before, but dammit, I’m going to use it again. If you don’t have the time to poke around with ARTHR, it should give you an idea of what you can do with it. If you do have time to have a go, head over here and try it out!)
There are three main components, all of which integrate tightly together.
The backend Ruby on Rails application: handles storage, persistence, integration with the main Newspaper Club site, background jobs, etc.
ERNIE, a Mac OS X/Cocoa rendering server: takes a document layout and converts it into JPG previews or print-ready PDFs
Much of the logic and intelligence in ARTHR runs in the browser, putting it as close to the user as possible to make it as fast as possible.
With so much of the app’s complexity running in the user’s browser, it becomes much harder to track down tricky bugs. To help with this, we wrote our own logging engine, with a server-side component. Browser exceptions are automatically sent over, along with a stracktrace if available, and contextual information like the git revision in use, and the IP address. For persistently tricky browser/OS combinations, we can flip a user’s account into remote debug mode, sending us live log statements, effectively letting us tail a user’s actions and get good diagnostic information.
The backend of ARTHR is quite a slim component, mostly handling persistence, storage of uploaded images, validation checks, and providing a conduit for the frontend to communicate with ERNIE, the rendering engine.
It’s a Rails 3.2 application, running a Postgres database, with Sidekiq for background jobs. We’re making good use of the hstore type to be able to store arbitrary key/value pairs for different content types.
All that sits in front of nginx, running the brilliant nginx-push-stream-module. This lets us handle long-running EventSource/Server-Sent Event connections while still using our standard Unicorn single-threaded Rails server. We just make an HTTP POST request containing the message payload to a specific channel, and it arrives at any clients listening on that channel shortly after. It’s simple, it just works, and keeps the number of server components to a minimum.
When we developed the first version of ARTHR, a few years ago, we used InDesign Server to provide layout, high quality typography with PDF and JPG output. It served us fairly well, but it’s difficult to integrate tightly with, and it’s often very slow. We needed to speed it up by a couple of orders of magnitude, and have much finer information about layout and text flows. It simply wasn’t possible to do that with IDS.
We needed something to let us layout a document, and quickly convert to both JPG (for previews) and PDF. To make things a little more complex, it needs to support CMYK, and integration with a high-quality typography library for things like ligatures, hyphenation and justification.
After working through most of the options, we ended up at Quartz, part of Cocoa’s CoreGraphics stack in OS X. It’s fast – really fast – because it’s designed to run at 60fps on the desktop. Because Quartz maps very closely to the PDF format rendering to a PDF context is fast too. And it natively supports CMYK throughout, something that Skia and Cairo, two of the main open-source rasterisation engines don’t. For typography there’s the Core Text Layout System, a collection of classes for handling typography and typesetting.
Using all this we built ERNIE, the Expertly Rendered Newspaper Internet Engine. It’s an OS X 10.8 application which exposes an HTTP RPC interface, receiving requests for content size calculations, or document rendering, and returning arrays of JPGs or PDF files.
We’re really pleased with the performance. It typically takes ~50ms to render a simple, text heavy document, up to ~700ms for a longer, more graphics heavy document. It’s often 100-200x faster than comparable documents in InDesign Server.
The PDFs Quartz produces have a few oddities, so we post-process them in Ghostscript to improve compatibility with our presses and ensure they’re PDF/X-3:2002 compatible.
Putting it all together
Let’s walk through adding a new bit of text to your document, as an illustration of how this all pieces together.
The user types into the text box and hits save. The frontend immediately sends the new content to the backend for storage. During that request, the backend takes the new content and performs a test rendering against ERNIE, calculating how the text will flow, where the paragraph breaks are, the number of lines per paragraph, etc.
When the request returns, the frontend uses the calculated flow to perform a prediction of where the text will display in the document, and renders the new piece of text in the blueprint. It’s then ready to be moved around the document by the user.
The preview images are now out of date, as they don’t include the piece of text we just added, so the frontend fades them out, and requests a rebuild of them. The backend returns a unique ID for the rebuild request, which the frontend uses to listen for changes in the state of the rebuild, using an EventSource request connected to the nginx-push-stream-module mentioned earlier.
The rebuild request is picked up by one of the background workers which takes the new document structure, including the new text, and passes it through to ERNIE. ERNIE processes the document layout and returns an array of JPG files. These are written to disk by the background worker, and the build is marked as completed. The frontend, listening for changes to the build status, is immediately notified, and requests those JPG previews from the server. As they load, it fades them in and they become visible to the user.
For most users this all happens in less than half a second, end to end. And if the user moves the piece of text around, it all happens again. Of course, the user doesn’t have to wait for the preview images before continuing to edit their document – it’ll keep them up to date as they go.
Hopefully that’s given you an idea of how an application like ARTHR works, all the way through the stack. We’re very happy with the technologies we’ve chosen, and we’re very grateful to everyone who has contributed to all the open source projects we’re building upon.
We’re keen to chat to folks working on similar online publishing tools, and always open to sharing what we’ve learnt. If that’s you: get in touch!