At work with Tom

Tom

Tom spoke with Magculture about Caravan Club bunting and our new print-on-demand Newsagent. It’s up on their blog now if you want to head over and have a read.

Posted by Sarah | Comments Off

Filed under: engineering, news, Newsagent

Newspaper Photo Filters

Before a recent trip to Berlin I wanted to experiment with printing a newspaper to use as photo filters.

all-pages

A single copy of a digital tabloid with a shape, pattern or colour on each page.

DSC_0008

DSC_0078

The pages then work as filters. To do this I expose each photo twice: the first exposure with my camera pointed at the thing I want to photograph and the second exposure pointing at the newspaper. This can be done with any SLR/DSLR or a smartphone (may require an app).

DSC_0041

DSC_0059

The newspaper is ideal for a portable set of filters– light, easy to fold into a pocket. Plus the texture of the newsprint adds to the charm of the overall effect.

DSC_0009

DSC_0018

It’s easier to shoot the subject first and the newspaper second. Anything white on the newspaper page will be obscured in the final image, with detail appearing in the dark areas.

DSC_0290

DSC_0043

I tried to remain experimental with these photos, not anticipating the effect the filter would have on the image and enjoying unexpected results.

DSC_0286

DSC_0117

As an experiment it worked well. On the next paper I’m going to keep the patterns to a minimum and use simpler shapes and colours.

Posted by Ralph | Comments Off

Filed under: case studies, engineering, Newspaper Stories

Long Good Read 004

Long Good Read 004

We popped into Guardian Coffee earlier to take a look at the Long Good Read issue 4, available today. It’s brilliant to see how far it’s come in just a few issues. (Find out more about the project.)

There’s a great selection of articles from all sections of the Guardian: from Doctor Who to Pussy Riot to Proteus.

This time round we’ve tried to explain what’s going on under the hood, with a natty centre spread designed by Ralph (at the top of this post).

Long Good Read 004

For each article we’ve unpacked the reasons for including it in the paper, showing how the algorithm (and sometimes the editor) makes its selection.

Long Good Read 004

Pick up your copy this week, for free, from #guardiancoffee in Shoreditch. We’d love to hear your thoughts.

Long Good Read 004

Posted by Tom | Comments (1)

Filed under: case studies, engineering, Newspaper Stories

New product pages!

productpages

We’re happy to announce that we launched some new product pages yesterday. We hope they’ll make it easier to figure out which type of paper will best suit your project, with a handy tool to calculate how much your newspaper run will cost and when it will be delivered. We’ve also added links to some brilliant examples of each size and style, to show off the many ways our customers have made the most of the formats. Let us know what you think!

Posted by Sarah | Comments Off

Filed under: Announcements, engineering, news

Project Looking Glass

Regular readers will know that Newspaper Club is split between two offices: London and Glasgow. Glasgow is now the HQ, where we do all the operations, logistics, customer service, and much more. And London is where we develop the products and services, writing code, drawing pictures, and so on.

We have a Campfire chat, which we natter in all day long, but for a while I’ve been trying to find other ways of joining the offices together that are less direct and a bit more, well, silly.

Our first go at this is the awfully titled ‘Project Looking Glass’. It’s a two way video screen that sits in the corner of each office, letting us wave at the other office, and them to wave back. It looks a bit like this:

Looking Glass #2

Looking Glass #3

It’s always on, it doesn’t need to dial up or sign in, and if the network drops, it should recover as soon as it can.

It’s built on a pair of Raspberry Pi’s, and you should be able to put one together yourself for under £150, if you can find a couple of spare monitors. I’ve written up much more about it, including how to make one, on my blog.

We’ve only had it running for a couple of days, so it’s a bit of an experiment for us. So far it seems quite fun, with some good drawings appearing on whiteboards. But that might fade, and it might end up being a bit weird, in which case we’ll turn it off and try something else.

But we also know that we’re going to have to get better at remote working, as we grow and as our business get more complicated. It feels like we can learn a lot from trying things like this out while we’re small enough, and those things will be useful as more people join us, in our offices and out.

We’ll keep you posted on how it’s working out for us, and if you end up making something similar for your offices, we’d love to hear how it’s going.

Morning Message from Glasgow

Posted by Tom | Comments Off

Filed under: engineering, running a business

ARTHR updates: No need to sign up, import pictures from Flickr and Instagram, and more!

ARTHR machine

The Engineering Dept have been hard at work on ARTHR (pictured above), and we’ve made a few improvements we’d like to tell you about.

First of all, you no longer need to make an account to get going with ARTHR. Just head over to the site, start making a newspaper, and when you’re ready, sign up and we’ll save your paper into your new account.

We encourage you to make an account as soon as possible after you’ve decided whether ARTHR is for you: not because we’re greedy for your details, but in case you move computers or change browser, and we can’t match you up with your newspaper any more – we don’t want you to lose your paper!

There’s also a lovely new homepage for ARTHR, so tell all your friends.

ARTHR Wizard Screenshot

But that’s not all! We’ve also:

  • Revamped the controls at the top of the screen, based on your feedback, making it clearer how to add and remove pages from your newspaper.
  • Added a feature to be able to add an image to your newspaper directly from a URL. You can even give us a Flickr or Instagram photo page and we’ll do the hard work to fetch the image out of it.
  • Made adding a new piece of content smarter, so it doesn’t overlap anything already on the page, picking the next suitable space in your newspaper.

And finally, a little reminder that the best paper in the Newsagent each month wins a £100 voucher. The Engineering Dept will even throw in a Twix if it’s made in ARTHR.

Posted by Tom | Comments Off

Filed under: engineering

This is my kettle

My Kettle

This is my kettle. There are many like it, but this one is mine. Kettles form a core part of Newspaper Club operations, and it’s important to pick one that has a number of key features to ensure a quality cup of tea. Below I will detail the factors that contribute to this kettle’s daily success, from a systems engineering point of view.

Resilience

Uptime is important with critical pieces of infrastructure. Things that move are things that break. This kettle has few moving parts – no electric heating element, or on/off switch. It’s heated by applying energy (ignited gas) to the base and waiting. This gives me the confidence that I can trust it in daily operations for many decades.

Graceful Degradation

However, if it was to fail, (say the handle fell off), it’ll still be partially usable (with an oven glove) until a repair can be made at a convenient time.

Control Surfaces

It’s not the fastest kettle, but by varying the gas throughput I can vary the time it takes to boil. If I dial it back a bit, it takes just a little less time to boil than my porridge takes to cook, so the tea is brewed just as the porridge is done.

Efficiency

Given a typical UK energy mix, heating things with gas is more efficient than heating things with electricity. However, as more of the UK energy mix (hopefully) shifts to renewable sources over the next few decades, gas will have a relatively larger carbon impact.

Aesthetics

It’s a nice red.

Posted by Tom | Comments (3)

Filed under: engineering

ARTHR is dead, long live ARTHR!

Today we’re very pleased to announce the full release of the new and improved version of our online newspaper making tool, ARTHR. It’s been in testing for a while, but it’s now the default tool if you go into your dashboard and press “Make a Newspaper“. Hurrah!

As part of this, we’re shutting down the old version of ARTHR, now known as ARTHR Classic. This post details how the shutdown will work, and if you’re still using it, important dates to finish your work by.

As of today you can still make a paper in ARTHR Classic, but it’s not recommended unless you have a good reason. If you do, we’d like to hear about it, so we can see how the new version of ARTHR might meet your needs.

Barring any problems, on 18th March, we’ll stop people being able to make new newspapers in ARTHR Classic. If you’ve already started one before then, it’ll still be editable.

Then, on 1st April, we’ll stop people being able to edit newspapers in ARTHR Classic, and the tool will disappear from the site. On that date your newspaper will be converted to a PDF on our site, and you’ll be able to download the contents as a PDF, InDesign document or a ZIP file containing the stories and pictures.

If you’re working on a paper in ARTHR Classic you’ll need to finish and order it before then. Unfortunately it’s not possible to automatically move your paper from ARTHR Classic to the new version of ARTHR as the structure of the document is significantly different.

If any of that causes you a problem, please let us know and we’ll do our best to help out.

Posted by Tom | Comments Off

Filed under: developments, engineering, news

How ARTHR & ERNIE work: Backbone.js, Rails, Cocoa, and more.

We thought it might be interesting to take you, the dear reader, on a little tour through ARTHR II and how it all works. This post is unashamedly technical, so if all gets a little bit too much, don’t worry: like the Underground at rush hour, there will be another post along shortly.

I will start at the beginning, for the beginning is a good place to start. ARTHR II (henceforth known as ARTHR – the Automated Real-time Text Handling Resource), is our online newspaper layout tool. It provides a drag-and-drop WYSIWYG style interface to make a great looking newspaper quickly and easily. And it’s entirely browser based, with no plugins required, and nothing to download.

It looks a bit like this:

(I know I’ve used this video twice before, but dammit, I’m going to use it again. If you don’t have the time to poke around with ARTHR, it should give you an idea of what you can do with it. If you do have time to have a go, head over here and try it out!)

Overview

There are three main components, all of which integrate tightly together.

ARTHR/ERNIE

  • The frontend Javascript application: the entire user interface, some layout intelligence, and more.
  • The backend Ruby on Rails application: handles storage, persistence, integration with the main Newspaper Club site, background jobs, etc.
  • ERNIE, a Mac OS X/Cocoa rendering server: takes a document layout and converts it into JPG previews or print-ready PDFs

The Frontend

Much of the logic and intelligence in ARTHR runs in the browser, putting it as close to the user as possible to make it as fast as possible.

The frontend is mostly a Backbone.js application, written in classical, Crockford-compliant, Javascript. We’ve been very happy with Backbone – it provides just enough structure to support building things quickly, but without forcing you to go ‘all in’ to their way of doing things. There’s plenty of room to pick your own templating language, or drop down to your own AJAX requests, without feeling like you’re going against the grain. It cleanly separates the presentation layer (HTML/CSS) from the logic, leading to good quality, accessible HTML. And last, but most certainly not least, there’s no magic going on: the entire source code is imminently readable, and we’ve regularly walked through it, by eye or in the debugger, to check we understand the behaviour of something.

We’ve tried to write it as you would a well-structured Object Orientated GUI application, and took much from Soundcloud’s blog post about building Next Soundcloud.

With so much of the app’s complexity running in the user’s browser, it becomes much harder to track down tricky bugs. To help with this, we wrote our own logging engine, with a server-side component. Browser exceptions are automatically sent over, along with a stracktrace if available, and contextual information like the git revision in use, and the IP address. For persistently tricky browser/OS combinations, we can flip a user’s account into remote debug mode, sending us live log statements, effectively letting us tail a user’s actions and get good diagnostic information.

The Backend

The backend of ARTHR is quite a slim component, mostly handling persistence, storage of uploaded images, validation checks, and providing a conduit for the frontend to communicate with ERNIE, the rendering engine.

It’s a Rails 3.2 application, running a Postgres database, with Sidekiq for background jobs. We’re making good use of the hstore type to be able to store arbitrary key/value pairs for different content types.

All that sits in front of nginx, running the brilliant nginx-push-stream-module. This lets us handle long-running EventSource/Server-Sent Event connections while still using our standard Unicorn single-threaded Rails server. We just make an HTTP POST request containing the message payload to a specific channel, and it arrives at any clients listening on that channel shortly after. It’s simple, it just works, and keeps the number of server components to a minimum.

ERNIE

When we developed the first version of ARTHR, a few years ago, we used InDesign Server to provide layout, high quality typography with PDF and JPG output. It served us fairly well, but it’s difficult to integrate tightly with, and it’s often very slow. We needed to speed it up by a couple of orders of magnitude, and have much finer information about layout and text flows. It simply wasn’t possible to do that with IDS.

We needed something to let us layout a document, and quickly convert to both JPG (for previews) and PDF. To make things a little more complex, it needs to support CMYK, and integration with a high-quality typography library for things like ligatures, hyphenation and justification.

After working through most of the options, we ended up at Quartz, part of Cocoa’s CoreGraphics stack in OS X. It’s fast – really fast – because it’s designed to run at 60fps on the desktop. Because Quartz maps very closely to the PDF format rendering to a PDF context is fast too. And it natively supports CMYK throughout, something that Skia and Cairo, two of the main open-source rasterisation engines don’t. For typography there’s the Core Text Layout System, a collection of classes for handling typography and typesetting.

Using all this we built ERNIE, the Expertly Rendered Newspaper Internet Engine. It’s an OS X 10.8 application which exposes an HTTP RPC interface, receiving requests for content size calculations, or document rendering, and returning arrays of JPGs or PDF files.

ERNIE logs

We’re really pleased with the performance. It typically takes ~50ms to render a simple, text heavy document, up to ~700ms for a longer, more graphics heavy document. It’s often 100-200x faster than comparable documents in InDesign Server.

The PDFs Quartz produces have a few oddities, so we post-process them in Ghostscript to improve compatibility with our presses and ensure they’re PDF/X-3:2002 compatible.

Putting it all together

Let’s walk through adding a new bit of text to your document, as an illustration of how this all pieces together.

The user types into the text box and hits save. The frontend immediately sends the new content to the backend for storage. During that request, the backend takes the new content and performs a test rendering against ERNIE, calculating how the text will flow, where the paragraph breaks are, the number of lines per paragraph, etc.

When the request returns, the frontend uses the calculated flow to perform a prediction of where the text will display in the document, and renders the new piece of text in the blueprint. It’s then ready to be moved around the document by the user.

The preview images are now out of date, as they don’t include the piece of text we just added, so the frontend fades them out, and requests a rebuild of them. The backend returns a unique ID for the rebuild request, which the frontend uses to listen for changes in the state of the rebuild, using an EventSource request connected to the nginx-push-stream-module mentioned earlier.

The rebuild request is picked up by one of the background workers which takes the new document structure, including the new text, and passes it through to ERNIE. ERNIE processes the document layout and returns an array of JPG files. These are written to disk by the background worker, and the build is marked as completed. The frontend, listening for changes to the build status, is immediately notified, and requests those JPG previews from the server. As they load, it fades them in and they become visible to the user.

For most users this all happens in less than half a second, end to end. And if the user moves the piece of text around, it all happens again. Of course, the user doesn’t have to wait for the preview images before continuing to edit their document – it’ll keep them up to date as they go.

The End

Hopefully that’s given you an idea of how an application like ARTHR works, all the way through the stack. We’re very happy with the technologies we’ve chosen, and we’re very grateful to everyone who has contributed to all the open source projects we’re building upon.

We’re keen to chat to folks working on similar online publishing tools, and always open to sharing what we’ve learnt. If that’s you: get in touch!

Posted by Tom | Comments (3)

Filed under: engineering

ARTHR II: Print Harder

Things are proceeding apace in the twin Newspaper Club offices of London and Glasgow. It’s shaping up to be a busy year already, with a sortie of new products and services being planned and built as I type, right this minute.

We released some improvements to our Dashboard pages yesterday, including a sneaky little link to try out the new version of ARTHR. Go on, click it.

ARTHR screenshot

ARTHR II (internal codename: Print Harder) will soon be our main tool, and we’ll be turning off ARTHR I shortly after. Now’s your opportunity to get ahead of the game and let us know what you think.

New features include:

  • Drag things around with your mouse!
  • Style your text with all sorts of headers and formatting and stuff like that!
  • Choose from a number of typefaces and sizes!
  • Superfast performance using our proprietary Expertly Rendered Newspaper Internet Engine (ERNIE)!
  • Automatically pull a webpage into your paper!
  • And much, much, more!

If you’re making a paper in the old version of ARTHR, don’t worry – we’ll give you plenty of notice before turning it off, and all your work will be available to download in a number of handy formats.

Everyone else: go go go!

Posted by Tom | Comments (1)

Filed under: engineering

Older Posts →