Quartz Things

The Quartz Things Team is Herman Wong, Zach Seward, David Yanofsky, and Nikhil Sonnad. Follow all of our work at http://qz.com/re/quartzthings/

Building our election dashboard in three days and 100 emails

On the eve/morning/WOAH TIME ZONES of Indians going to the polls we published our India election results tracker. With any luck, it looks like we spent loads of time on it—contemplating for weeks or months the functionality and design, then responsibly iterating and testing.

In actuality, we decided to make it and build it in just over three days, collaborating with our colleagues in four different time zones. Miraculously, it worked.

Here’s a paraphrased selection of the more than 100 emails that were sent in that time to make it happen. (The timestamps are in Central European Summer Time.)

Mon, May 12, 2014 at 12:21 PM:
Mitra: We want to make a graphic about the Indian election. Yano how can we do this? We’ll need to update it a lot.

Mon, May 12, 2014 at 7:00 PMish
I call Mitra to discuss

Mon, May 12, 2014 at 10:59 PM 
Me: See the attached sketch would something like this work?

image

Tue, May 13, 2014 at 3:43 PM 
SK: We need something to show the control of power.

Tue, May 13, 2014 at 11:14 PM 
Me: How about something like this?

image

Wed, May 14, 2014 at 2:45 AM
Heather:Looks good, but are we going to have to frantically update data for every party? India has a lot of political parties.

Wed, May 14, 2014 at 5:56 AM 
SK: We only want to show like five or six parties, anyway, don’t worry. Also, what are we going to do if the Election Commission website crashes like it did in 2009?

Wed, May 14, 2014 at 12:29 PM 
Me: I’ll add in a way to drop a message at the top. That way we can let people know what’s up with the data. Also, here’s what it looks like now and how it works.

Desktop:

image

Mobile:

image

Wed, May 14, 2014 at 12:29 PM 
SK: The way you have this working wont work for us logistically

Wed, May 14, 2014 at 4:09 PM 
Me: Okay I’ll try and figure something out. In the meantime can you populate this google doc with real information? Also, here’s the current state of the page:

image

Wed, May 14, 2014 at 4:56 PM
SK: You know that some stuff isn’t working?

Wed, May 14, 2014 at 5:46 PM 
Me: Yes, I know some stuff isn’t working.

Wed, May 14, 2014 at 8:22 PM
Me: Okay everything is working now. But it still doesn’t auto update or have a legend.

image

Wed, May 14, 2014 at 9:26 PM
SK: I think we should get rid of the names of the leaders

Thu, May 15, 2014 at 1:40 AM 
Heather: I think we should get rid of the names of the leaders, too.

Thu, May 15, 2014 at 10:03 AM
Me: I think we should keep the names of the leaders.

Thu, May 15, 2014 at 2:26 PM 
SK: I think we should get rid of the names of the leaders.

Thu, May 15, 2014 at 3:05 PM
Me: I’ll get rid of the names of the leaders.

Thu, May 15, 2014 at 7:53 PM
Me: Here is the almost final page! Here are the docs on how to use it!

image

Thu, May 15, 2014 at 9:55 PM 
Mitra: Can we pub?

Thu, May 15, 2014 at 10:21 PM
Me:DON’T PUB. I need to test more, and Heather needs to edit the text.

Fri, May 16, 2014 at 12:19 AM
Me: Everything is tested, bugs are fixed, and we’re ready to go once Heather edits.

Fri, May 16, 2014 at 12:59 AM 
Heather: Good morning from Hong Kong! We’re edited and ready to go.

Fri, May 16, 2014 at 5:33 AM 
Wordpressbot: Post #209809 was published.

Fri, May 16, 2014 at 10:16 AM
Me: Good morning from Geneva. I added a BJP bar to the chart at the top and numbers next to all of the party names.

image

- David Yanofsky

Chartbuilder’s many friends and the value of open source

It’s been eight months since we open-sourced Chartbuilder, our tool for quickly making good-looking charts. In releasing the project to the public, we hoped it might promote visual journalism and that new contributors would help make Chartbuilder better.

We’re happy to report success toward both goals.

Since we released it to the public, Chartbuilder has been used to make thousands of charts for at least a dozen media organizations, including NPR, the Wall Street Journal, the New Yorker, CNBC, the Press-Enterprise, and New Hampshire Public Radio. Just last week, ESPN’s FiveThirtyEight joined the Chartbuilder club.

image

image

image

image

Isn’t that beautiful? Chartbuilder’s proliferation gives us renewed confidence in our commitment to open-source journalism. We hope it also inspires others to implement the tool and contribute back to the project on GitHub. Chartbuilder needs your help. It still has plenty of bugs and room for improvement.

In the coming weeks, we will be rolling out a new version of Chartbuilder that Quartz reporters have been using. It resolves some of the strange behaviors that have persisted for some time.

If you want to check it out before then, see this branch on GitHub. Or as always, use the current version online here and submit any bugs you come across here.

If you want to join us making tools like this and reporting through code: we’re hiring.

Quartz: Quartz seeks a fall intern to cover technology and science →

quartz:

Quartz is looking for an intern to help us cover technology and science, starting immediately. It’s a paid gig with plenty of opportunities to write. You don’t have to be a current tech journalist, but a familiarity with how we cover the news is essential. The best candidate will be…

From the Hands of David Yanofsky: That was terrifying, exhilarating, and distracting →

yanofsky:

I released Chartbuilder last week not knowing what to expect. I had thrown up some gists in the past, but I’d never open-sourced an ongoing project.

I had a fear was that it would be shrugged off with a whimper of notice. I was worried that people who I admired in the industry would…

We’ve just open-sourced Chartbuilder, the tool that all reporters use at Quartz to quickly make simple charts at graphics-desk quality. Read more about how Chartbuilder came to be and how we use it in David Yanofsky’s piece for the Nieman Journalism Lab.

We’ve just open-sourced Chartbuilder, the tool that all reporters use at Quartz to quickly make simple charts at graphics-desk quality. Read more about how Chartbuilder came to be and how we use it in David Yanofsky’s piece for the Nieman Journalism Lab.

How we found the center of international air travel

While I was making “29 of the world’s largest bike-sharing programs in one map,” my colleague Ritchie sent along a New Yorker article from March about Neil Freeman, the man who created “Subway systems at the same scale" that inspired the bike map.

In the article, Freeman brought the New Yorker’s Jason Fagone to the geographic center of New York City’s road intersections.

It was a unfamiliar concept to grasp, but Fagone described it like this:

if the city was a giant cookie tray and you put a tiny weight on every intersection, the tray would balance on this point.

We like writing about air travel at Quartz, so I thought I’d try creating a similar map, at a global scale, with airports. Then, to make the results more meaningful than “this is the center-most airport,” I decided to weight every airport by the number of routes that came out of it.

When calculated for the entire world, it would show the average midpoint of every airline route. When calculated for a country, it could show the average destination.

I wasn’t entirely sure of the proper math to do this, so I just tried to replicate the cookie tray analogy: For every country, take the destinations of every departing route and average the latitudes and longitudes of every destination.

This was the result

image

All the calculated center’s of gravity were clustered in the Atlantic Ocean: clearly wrong.

The limits of the cookie sheet method was revealed. Cookie sheets are flat and have a cartesian coordinate system, but planets are round and have a polar coordinate system.

Before submitting myself to the mind-bending geometry that is three-dimensional trigonometry, I searched the internet to see if someone else has tried to accomplish the same. 

I found this question on the GIS Sack Exchange—”Computing an averaged latitude and longitude coordinates"—and converted the code to Python, which is what I was using to parse and combine the three databases from OpenFlights.

import math
import numpy

def toCartesian(t):
        """Convert a tuple of lat,long to x,y,z"""
        latD,longD = t
        latR = math.radians(latD)
        longR = math.radians(longD)
        return (
            math.cos(latR)*math.cos(longR), 
            math.cos(latR)*math.sin(longR), 
            math.sin(latR)
            )

def toSpherical(t):
    """Convert a tuple of x,y,z to lat,long"""
    x,y,z = t

    r = math.hypot(x,y)
    if r == 0:
        if z > 0:
            return (90,0)
        elif z< 0:
            return (-90,0)
        else:
            return None
    else:
        return (
                math.degrees(math.atan2(z, r)), 
                math.degrees(math.atan2(y,x))
                3)

#use numpy to allow for easy vector and matrix math
xyz = numpy.asarray([0.0,0.0,0.0]) 
total = 0
for p in points:
        weight = p["weight"]
        total += weight
        xyz += numpy.asarray(toCartesian((p["lat"],p["long"])))*weight

avgXYZ = xyz/total
avgLat, avgLong = toSpherical(avgXYZ)
print avgLat,avgLong

New code written, I tested out another map:

image

That pink dot is supposed to be the center of gravity for the flights from Saudi Arabia shown… It’s not.

Looking for some help, I posted my code to the GIS Stack Exchange, tweeted it, and retweeted it. 

Turns out I had a bug in my code (not shown above), but when I went and tried the data again, I still had issues.

A math nerd friend chatted me. We talked it over. He asked if I had tested it. Turns out I hadn’t…I’m an idiot.

I ran some tests on the averaging code I wrote and made some test maps. (The pink dot is the calculated center)

image

image

image

Turns out that was all working, it was my input data that was wrong. So I rewrote my parser and tried again:

image

It worked.

If you’re wondering, I was using Tilemill to make these maps from the CSV files my Python script exported. To create the flight paths and lines connecting the center of gravities to the country I put a geojson blob in the CSV.

That method allowed me to make a map of all of the international routes:

image

Unfortunately, Tilemill draws the lines connecting points after the map data is projected. This means that the paths shown are not actually the shortest distance between the two points.

The only way to accomplish this was to write some more code that would convert two endpoints into a series of points along the great circle that runs through it. I found the formula here, converted it to Python and created a way to loop through it.

There were typos and bugs:

image

image

But I figured out the code eventually:

def greatCircPoint(start,end,d,f):
#from http://williams.best.vwh.net/avform.htm#Intermediate lat1 = math.radians(start[0]) lon1 = math.radians(start[1]) lat2 = math.radians(end[0]) lon2 = math.radians(end[1]) A = math.sin((1-f)*d)/math.sin(d) B = math.sin(f*d)/math.sin(d) x = A * math.cos(lat1) * math.cos(lon1) + B * math.cos(lat2) * math.cos(lon2) y = A * math.cos(lat1) * math.sin(lon1) + B * math.cos(lat2) * math.sin(lon2) z = A * math.sin(lat1) + B * math.sin(lat2) return toSpherical((x,y,z)) def greatCirclePoints(start,end): # 3963 is the radius of the earth num_points = float(round(greatCircDistance(start[0],start[1],end[0],end[1])/(3963/2)*15)) if num_points < 4: #make sure there are at least 4 points on the path num_points = 4 pnts = [0]*int(num_points+1) dist = arcD(start[0],start[1],end[0],end[1]) for i in range(int(num_points+1)): pt = greatCircPoint(start,end,dist,(i/num_points)) pnts[i] = pt return pnts

The exported maps from Tilemill looked something like this:

image

image

and after the final touches in Illustrator:

image

image

See the rest of the maps and read “The average destination of international flights from every country in the world" and "The center of the international airline industry is in the middle of rural Poland" on Quartz.

Get your hands on the same flight data we used from Openflights.

–David Yanofsky

Quartz Things B-roll vol. 3

See the slides that made the cut here: A brief history of the US government’s awful graphic design

Plotting some dots

Like every news organization and their sisters this week, Quartz felt compelled to acknowledge the (delayed) launch of the New York City bike sharing program.

As is always the case here at Quartz we try to keep our content globally relevant. While the New York system is large, it’s really late to implementing this type of policy compared to other cities around the world. There are already more than 500 municipalities with similar systems.

To say the least, I was skeptical that we should be covering this story.

But then like a super hero in comes Roberto with a instant message, “here’s a website with all of the information you’d ever want about a whole bunch of bike sharing systems http://bikes.oobrien.com/

The site had geolocation data for 85 cities.

I saw it and immediately knew what I was going to do [1], then sketched something highly detailed to sell everyone on it: 

image

The top was what I drew then, the bottom came later. I tried to find links to the subway maps I had seen displayed in similar ways, but I couldn’t find them at the time. “Trust me, It’ll be great,” I said.

(I found those links later, here, and here)

I collected the geo data from bikes.oobrien.com using a python script, opened it up in tilemill and exported a really big pdf map that looked like this: 

image

An hour of Illustrator selecting grouping and resizing later and we had this:

image

Then some more fiddling, versioning, and alt-dragging:

image

some copy writing, and editing, and hand wringing over a Headline and kicker later, we published (sans kicker).

image

Read the story on Quartz: 29 of the world’s largest bike-sharing programs in one map

[1] It turns out that Oliver O’brien had made a similar map last year 

The best things can’t be planned. The “sleeps” post began life as something Leo wrote, but making a different argument. David and Ritchie were separately working on an analysis of the tweets, but didn’t really know what to say about them. I suggested pairing the two. Zach saw a way to do that by giving the post a new spin.

— Gideon Lichfield regarding “We know when Dzhokhar Tsarnaev sleeps

Quartz Things B-roll Vol. 2

Everything you ever wanted to know about baseball ticket prices