tag:thingsivelearned.posthaven.com,2013:/posts Things I've Learned 2016-04-03T01:45:04Z John Keefe tag:thingsivelearned.posthaven.com,2013:Post/481417 2011-08-02T17:32:00Z 2013-10-08T17:04:32Z Mapping Restaurant Grades

Today I took up a quick challenge to see if I could map NYC's restaurant grades by zip code, and whether it would be interesting.

I gave myself 50 minutes, and it took about 60. Answers: 1) Yes and 2) doesn't seem to be.

Great analysis and other maps I really like on this front are already out there from Dan Nguyen and Steven Romalewski.

For me, it was more a data-map-time challenge to myself.

Below are my quick proof-of-concept (or no proof!) maps. There aren't any legends on them yet, but the color gradients indicate the number of restaurants getting that grade as a percentage of all the restaurants in the zip code (0-2-4-6-8-10%+).

Grade A map

Grade B map

Grade C map

See any patterns? I don't ... yet.

But I did learn how to take a whole bunch of records and run a couple of aggregations and merges in Fusion Tables to actually visualize them. In about an hour.

UPDATE: Because some folks have asked: The sum includes all restaruants in the zip code, including those with "Pending" grades and those that haven't been inspected, which is the vast majority in many zip codes.

UPDATE2: Pulled together key parts of the twexchange using Storify, below.

UPDATE3: Steven Romalewski's look at a Wall Street Journal story let me to realize that even though only a small fraction of the restaurants in the data set have letter grades, almost all of them have a "score" -- which means I ought to re-analyze based on that column instead (which I'll do tomorrow).


View "The Restaurant Grade Challenge & Chat" on Storify

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481420 2011-06-22T17:45:00Z 2013-10-08T17:04:32Z Playing with SeeClickFix

I found out yesterday that every report collected by SeeClickFix -- from pothole to trash pile -- is openly available via their widgets and an API.

That is pretty cool. It means that anyone who contributes online, whith their phone or via a SeeClickFix interface, such as the embedded one below, adds to a communal set of civic issues.

And if the 311 folks are watching and responding -- like they are in San Francisco -- then it actually can result in civic, or at least municipal, action.

So when will New York City move to a more open 311 reporting system?

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481428 2011-05-14T03:27:00Z 2016-04-03T01:45:04Z Allowing Cross-Domain Access in Sinatra

I've been trying to make a little Sinatra app that accepts a "GET" request and sends back some JSON. But I kept getting a version of this error in my JavaScript console:

Origin http://domain.com is not allowed by Access-Control-Allow-Origin

Eventually I learned this was because I was making AJAX requests from one domain to another, which is generally not allowed for very good security reasons. That is, unless you explicitly allow it on the server side.

Naturally, Sinatra provides a ridiculously easy way to do this, which is explained extremely well by Functional Objects. Just add this line in the route somewhere before the result is sent to the requester:

response['Access-Control-Allow-Origin'] = 'http://domain.com'

To grant access to everyone, it's

response['Access-Control-Allow-Origin'] = '*'

I am so glad to see those errors disappear.

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481434 2011-03-29T03:59:00Z 2013-10-08T17:04:32Z Turning PDF Charts Into Usable Data

Someone once opined that nothing set back information exchange more than the fax machine.

For data, I'd rank PDFs as a close contender.

Tonight I became convinced that I needed to begin pulling City of New York numbers out of a PDF chart by hand (I'll share the chart once the story comes out), or plunk down $100 for some conversion software.

Then, through this blog post, I discovered this online conversion service from Nitro. For free. 

It took a few minutes for it to convert ... in fact I thought it didn't work. But then, it appeared in my email box: A perfect Excel file, correctly formatted, funky columns and all. Whew.

Now if NYC.gov would post this info in a more data-friendly format, that'd be really fantastic.

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481437 2011-02-09T22:12:00Z 2013-10-08T17:04:32Z Custom Maps a la Mapping L.A.

I'm pretty thrilled about this. I've managed to quickly whip together a Google map of NYC public schools using style changes, custom icons and data from the NYC DataMine. I'll share how I did it in a longer post soonest.

Full admiration of and inspriation from the Mapping L.A. folks, and their great neighborhood maps.

This table uses iFrame, which isn't supported on your browser. </iframe

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481442 2011-01-30T04:52:00Z 2013-10-08T17:04:32Z Drawing Data with Processing

Dipping my toes into making drawings from data.

I've started into Jer Thorp's great introduction, in which he walks you through pulling a column of numbers from a Google spreadsheet into some neat drawings. In this case, the data are a set of numbers he asked of his Twitter followers. Below, I've plotted the frequency of numbers from 0 to 100 as a bar chart.

Numbers, from 0-100, picked out of the air by @blprnt's Twitter followers. 17, 37 and 42 were most common.

Jer even provides a Processing sketch you can download to grab numbers from simple Google spreadsheets. Nice.

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481444 2011-01-10T02:37:00Z 2013-10-08T17:04:32Z Painting With Grass

"I'm painting with grass!" said my 5-year-old when she tried the app below.


"Can I make it red?"

I'll get right on that!

Click and drag in the box to draw. Then type r for red, b for blue, g for green. c clears it.

Update: p is for purple, as requested.

[The content that should be here uses "iframe," which isn't supported either by your browser or this server.]

The Processing code for this is here.

Originally written during Jer Thorp's Processing workshop.

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481449 2011-01-09T19:36:00Z 2013-10-08T17:04:32Z Drawing Using Processing

Yesterday I took a Processing workshop from the brilliant and talented Jer Thorp. I learned a ton about programming in general, programming in Processing, coding visuals and the concept of Perlin landscapes.

Today, I figured out how to put my Processing experiments online using Processing.js. A sample is below (requires a newish, HTML5-capable browser). 

More to come!

This table uses iFrame, which isn't supported on your browser.

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481452 2011-01-06T05:08:00Z 2013-10-08T17:04:32Z Gem Info Right Under My Nose

This is probably the most useful thing I learned today. 

As a ruby newbie, I'm always looking for documentation for stuff, including information on the "gems" -- or little extra programs others have built -- I've installed on my computer. I didn't realize there was a trove of that info actually sitting locally on my computer.

Thanks to a post by Corey Ehmke, I now know how to see it:

You can see what gems are installed on your local system using gem list in Terminal, but did you know that there's an even better way?

Try this instead: gem server

This starts a WEBrick instance running. Fire up the browser of your choice and navigate to localhost:8808. You'll see a page that not only shows a list of the gems that are installed, but also provides quick and easy access to each gem's RDoc [documentation] and home page.

Very cool. Extremely useful.

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481454 2011-01-04T03:59:00Z 2013-10-08T17:04:32Z Floating Point Results in Ruby

Sometimes it's the little things.

Today I couldn't figure out the syntax that would tell Ruby to give me a fraction result -- that is, a floating-point answer instead of an integer. 

> 5/3

 => 1 

The rule is that one of the numbers has to be floating point to get a floating point result. And if none is, you make one so! Adding .to_f (to float) makes the 3 into 3.0000000 ... and we're set.

 > 5/3.to_f

 => 1.66666666666667 

Thanks to Melvin Ram for being on the other end of my Google search.

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481458 2010-12-17T21:42:00Z 2013-10-08T17:04:33Z Making Google Maps from Flickr Sets

Found a great little tool that does exactly what I was about to code. 

Allows you to build a Google map from a geotagged set of pictures, like this one of mine taken in the "Frogtown" section of St. Paul. It works via the Flickr API. Slick.

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481460 2010-11-30T05:57:00Z 2013-10-08T17:04:33Z More Huge Taxi Data

Just did a heavy-duty Mongo database query to generate 164,562 taxi trips from February through July 2010. All were from 4 a.m. to 6 a.m. and were at least 10 miles long. I then uploaded the data to Google Fusion Tables. 

Full map is here.

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481462 2010-11-17T23:23:00Z 2013-10-08T17:04:33Z Making Clean, Updatable Online Tables

I tried a cool tool from the folks at ProPublica called TableSetter, which makes really nice, dynamic tables from .csv (comma-separated-values) files or Google Docs spreadsheets:



Nice thing is that if the data changes, you just update the .csv file without having to remake the page. I managed to install it on my computer, make it work, deploy it on Heroku and make it work there, too.

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481463 2010-11-15T06:07:00Z 2013-10-08T17:04:33Z How to Wrangle Huge Amounts of Data

I used MongoDB, Ruby and Google Fusion tables to make a map of long, early-morning taxi trips.

It shows the start points of 5533 NYC taxi trips that were at least 10 miles long and began between 4 a.m. and 6 a.m. one week in March 2009. Data from the NYC Taxi & Limousine Commission.

I imported the TLC's data into MongoDB using MongoImport and then got the Mongo Ruby Gem. Using these three pages, I ran a bunch of "finds" on the data to see what I could get, just using "puts" to print them to the screen.

Even figured out some regular expressions to pull the only records where the hour was 04 or 05, and also to get rid of the commas and extra spaces in the address fields.

Here's the code for where I ended up.

Instead of writing to a file, I just copy-pasted the output from my terminal as a .csv file and uploaded it to Google Fusion Tables.

And here's a bigger version of the map.

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481464 2010-11-12T03:07:34Z 2013-10-08T17:04:33Z The Basics of MongoDB

I'm going to try to use MongoDB and Sinatra together in a project. But first, gotta learn about and install MongoDB. First lesson I found here.

Introduction to MongoDB - Part I from Charles Max Wood on Vimeo.

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481465 2010-10-14T02:33:00Z 2013-10-08T17:04:33Z A Cool Tool for Coloring Maps

Heard about this cool tool -- Colorbrewer -- from a fantastic tutorial on Flowingdata about How to Make a US County Themed Map using free tools. Even tho I'm working in Ruby instead of Python, it helped a ton.

John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481466 2010-10-07T01:00:00Z 2013-10-08T17:04:33Z I can make maps with text

I learned from a Flowing Data article about making themed maps that I can download .svg files from Wikimedia Commons and that that those are just text files you can tinker with ... like shade the counties according to data by changing the fill text!  And when you're done, you can embed them into HTML like this: for browsers that support 'em!

Here's what it looks like:

Using embed code like this:

<object width="640" height="480" type="image/svg+xml" data="http://www.prototypecloud.com/linked/Map_of_New_York_County_Outlines.svg">




John Keefe
tag:thingsivelearned.posthaven.com,2013:Post/481467 2010-09-30T04:03:00Z 2013-10-08T17:04:33Z Mining Nested Hashes in Sinatra

I've been tinkering with Sinatra (a framework on Ruby) lately, but was having trouble figuring out how to get information out of a huge nested hash. So I tweeted the below to my accidental mentor, Al Shaw, of Talking Points Memo:

@A_L Say @foo={"people"=>{ "nabe"=>"dumbo", "kids"=>[{"name"=>"ann", "age"=>"8"}, {"name"=>"joe", "age"=>"10"}]} How do I return "joe"?

He replied that my example was missing a } (yup) and that the answer was:

@foo = {"people"=>{"nabe"=>"dumbo", "kids"=>[{"name"=>"ann", "age"=>"8"}, {"name"=>"joe", "age"=>"10"}]}}

["people"]["kids"][1]["name"]  #=>  "joe"

Now I get it!

John Keefe