Here is the data released today by the Met Office:
http://www.metoffice.gov.uk/climatechange/science/monitoring/subsets.html
Here is the data released today by the Met Office:
http://www.metoffice.gov.uk/climatechange/science/monitoring/subsets.html
possibly easier to access subsets/copies are available
here
http://timetric.com/tags/HadCRUT3/
and here
http://www.guardian.co.uk/environment/datablog/2009/dec/08/uk-us-temperature-change-global-met#data
Thanks, Tim.
I'm teaching right now but I'll have a closer look at those links when I get back home. The Guardian Data store typically releases data in Google Spreadsheets format which would be a nice accessible place for people to work from, but in this case it looks like they are in CSV format.
Thanks for the additional links.
I had a look into the guardian CSV files. Unfortunately the US file is in 'invalid' CSV-format, as the Station Name contains unescaped commas, e.g. KEY WEST, FLORIDA - beware when parsing..
Hey Folks, for kicks I wrote a perl script to parse the data and turn it into a MySQL db. Here's a dump of that database:
http://muffinlabs.com/sites/default/files/temp-readings-120909.sql_.gz
Hopefully a smarter person than myself can take it and run with it.
Thanks,
Colin
Hey Colin - thanks for that database dump. It ran ok and I got the tables created and the stations inserted, but the readings data is missing and I couldnt find it in the SQL file. As I'm not a frequent mysql user I could totally be missing something?
Oops, should've noticed how small that file was. The readings data wasn't there. It should be now.
Colin
Also, the original script I used to generate the SQL is here:
http://muffinlabs.com/content/global-temperature-readings-mysql-form
It's in perl, and reasonably basic.
Enjoy!
Thanks, Colin!
I am getting my DB set up as we speak. Hopefully I can do some simple data exploration over the next couple of days. I'll post some notes here as I get into things a bit further.
-Jer
OK - I get the opposite problem that cr23 had: I only get the readings table. Strange.
Any suggestions?
the dump works for me. I got readings and stations without doing any special.
I got 1.670.364 readings for 1.729 stations
Thanks Colin!
Yeah I'm fairly certain both tables are in there. The way the script works, the inserts into the stations table are interspersed throughout the script. Are you seeing any errors?
Something strange is definitely going on. I'm importing the .SQL file from phpMyAdmin - it takes a while and eventually gives me just the readings table, with 142,240 records. Maybe it's a memory issue and the whole file isn't importing correctly...
It seems the phpMyAdmin aborts the process. The INSERT statements for the station are behind the readings in the sql dump.
Try something like
mysql -u root -p climatedata < ~/Downloads/temp-readings-120909.sql
where 'climatedata' is the name of your database.
Awesome. Worked like a charm. 1,670,364 readings from 1,729 stations.
Now, the tricky part!!
Yeah, I'm really not sure what to with this data haha. But how are there readings from the year 1701?
let the fun begin!
just realized i had to clear my cache / cookies to actually download the 9mb file instead of the original! duh.
There is a reasonable amount of junk in the data. There's a fair number of -99 temperature readings, misspellings, etc. It'll take some work to clean it out.
Thanks so much for the data. Just a quick note on the readings table. If you add an index on theYear, theMonth then the lookups in the code to get the readings for a given year/month will be much much faster.
The following sql will do it.
create index date_index on readings(theYear, theMonth);
Yes - I've noticed the -99 temperature readings, too! I sure hope those weren't included in the 'real' released analysis!
Thanks to Flinklabs for the index tip!
Hmm I can't get any of the lat / lng values to plot correctly on Google maps. I'm pretty sure Google maps wants decimal values, which is what they appear to be in the SQL. Even C+P the values into http://www.gorissen.info/Pierre/maps/googleMapLocationv3.php
Any ideas?
Nice work muffinista. I'm in the process of cleaning and indexing the data. Will post a mysql-dump when I'm finished.
@muffinista on the met site they say "A value of -99 indicates missing data."
@staydecent change the sign of the lon value to work correctly with google maps. I'm not sure why, but that worked for me.
You must log in to post.