Here is the data released today by the Met Office:
http://www.metoffice.gov.uk/climatechange/science/monitoring/subsets.html
#climatedata
Data Visualization & Climate Reality
-
blprnt
Key MasterPosted 2 years ago # -
tim
Memberpossibly easier to access subsets/copies are available
here
http://timetric.com/tags/HadCRUT3/and here
http://www.guardian.co.uk/environment/datablog/2009/dec/08/uk-us-temperature-change-global-met#dataPosted 2 years ago # -
blprnt
Key MasterThanks, Tim.
I'm teaching right now but I'll have a closer look at those links when I get back home. The Guardian Data store typically releases data in Google Spreadsheets format which would be a nice accessible place for people to work from, but in this case it looks like they are in CSV format.
Posted 2 years ago # -
Thanks for the additional links.
I had a look into the guardian CSV files. Unfortunately the US file is in 'invalid' CSV-format, as the Station Name contains unescaped commas, e.g. KEY WEST, FLORIDA - beware when parsing..
Posted 2 years ago # -
Hey Folks, for kicks I wrote a perl script to parse the data and turn it into a MySQL db. Here's a dump of that database:
http://muffinlabs.com/sites/default/files/temp-readings-120909.sql_.gz
Hopefully a smarter person than myself can take it and run with it.
Thanks,
ColinPosted 2 years ago # -
Oops, should've noticed how small that file was. The readings data wasn't there. It should be now.
Colin
Posted 2 years ago # -
Also, the original script I used to generate the SQL is here:
http://muffinlabs.com/content/global-temperature-readings-mysql-form
It's in perl, and reasonably basic.
Enjoy!
Posted 2 years ago # -
blprnt
Key MasterThanks, Colin!
I am getting my DB set up as we speak. Hopefully I can do some simple data exploration over the next couple of days. I'll post some notes here as I get into things a bit further.
-Jer
Posted 2 years ago # -
blprnt
Key MasterOK - I get the opposite problem that cr23 had: I only get the readings table. Strange.
Any suggestions?
Posted 2 years ago # -
the dump works for me. I got readings and stations without doing any special.
I got 1.670.364 readings for 1.729 stationsThanks Colin!
Posted 2 years ago # -
Yeah I'm fairly certain both tables are in there. The way the script works, the inserts into the stations table are interspersed throughout the script. Are you seeing any errors?
Posted 2 years ago # -
blprnt
Key MasterSomething strange is definitely going on. I'm importing the .SQL file from phpMyAdmin - it takes a while and eventually gives me just the readings table, with 142,240 records. Maybe it's a memory issue and the whole file isn't importing correctly...
Posted 2 years ago # -
It seems the phpMyAdmin aborts the process. The INSERT statements for the station are behind the readings in the sql dump.
Try something like
mysql -u root -p climatedata < ~/Downloads/temp-readings-120909.sql
where 'climatedata' is the name of your database.Posted 2 years ago # -
blprnt
Key MasterAwesome. Worked like a charm. 1,670,364 readings from 1,729 stations.
Now, the tricky part!!
Posted 2 years ago # -
Yeah, I'm really not sure what to with this data haha. But how are there readings from the year 1701?
Posted 2 years ago # -
There is a reasonable amount of junk in the data. There's a fair number of -99 temperature readings, misspellings, etc. It'll take some work to clean it out.
Posted 2 years ago # -
Thanks so much for the data. Just a quick note on the readings table. If you add an index on theYear, theMonth then the lookups in the code to get the readings for a given year/month will be much much faster.
The following sql will do it.
create index date_index on readings(theYear, theMonth);
Posted 2 years ago # -
blprnt
Key MasterYes - I've noticed the -99 temperature readings, too! I sure hope those weren't included in the 'real' released analysis!
Thanks to Flinklabs for the index tip!
Posted 2 years ago # -
Hmm I can't get any of the lat / lng values to plot correctly on Google maps. I'm pretty sure Google maps wants decimal values, which is what they appear to be in the SQL. Even C+P the values into http://www.gorissen.info/Pierre/maps/googleMapLocationv3.php
Any ideas?
Posted 2 years ago # -
@muffinista on the met site they say "A value of -99 indicates missing data."
@staydecent change the sign of the lon value to work correctly with google maps. I'm not sure why, but that worked for me.
Posted 2 years ago #
Reply
You must log in to post.