#climatedata » Tag: exploration

#climatedata » Tag: exploration - Recent Posts http://climatedata.blprnt.com/tags/exploration Data Visualization & Climate Reality en-US Tue, 07 Feb 2012 18:01:16 +0000 http://bbpress.org/?v=1.0.2 <![CDATA[Search]]> q http://climatedata.blprnt.com/search.php rockitbaby on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-39 Sat, 12 Dec 2009 16:46:47 +0000 rockitbaby 39@http://climatedata.blprnt.com/ @muffinista on the met site they say "A value of -99 indicates missing data." @staydecent change the sign of the lon value to work correctly with google maps. I'm not sure why, but that worked for me. jowza on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-35 Fri, 11 Dec 2009 22:21:51 +0000 jowza 35@http://climatedata.blprnt.com/ Nice work muffinista. I'm in the process of cleaning and indexing the data. Will post a mysql-dump when I'm finished. staydecent on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-34 Fri, 11 Dec 2009 21:42:27 +0000 staydecent 34@http://climatedata.blprnt.com/ Hmm I can't get any of the lat / lng values to plot correctly on Google maps. I'm pretty sure Google maps wants decimal values, which is what they appear to be in the SQL. Even C+P the values into <a href="http://www.gorissen.info/Pierre/maps/googleMapLocationv3.php" rel="nofollow">http://www.gorissen.info/Pierre/maps/googleMapLocationv3.php</a> Any ideas? Flinklabs on "Data Exploration - Monthly Sets" http://climatedata.blprnt.com/topic/data-exploration-monthly-sets#post-27 Fri, 11 Dec 2009 08:48:12 +0000 Flinklabs 27@http://climatedata.blprnt.com/ So I've been thinking and wondering if rather than using the monthly values which ebb and flow the seasons we could use the average from the year but better than that it might be better to use the max for the year to show the growth in the max temp or maybe a band of min and max rather than avg...what do you think? Maybe you could take your ones and show them for each continent and use the min max every year? blprnt on "Data Exploration - Monthly Sets" http://climatedata.blprnt.com/topic/data-exploration-monthly-sets#post-26 Fri, 11 Dec 2009 08:21:49 +0000 blprnt 26@http://climatedata.blprnt.com/ <a href="http://www.flickr.com/photos/blprnt/4176309538/" title="Met Office Climate Data - Month by Month (January) by blprnt_van, on Flickr"><img src="http://farm5.static.flickr.com/4037/4176309538_96cb0f59f3.jpg" alt="Met Office Climate Data - Month by Month (January)" /></a> I spent a few hours with the data this evening. Mainly the purpose of these images is to get an idea of the scope of the data. I thought it would be helpful to break the records down into monthly sets. Also I sorted the stations by latitude, which groups like measurements together. I want to do some more exports like this focusing on certain geographical areas (N. Africa, for instance) to see if I can identify some trends... <a href="http://www.flickr.com/photos/blprnt/sets/72157622978635070/" rel="nofollow">http://www.flickr.com/photos/blprnt/sets/72157622978635070/</a> blprnt on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-24 Fri, 11 Dec 2009 02:31:41 +0000 blprnt 24@http://climatedata.blprnt.com/ Yes - I've noticed the -99 temperature readings, too! I sure hope those weren't included in the 'real' released analysis! Thanks to Flinklabs for the index tip! Flinklabs on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-23 Fri, 11 Dec 2009 01:53:55 +0000 Flinklabs 23@http://climatedata.blprnt.com/ Thanks so much for the data. Just a quick note on the readings table. If you add an index on theYear, theMonth then the lookups in the code to get the readings for a given year/month will be much much faster. The following sql will do it. <code> create index date_index on readings(theYear, theMonth); </code> muffinista on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-21 Thu, 10 Dec 2009 20:09:55 +0000 muffinista 21@http://climatedata.blprnt.com/ There is a reasonable amount of junk in the data. There's a fair number of -99 temperature readings, misspellings, etc. It'll take some work to clean it out. cr23 on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-20 Thu, 10 Dec 2009 20:05:44 +0000 cr23 20@http://climatedata.blprnt.com/ let the fun begin! just realized i had to clear my cache / cookies to actually download the 9mb file instead of the original! duh. staydecent on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-18 Thu, 10 Dec 2009 19:30:42 +0000 staydecent 18@http://climatedata.blprnt.com/ Yeah, I'm really not sure what to with this data haha. But how are there readings from the year 1701? blprnt on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-17 Thu, 10 Dec 2009 19:11:03 +0000 blprnt 17@http://climatedata.blprnt.com/ Awesome. Worked like a charm. 1,670,364 readings from 1,729 stations. Now, the tricky part!! rockitbaby on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-16 Thu, 10 Dec 2009 18:07:31 +0000 rockitbaby 16@http://climatedata.blprnt.com/ It seems the phpMyAdmin aborts the process. The INSERT statements for the station are behind the readings in the sql dump. Try something like <code>mysql -u root -p climatedata < ~/Downloads/temp-readings-120909.sql</code> where 'climatedata' is the name of your database. blprnt on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-15 Thu, 10 Dec 2009 18:05:10 +0000 blprnt 15@http://climatedata.blprnt.com/ Something strange is definitely going on. I'm importing the .SQL file from phpMyAdmin - it takes a while and eventually gives me just the readings table, with 142,240 records. Maybe it's a memory issue and the whole file isn't importing correctly... muffinista on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-14 Thu, 10 Dec 2009 13:52:28 +0000 muffinista 14@http://climatedata.blprnt.com/ Yeah I'm fairly certain both tables are in there. The way the script works, the inserts into the stations table are interspersed throughout the script. Are you seeing any errors? rockitbaby on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-13 Thu, 10 Dec 2009 13:19:30 +0000 rockitbaby 13@http://climatedata.blprnt.com/ the dump works for me. I got readings and stations without doing any special. I got 1.670.364 readings for 1.729 stations Thanks Colin! blprnt on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-12 Thu, 10 Dec 2009 06:33:16 +0000 blprnt 12@http://climatedata.blprnt.com/ OK - I get the opposite problem that cr23 had: I only get the readings table. Strange. Any suggestions? blprnt on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-11 Thu, 10 Dec 2009 06:17:03 +0000 blprnt 11@http://climatedata.blprnt.com/ Thanks, Colin! I am getting my DB set up as we speak. Hopefully I can do some simple data exploration over the next couple of days. I'll post some notes here as I get into things a bit further. -Jer muffinista on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-10 Wed, 09 Dec 2009 21:16:52 +0000 muffinista 10@http://climatedata.blprnt.com/ Also, the original script I used to generate the SQL is here: <a href="http://muffinlabs.com/content/global-temperature-readings-mysql-form" rel="nofollow">http://muffinlabs.com/content/global-temperature-readings-mysql-form</a> It's in perl, and reasonably basic. Enjoy! muffinista on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-9 Wed, 09 Dec 2009 20:58:28 +0000 muffinista 9@http://climatedata.blprnt.com/ Oops, should've noticed how small that file was. The readings data wasn't there. It should be now. Colin cr23 on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-8 Wed, 09 Dec 2009 20:15:08 +0000 cr23 8@http://climatedata.blprnt.com/ Hey Colin - thanks for that database dump. It ran ok and I got the tables created and the stations inserted, but the readings data is missing and I couldnt find it in the SQL file. As I'm not a frequent mysql user I could totally be missing something? muffinista on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-7 Wed, 09 Dec 2009 19:36:21 +0000 muffinista 7@http://climatedata.blprnt.com/ Hey Folks, for kicks I wrote a perl script to parse the data and turn it into a MySQL db. Here's a dump of that database: <a href="http://muffinlabs.com/sites/default/files/temp-readings-120909.sql_.gz" rel="nofollow">http://muffinlabs.com/sites/default/files/temp-readings-120909.sql_.gz</a> Hopefully a smarter person than myself can take it and run with it. Thanks, Colin rockitbaby on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-6 Wed, 09 Dec 2009 19:29:50 +0000 rockitbaby 6@http://climatedata.blprnt.com/ Thanks for the additional links. I had a look into the guardian CSV files. Unfortunately the US file is in 'invalid' CSV-format, as the Station Name contains unescaped commas, e.g. KEY WEST, FLORIDA - beware when parsing.. blprnt on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-4 Wed, 09 Dec 2009 18:55:16 +0000 blprnt 4@http://climatedata.blprnt.com/ Thanks, Tim. I'm teaching right now but I'll have a closer look at those links when I get back home. The Guardian Data store typically releases data in Google Spreadsheets format which would be a nice accessible place for people to work from, but in this case it looks like they are in CSV format. tim on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-3 Wed, 09 Dec 2009 07:54:15 +0000 tim 3@http://climatedata.blprnt.com/ possibly easier to access subsets/copies are available here <a href="http://timetric.com/tags/HadCRUT3/" rel="nofollow">http://timetric.com/tags/HadCRUT3/</a> and here <a href="http://www.guardian.co.uk/environment/datablog/2009/dec/08/uk-us-temperature-change-global-met#data" rel="nofollow">http://www.guardian.co.uk/environment/datablog/2009/dec/08/uk-us-temperature-change-global-met#data</a> blprnt on "Here is the data" http://climatedata.blprnt.com/topic/here-is-the-data#post-2 Wed, 09 Dec 2009 06:10:55 +0000 blprnt 2@http://climatedata.blprnt.com/ Here is the data released today by the Met Office: <a href="http://www.metoffice.gov.uk/climatechange/science/monitoring/subsets.html" rel="nofollow">http://www.metoffice.gov.uk/climatechange/science/monitoring/subsets.html</a>