Author Topic: Can you get several years' worth of hourly temp data somewhere online? (U.S.)  (Read 6879 times)

0 Members and 1 Guest are viewing this topic.

Offline DistFunc

  • Member
  • *
  • Posts: 26
Hi everyone,

I am trying to find hourly temperature data for Atlanta GA USA for the past 3 to 10 years. Does anyone know where I can get this online? For free.

The real question here is, HOW CAN I GET IT IN BULK. There are plenty of places to see it for individual days or maybe weeks or months. But I can't find anywhere that would not require a ton of page loads and copies, to get 3+ years.

With that much data, one really needs a single CSV, delimited, or spreadsheet download.

Please don't respond if the only thing you know is that a site has that kind of data. Only respond if you personally have gotten it en masse, or can specifically tell me how.

Sorry to be so particular. But there are lots of sites with that data, like NOAA and Weather Underground - but I can't see anyway to get a big data dump after trying *many* things and links. So that is the point of this post. Not "who has the data", but "specifically how can I get years worth of that data" fairly readily.

It is only for a little home improvement question, so I don't want to sink hours and hours, or any money, into it.

Thanks if you can help!

P.S. My first post!

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
I can't help you with the website/data you need but my curiosity is killing me:  what sort of little home improvement project requires YEARS of hourly weather data??
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline ValentineWeather

  • Forecaster
  • *****
  • Posts: 6367
    • Valentine Nebraska's Real-Time Weather
Sign up for mesowest its free. You can download the data but hourly may not allow full year at a time, uncheck the fields not needed.   http://mesowest.utah.edu/cgi-bin/droman/mesomap.cgi?state=GA&rawsflag=3
Randy

Offline DistFunc

  • Member
  • *
  • Posts: 26
Thanks Valentine, that pretty much got it! Right, they only let you get 365 days at most, so I had to do it 5 times to get five years. And hope none was a leap year, lol (j/k). But hey, that's close enough.

Bushman, it's a long story and I oversimplified. Hoping not to have to say all this plus create a major distraction that meant my question didn't get answered.

Just remember, you did ask  :shock:

I live in Atlanta GA USA. It can get below freezing in winter, but rarely goes far below.

Our very modest little old (1978) house has the wonky construction that the main (mid split) level is on a concrete slab, and this slab is exposed to open air at the back of the house.

Meanwhile, the kitchen sink is against the back wall (so there's a window over the sink) and - sigh - the pipes go straight down into the concrete slab, then angle away from the wall (diagonally under the floor, in the slab). I didn't know it when we bought the house but this is unbelievably inefficient, because you essentially have to heat the whole slab of the kitchen floor to get hot water at that sink. And if you stop for a minute, you've lost all your heat headway. It is the opposite of insulation, lol.

Anyway, right at the point where the water pipes go into the slab, before they angle away (an inch from the exposed back of the house), they are liable to freeze if it gets REAL cold outside. (Of course, the slab itself is being heated by the house, which we keep at 68 in winter.) By experience, we have found that the hot water pipe freezes when it gets to the low teens.

It is my tendency to shut off my PC and go stream NetFlix or something on the big screen for a couple hours, before going to bed around midnight. When I go to bed, I pass by our thermostat, which includes an outside sensor so I can see how cold it is outside just before sleepily slouching off to bed in total crash mode.

To finally get to the point, I would like to know which temperatures (at midnight) mean it will probably get in the low teens by dawn. Then I can know to leave the tap dripping, simple as that, without having to start the whole 3-5 minute process of starting my PC again, etc etc, to get all the way to a forecast site.

Said another way: There really should be a very simple relation between the temp at midnight, and the chance it will get to 13 by dawn.

But it is pretty rare. It only happens a few times a winter.

That is why I need many years of data...

What I want to do is find the mornings it got to 13 (or below). First I will use it to set a boundary condition, as in "it never got down to 13 if the midnight temp was at least 24" (or whatever). So then I can discard everything above 24 at midnight (plus know I never have to worry if it's 24).

Then, with what's left, I can plot a "distribution function" (notice my handle is DistFunc) something like this:

If the temp is 23 at midnight, there's a 5% chance it will get to 13 at dawn
If the temp is 20 at midnight, there's a 50% chance it will get to 13 at dawn
If the temp is 16 at midnight, there's a 95% chance it will get to 13 at dawn

Something like that.

So I will have made myself a "rule of thumb" that, somewhere between 16 to 23, I need to start the the tap dripping.

Probably some folks would shake their head and say, WTH, why not either check the forecast OR just leave it dripping. But data is my hobby, what can I say... if I have a chance to analyze a ton of data, plus learn a way to get my hands on tons of it (when I always have home insulation and degree-day concerns on my mind), plus finally make a nice simple rule of thumb for when I'm sleepily heading off to bed, well... I'm going to take it.  UU

So it's not really home improvement, as much as home-life simplification. A very nerdy way to be lazy.

You asked.

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
Not at all sorry I asked.  :)  Thanks for the explanation.  Interesting problem that I would solve this way:  use a recirculation pump that is started by outside temps.  Total cost might be 100 bucks or so depending on the pump you choose.  Having to turn a tap to drip at midnight seems like a lot of work.  As for getting instant hot water, I would install an undersink instant water heater.
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline geofb13

  • Contributor
  • ***
  • Posts: 131
Just as an aside if you need a lot of data in the future, besides Mesowest you can always go check out NCDC if you want a large range of data in one shot. Either way is fine but just thought I'd give you another option for the future. Good luck with the data analysis.

Offline ocala

  • Forecaster
  • *****
  • Posts: 4396
  • The blues had a baby and named it rock n roll


I'm sorry, I didn't notice in your post that you have an outside temp sensor. Please disregard,
 
« Last Edit: March 01, 2015, 11:52:08 AM by ocala »

Offline DistFunc

  • Member
  • *
  • Posts: 26
Bushman, yes well it only takes 15-30 seconds to start water dripping, and zero expense, three nights a year. But it would cost hours if not days to research and install something. (I can't buy even the smallest thing these days without spending hours online comparing, it seems.) I don't even know how a recirculation pump would work on a pipe stuck in a slab, but it doesn't matter because it would be way more than 60 seconds a year and no money.  ;)

The undersink water heater is a great idea in theory, but in reality in this little 1800 sq. ft. split level, the upstairs bathrooms are right over the water heater in the lowest level, and get instant hot water. So we just live with it, and just walk the 25' and half a flight of stairs if we only need a little hot water. If we need a lot (washing dishes), then using the kitchen hot water is worth it anyway. It isn't actually horrible; it does okay if you wait and then keep the flow... cement isn't a perfect heat sink.  8-[

geofb13, I specifically called out NOAA as one of the places I asked for specifics on how to get the lots of data, instead of being told that they have lots of data. I know they have lots of data. I've taken many failed shots at trying to get my hands on all their data. The question is, how to get 3-10 years of weather data in a reasonable way. From NOAA or anywhere else. Valentine came through, with Mesowest. Can you tell me specific steps to get three years of a station's hourly temperature data from NOAA in one fell swoop?

Thanks all!
« Last Edit: March 01, 2015, 04:23:20 PM by DistFunc »

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
And...  how much time have you spent so far?  :)
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline DistFunc

  • Member
  • *
  • Posts: 26
And...  how much time have you spent so far?  :)

Perfectly valid question but I mean it when I say I'm a data hobbyist. "if I have a chance to analyze a ton of data, plus learn a way to get my hands on tons of it (when I always have home insulation and degree-day concerns on my mind), plus finally make a nice simple rule of thumb for when I'm sleepily heading off to bed, well... I'm going to take it."
UU

Most people wouldn't want to do this. But I do. I enjoy it. I may one day start a consumer-oriented small business dealing with heating and cooling personal data analysis, too. (Whether it's cost effective for them to buy more insulation, etc.) It's all good, to me.

So, this is what works for me. But I'm in a real particular situation... data skills, but don't want to actually check forecast (I don't have a smartphone) plus don't want to permanently fix it hardware wise, shrug.

It occurs to me that I'm sort of on to something... I bet there are other situations where folks could use a rule of thumb based on a few simple principles and a little weather data analysis. Just cut offs or boundaries as to what temperatures mean whether one should worry...

...but then again, what's the incentive when so many can just check a forecast so easily. shrug.



Offline geofb13

  • Contributor
  • ***
  • Posts: 131
geofb13, I specifically called out NOAA as one of the places I asked for specifics on how to get the lots of data, instead of being told that they have lots of data. I know they have lots of data. I've taken many failed shots at trying to get my hands on all their data. The question is, how to get 3-10 years of weather data in a reasonable way. From NOAA or anywhere else. Valentine came through, with Mesowest. Can you tell me specific steps to get three years of a station's hourly temperature data from NOAA in one fell swoop?

Thanks all!

Sorry for my obvious miss on that from your original post though it's not all that hard at least in my mind to get data from their site. That was the reasoning behind my not just giving you a link directly to a map.

From the NCDC homepage:
Click on Data Access and then click on Land-Based Station for this instance, obviously if you'd want satellite or radar data you'd click on the respective link. From the land-based station page I'd click on the Climate Data Online (CDO) link which will bring you to a nice new website where you can search or map to your hearts content to find the station that's nearest to you with the data that you want.

If you wanted the data from the Atlanta Heartsfield International Airport you'd get to this page through CDO. From there you'd click Add to cart, and when you're in the cart you can select how you want the data (pdf, csv, txt) and the range for which you want the data. Depending on the format you'd then be able to select if you want all of the extra data regarding the station site info, any flagged or marked observations, and which meteorological data you'd specifically want. You finish that up by giving them your email and then they email you a link once the data is processed which for me for Atlanta Heartsfield took a minute from the submission email to completed email.

The NCDC site has changed a lot over the last year or so and I always used to get my data solely through their HDSS system but it required me to know the station's number beforehand. I hope that helps, and I really didn't mean to say anything bad about Mesowest if my initial comment sounded that way. NCDC isn't necessarily the easiest site to get around but once you know where the stuff is that you need it's fairly quick and easy to get years and years of data.



Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
NCDC is only daily SUMMARY is it not?
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline geofb13

  • Contributor
  • ***
  • Posts: 131
NCDC is only daily SUMMARY is it not?

No, you should be able to get hourly data. Perhaps not for every station through the CDO, but if you go to the ASOS or AWOS data sets they have them at the hourly.

Surface Data Hourly Global (DS3505)

If you know the station number and it's ASOS station that updates at a faster frequency you should still be able to get that data through the FTP service.

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
Bingo! 
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline DistFunc

  • Member
  • *
  • Posts: 26
NCDC is only daily SUMMARY is it not?

No, you should be able to get hourly data. Perhaps not for every station through the CDO, but if you go to the ASOS or AWOS data sets they have them at the hourly.

Surface Data Hourly Global (DS3505)

If you know the station number and it's ASOS station that updates at a faster frequency you should still be able to get that data through the FTP service.

geofb13, I'm sorry, but when I follow your CDO link, I can only get daily summaries. What am I doing wrong? If I switch to Hourly Normals, suddenly the stations available evaporate and the date ranges become very hard to work with (esp. for multiple years).

Do you mind humoring me, cut and paste just the first 10 lines of the data they sent you into your reply here. (Column headers and some data.) To prove you can get hourly data that way.

I spent 40 minutes on that site, again, just now. Again, no luck. This is on top of hours spent in the past.

As for the Surface Data, I spent 20 minutes trying to figure out what the Atlanta airport station ID could possibly be. No luck, not really. Where is the secret decoder ring for Station IDs? Maybe I need to be at 28.8 kb/sec to see it, as the Help mentions, lol.

I'm not trying to be glib and I appreciate you trying to help. But I am very tired and frustrated at looking at these same sites over and over and having no luck. How can it be so hard?



« Last Edit: March 02, 2015, 10:23:42 AM by DistFunc »

Offline ValentineWeather

  • Forecaster
  • *****
  • Posts: 6367
    • Valentine Nebraska's Real-Time Weather
Mesowest gives the ID's  KATL is  Atlanta, Hartsfield - Jackson Atlanta International
KFTY-  Atlanta, Fulton County Airport-Brown Field,   KPDK - Atlanta, De Kalb-Peachtree Airport
There are more in the Atlanta area.
Randy

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
DistFunc here's the file format.  http://hurricane.ncdc.noaa.gov/cdo/3505doc.txt It can take up to 24 hrs to fill a request.  I found the hourly obs for Atlanta airport http://www1.ncdc.noaa.gov/pub/orders/9616886634267stn.txt  and selected last 4 months  - http://www1.ncdc.noaa.gov/pub/orders/9616886634267dat.txt  Should be there shortly.
« Last Edit: March 02, 2015, 10:55:21 AM by Bushman »
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline DistFunc

  • Member
  • *
  • Posts: 26
Thanks Valentine, but only numeric Station IDs are allowed.

And on the station lists there are places in Georgia, some of which also say Atlanta, but not a one actually looks like the Atlanta Airport to me. I don't know what or exactly where those stations are.

I don't want to start the whole long process of manipulating and analyzing data, if I don't know what I have to begin with.

Offline DistFunc

  • Member
  • *
  • Posts: 26
DistFunc here's the file format.  http://hurricane.ncdc.noaa.gov/cdo/3505doc.txt It can take up to 24 hrs to fill a request.  I found the hourly obs for Atlanta airport http://www1.ncdc.noaa.gov/pub/orders/9616886634267stn.txt  and selected last 4 months  - http://www1.ncdc.noaa.gov/pub/orders/9616886634267dat.txt  Should be there shortly.
Hmm, that's the Augusta airport, Bushman. I appreciate you sticking with it though, let's see if you can get hourly results.

Offline ValentineWeather

  • Forecaster
  • *****
  • Posts: 6367
    • Valentine Nebraska's Real-Time Weather
Try this for codes. http://airport-codes.findthebest.com/
Atlanta, GA: Atlanta Airport
Since airport codes can be reused, DOT also assigned the airport the ID #58515, which identifies a unique airport.
« Last Edit: March 02, 2015, 11:10:48 AM by ValentineWeather »
Randy

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
http://www1.ncdc.noaa.gov/pub/orders/72218003820-2014-12_9616886634267dat.html  Yeah, Augusta not Atlanta but you get the idea (that's the HTML page; txt file is the same without the formatting).  The txt file is one big file; the HTML you would have to combine/CAT.  Not sure what SW you are using for your analysis but  Excel (ugh) could probably handle it.  I'd use SAS or some other industrial strength data analysis SW.
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
Here's a snippet from the txt file:

  USAF  WBAN YR--MODAHRMN DIR SPD GUS CLG SKC L M H  VSB MW MW MW MW AW AW AW AW W TEMP DEWP    SLP   ALT    STP MAX MIN PCP01 PCP06 PCP24 PCPXX SD
722180 03820 201412010000 170   6 *** *** *** * * * 10.0 ** ** ** ** ** ** ** ** *   48   42 1026.1 ***** 1020.7  71  29 ***** ***** ***** ***** **
722180 03820 201412010053 ***   0 *** 722 CLR * * * 10.0 ** ** ** ** ** ** ** ** *   49   43 1026.7 30.32 1021.3 *** ***  0.00 ***** ***** ***** **
722180 03820 201412010153 ***   0 *** 722 CLR * * * 10.0 ** ** ** ** ** ** ** ** *   46   42 1027.2 30.33 1021.6 *** ***  0.00 ***** ***** ***** **
722180 03820 201412010253 ***   0 *** 722 CLR * * * 10.0 ** ** ** ** ** ** ** ** *   43   42 1027.7 30.35 1022.3 *** ***  0.00 ***** ***** ***** **
722180 03820 201412010353 ***   0 *** 722 CLR * * * 10.0 ** ** ** ** ** ** ** ** *   41   40 1028.1 30.36 1022.6 *** ***  0.00 ***** ***** ***** **
722180 03820 201412010453 ***   0 *** 722 CLR * * *  6.0 ** ** ** ** 10 ** ** ** *   40   39 1028.0 30.36 1022.6  71  29  0.00 ***** ***** ***** **
722180 03820 201412010459 *** *** *** *** *** * * * **** ** ** ** ** ** ** ** ** * **** **** ****** ***** ******  71 *** ***** *****  0.00 *****  0
722180 03820 201412010553 020   3 *** 722 CLR * * *  8.0 ** ** ** ** ** ** ** ** *   38   37 1028.2 30.37 1023.0  52  37  0.00 ***** ***** ***** **
722180 03820 201412010600 020   3 *** *** *** * * *  7.5 ** ** ** ** ** ** ** ** *   38   37 1028.2 ***** 1023.1  70  29 ***** ***** ***** ***** **
722180 03820 201412010653 ***   0 *** 722 CLR * * *  2.5 ** ** ** ** 10 ** ** ** *   39   38 1028.3 30.37 1023.0 *** ***  0.00 ***** ***** ***** **
722180 03820 201412010753 ***   0 *** 722 CLR * * *  8.0 ** ** ** ** ** ** ** ** *   37   36 1028.4 30.37 1023.0 *** ***  0.00 ***** ***** ***** **
722180 03820 201412010853 ***   0 *** 722 CLR * * *  8.0 ** ** ** ** ** ** ** ** *   38   36 1028.5 30.38 1023.3 *** ***  0.00 ***** ***** ***** **
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline DistFunc

  • Member
  • *
  • Posts: 26
Thanks very much Bushman - ok, now we see that it does indeed have hours. Sure, that kind of data can be imported easily. I use MS Access (and export stats to Excel for graphs).

Now if only we can figure out what is up with the Stations. Preferably, an actual map to each Station ID.

geofb, let us know if you can get hourly data for a long time period (years) out of NOAA/NCDC/CDO, otherwise we will assume you mis-spoke and one can't get hourly data for long periods from CDO. Which is a shame because the CDO interface looks newer and perhaps more useful. Instead we have to use the Surface Data Hourly Global, which I presume, but don't know, has access to the exact same data. Oh well.


I'm starting to put together some stats from Valentine's Mesowest data. I have *only* used the minimum temperature for each day in this chart and gotten stats across all five years' data, for the twelve months of the year (see attached). Temperature (in F) is on left axis, time of day is on right.  Each month here summarizes ~150 days (~30 days/month x 5 years).

For one thing, we can instantly see that it only gets to low teens in December and January (MinOfMinTemp, the minima of the minima for each month). And that the minimum temps happened at about 9 a.m. in those two months.

Don't read too much into this chart; it has a lot of limitations. For example, the data is close to, but not actually hourly, if you look close. Sometimes it's more often, and there are gaps. The data is actually March 2010 - Feb 2015.

Next I will assess what temps looked like at midnight, for dawns when it got to 13 F. Pairing hourly data for days it went that low.
« Last Edit: March 02, 2015, 01:08:20 PM by DistFunc »

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
Just dump the list of all Georgia stations from the link I gave earlier and you will get the lat/long for each.  Plot 'em in Google Maps or ESRI.  This is all pretty simple.
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline DistFunc

  • Member
  • *
  • Posts: 26
Huh, in the past 5 years, the minimum temp was 13 or below only nine times. And two are clear anomalies ... I guess that's only natural when you summarize so much data (24 hours x 365 days x 5 years); of course it won't be perfect, and glitches are liable to stand out crazily as mins or maxs.

Of the seven that were for real, the highest temp at midnight was 21. Some of the others were so low it was almost 13 anyway.

I'm not going to bother with turning it into percentile data. (How many times it didn't get down to 13 when it was 21. Etc.) The rule of thumb here is, 21 or lower. That's good enough. This has already been more work than I thought.

Just dump the list of all Georgia stations from the link I gave earlier and you will get the lat/long for each.  Plot 'em in Google Maps or ESRI.  This is all pretty simple.
Pretty clunky. Oh well.

I found that, instead of trying to know/guess/figure out/map plot the Station ID, that instead one can select e.g. Country of interest, then follow down a path to find state and stations. The station names here make a lot more sense (unlike trying to guess at it from those other Station ID Help files), and you can clearly see other info such as times spanned, and missing observation counts.

You can also see that the Station IDs where nowhere near what anyone might've guessed. It is the USAF ids, plus apparently, some totally arbitrary numbers tacked on, lol. That's why I did not want to try to guess or "take a shot at" just using data with some/any old id, Valentine. Truly, you need that data to tell you what it wants - not use some totally unrelated sites to guess what it's using under the hood.

Thanks everyone, we have gotten there two separate ways, so special thanks to Valentine and Bushman. But if we can ever get the NCO site to output long intervals of hourly data, I wish we could. (Or otherwise know for sure that it is the exact same data as the SDHG data, with absolutely no additional qa, value added, or anything at all. Zero difference.)


 

anything