[Met_help] [rt.rap.ucar.edu #43935] History for June 2006 data?
RAL HelpDesk {for John Halley Gotway}
met_help at ucar.edu
Wed Feb 23 14:43:08 MST 2011
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello Help Desk,
I would like to locate and download as much data as possible to validate modeling between May 28, 2006 and July 3, 2006. From the MET observational datasets page I downloaded Stage IV precipitation data. I wanted to download the PrepBufr for GDAS (Global) - archive starting January 2006: http://nomads.ncdc.noaa.gov/data/gdas without have to click link after link. Is there a mirror ftp site that I can reach with wget?
If you can point me to other archived observational datasets for this time period I would greatly appreciate it.
Regards,
Bright Dornblaser
Dr. Bright Dornblaser
Technical Analysis Division MC-164
Texas Commission on Environmental Quality
P.O. Box 13087
Austin, TX 78711-3087
(512) 239-1978 (512) 239-1500 fax
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #43935] June 2006 data?
From: John Halley Gotway
Time: Mon Jan 31 12:00:07 2011
Bright,
Actually, I believe you've found the best site for exactly what you're
after. I'd suggest that you figure out the naming convention for the
PREPBUFR files you want. For example:
http://nomads.ncdc.noaa.gov/data/gdas/201010/20101008/gdas1.t00z.prepbufr.nr
http://nomads.ncdc.noaa.gov/data/gdas/201010/20101008/gdas1.t06z.prepbufr.nr
http://nomads.ncdc.noaa.gov/data/gdas/201010/20101008/gdas1.t12z.prepbufr.nr
http://nomads.ncdc.noaa.gov/data/gdas/201010/20101008/gdas1.t18z.prepbufr.nr
The listing above gives the name of the 4 PREPBUFR files for the day
20101008. Each of these files contains 6 hours of global observations
(A LOT OF DATA). You can use wget to retrieve these files.
I'd suggest creating a text file that contains a list of these file
names, and then use wget with the "-i" option to retrieve all the
files you have listed in that text file.
For example,
wget -i pb_files.txt
where pb_files.txt contains the for URLs listed above.
Lastly, just write some sort of shell script to create a list of all
the PREPBUFR file names you want. Just loop through all the days and
populate the list of the file names. The follow shell script
written for the C-shell would generate a list of all the files for
2010:
---------------------------------------------------
#!/bin/csh
# Remove the pb_files.txt if it exists
rm -f pb_files.txt
# Set the starting and ending dates
set BEG_DATE=20100101
set END_DATE=20110101
# Convert those dates to unixtime
set BEG_UT=`date -ud ${BEG_DATE} +%s`
set END_UT=`date -ud ${END_DATE} +%s`
set CUR_UT=${BEG_UT}
# Loop through the time, one day at a time
while ( ${CUR_UT} <= ${END_UT} )
# Get the current year, month, and day
set YY=`date -ud '1970-01-01 UTC '${CUR_UT}' seconds' +%Y`
set MM=`date -ud '1970-01-01 UTC '${CUR_UT}' seconds' +%m`
set DD=`date -ud '1970-01-01 UTC '${CUR_UT}' seconds' +%d`
# Write out the URLs for the current day
echo
"http://nomads.ncdc.noaa.gov/data/gdas/${YY}${MM}/${YY}${MM}${DD}/gdas1.t00z.prepbufr.nr"
>> pb_files.txt
echo
"http://nomads.ncdc.noaa.gov/data/gdas/${YY}${MM}/${YY}${MM}${DD}/gdas1.t06z.prepbufr.nr"
>> pb_files.txt
echo
"http://nomads.ncdc.noaa.gov/data/gdas/${YY}${MM}/${YY}${MM}${DD}/gdas1.t12z.prepbufr.nr"
>> pb_files.txt
echo
"http://nomads.ncdc.noaa.gov/data/gdas/${YY}${MM}/${YY}${MM}${DD}/gdas1.t18z.prepbufr.nr"
>> pb_files.txt
# Increment to the next day by adding 86400 seconds
set CUR_UT=`expr ${CUR_UT} + 86400`
end
----------------------------------------------
Hope that helps.
John Halley Gotway
met_help at ucar.edu
On 01/28/2011 01:54 PM, RAL HelpDesk {for Bright Dornblaser} wrote:
>
> Fri Jan 28 13:54:19 2011: Request 43935 was acted upon.
> Transaction: Ticket created by Bright.Dornblaser at tceq.texas.gov
> Queue: met_help
> Subject: June 2006 data?
> Owner: Nobody
> Requestors: Bright.Dornblaser at tceq.texas.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=43935 >
>
>
> Hello Help Desk,
>
> I would like to locate and download as much data as possible to
validate modeling between May 28, 2006 and July 3, 2006. From the MET
observational datasets page I downloaded Stage IV precipitation data.
I wanted to download the PrepBufr for GDAS (Global) - archive starting
January 2006: http://nomads.ncdc.noaa.gov/data/gdas without have to
click link after link. Is there a mirror ftp site that I can reach
with wget?
>
> If you can point me to other archived observational datasets for
this time period I would greatly appreciate it.
>
> Regards,
>
> Bright Dornblaser
>
>
> Dr. Bright Dornblaser
> Technical Analysis Division MC-164
> Texas Commission on Environmental Quality
> P.O. Box 13087
> Austin, TX 78711-3087
> (512) 239-1978 (512) 239-1500 fax
------------------------------------------------
Subject: June 2006 data?
From: Bright Dornblaser
Time: Mon Jan 31 17:05:06 2011
Hello John,
Thank you for your reply. I have been browsing through June 2006,
(for example http://nomads.ncdc.noaa.gov/data/gdas/200606/20060606...)
to try and find the right file names to script as you suggest below.
I don't actually see any prepbufr files on June 2006 days. So, if I
want to try the MET software (point-stat or grid-stat) it seems like
the gdas-adpupa/gdas-adpsfc/gdas-proflr 999 20060606
0000/0600/1200/1800 000.bufr files might be appropriate.
Is this correct? Are the other files useful too?
Thanks again for taking the time to help direct me to the right files.
Bright
>>> "RAL HelpDesk {for John Halley Gotway}" <met_help at ucar.edu>
1/31/2011 1:00 PM >>>
Bright,
Actually, I believe you've found the best site for exactly what you're
after. I'd suggest that you figure out the naming convention for the
PREPBUFR files you want. For example:
http://nomads.ncdc.noaa.gov/data/gdas/201010/20101008/gdas1.t00z.prepbufr.nr
http://nomads.ncdc.noaa.gov/data/gdas/201010/20101008/gdas1.t06z.prepbufr.nr
http://nomads.ncdc.noaa.gov/data/gdas/201010/20101008/gdas1.t12z.prepbufr.nr
http://nomads.ncdc.noaa.gov/data/gdas/201010/20101008/gdas1.t18z.prepbufr.nr
The listing above gives the name of the 4 PREPBUFR files for the day
20101008. Each of these files contains 6 hours of global observations
(A LOT OF DATA). You can use wget to retrieve these files.
I'd suggest creating a text file that contains a list of these file
names, and then use wget with the "-i" option to retrieve all the
files you have listed in that text file.
For example,
wget -i pb_files.txt
where pb_files.txt contains the for URLs listed above.
Lastly, just write some sort of shell script to create a list of all
the PREPBUFR file names you want. Just loop through all the days and
populate the list of the file names. The follow shell script
written for the C-shell would generate a list of all the files for
2010:
---------------------------------------------------
#!/bin/csh
# Remove the pb_files.txt if it exists
rm -f pb_files.txt
# Set the starting and ending dates
set BEG_DATE=20100101
set END_DATE=20110101
# Convert those dates to unixtime
set BEG_UT=`date -ud ${BEG_DATE} +%s`
set END_UT=`date -ud ${END_DATE} +%s`
set CUR_UT=${BEG_UT}
# Loop through the time, one day at a time
while ( ${CUR_UT} <= ${END_UT} )
# Get the current year, month, and day
set YY=`date -ud '1970-01-01 UTC '${CUR_UT}' seconds' +%Y`
set MM=`date -ud '1970-01-01 UTC '${CUR_UT}' seconds' +%m`
set DD=`date -ud '1970-01-01 UTC '${CUR_UT}' seconds' +%d`
# Write out the URLs for the current day
echo
"http://nomads.ncdc.noaa.gov/data/gdas/${YY}${MM}/${YY}${MM}${DD}/gdas1.t00z.prepbufr.nr"
>> pb_files.txt
echo
"http://nomads.ncdc.noaa.gov/data/gdas/${YY}${MM}/${YY}${MM}${DD}/gdas1.t06z.prepbufr.nr"
>> pb_files.txt
echo
"http://nomads.ncdc.noaa.gov/data/gdas/${YY}${MM}/${YY}${MM}${DD}/gdas1.t12z.prepbufr.nr"
>> pb_files.txt
echo
"http://nomads.ncdc.noaa.gov/data/gdas/${YY}${MM}/${YY}${MM}${DD}/gdas1.t18z.prepbufr.nr"
>> pb_files.txt
# Increment to the next day by adding 86400 seconds
set CUR_UT=`expr ${CUR_UT} + 86400`
end
----------------------------------------------
Hope that helps.
John Halley Gotway
met_help at ucar.edu
On 01/28/2011 01:54 PM, RAL HelpDesk {for Bright Dornblaser} wrote:
>
> Fri Jan 28 13:54:19 2011: Request 43935 was acted upon.
> Transaction: Ticket created by Bright.Dornblaser at tceq.texas.gov
> Queue: met_help
> Subject: June 2006 data?
> Owner: Nobody
> Requestors: Bright.Dornblaser at tceq.texas.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=43935 >
>
>
> Hello Help Desk,
>
> I would like to locate and download as much data as possible to
validate modeling between May 28, 2006 and July 3, 2006. From the MET
observational datasets page I downloaded Stage IV precipitation data.
I wanted to download the PrepBufr for GDAS (Global) - archive starting
January 2006: http://nomads.ncdc.noaa.gov/data/gdas without have to
click link after link. Is there a mirror ftp site that I can reach
with wget?
>
> If you can point me to other archived observational datasets for
this time period I would greatly appreciate it.
>
> Regards,
>
> Bright Dornblaser
>
>
> Dr. Bright Dornblaser
> Technical Analysis Division MC-164
> Texas Commission on Environmental Quality
> P.O. Box 13087
> Austin, TX 78711-3087
> (512) 239-1978 (512) 239-1500 fax
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #43935] June 2006 data?
From: John Halley Gotway
Time: Tue Feb 01 10:59:28 2011
Bright,
Unfortunately no, the PB2NC tool won't be able to read those ".bufr"
files. It's only set up to read the PREPBUFR files with "prepbufr" in
their name. I took a closer look at that NOMADS site and
see that the first date for which PREPBUFR data is available is
20070501:
http://nomads.ncdc.noaa.gov/data/gdas/200705/20070501/gdas1.t00z.prepbufr.nr
I'd suggest trying to NCAR CISL site:
http://dss.ucar.edu/datasets/ds337.0
It looks like their archive extends much further back than we had
listed on the MET website. They claim it goes back to 1997! So I
updated the dates on the MET observations page to reflect these
changes. To access the CISL data, you'll need to create an account,
but I don't think there should be any problem with doing so.
Hopefully that'll do the trick for you.
Thanks,
John
On 01/31/2011 05:05 PM, RAL HelpDesk {for Bright Dornblaser} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=43935 >
>
> Hello John,
>
> Thank you for your reply. I have been browsing through June 2006,
(for example http://nomads.ncdc.noaa.gov/data/gdas/200606/20060606...)
to try and find the right file names to script as you suggest below.
I don't actually see any prepbufr files on June 2006 days. So, if I
want to try the MET software (point-stat or grid-stat) it seems like
the gdas-adpupa/gdas-adpsfc/gdas-proflr 999 20060606
0000/0600/1200/1800 000.bufr files might be appropriate.
>
> Is this correct? Are the other files useful too?
>
> Thanks again for taking the time to help direct me to the right
files.
>
> Bright
>
>>>> "RAL HelpDesk {for John Halley Gotway}" <met_help at ucar.edu>
1/31/2011 1:00 PM >>>
> Bright,
>
> Actually, I believe you've found the best site for exactly what
you're after. I'd suggest that you figure out the naming convention
for the PREPBUFR files you want. For example:
>
http://nomads.ncdc.noaa.gov/data/gdas/201010/20101008/gdas1.t00z.prepbufr.nr
>
http://nomads.ncdc.noaa.gov/data/gdas/201010/20101008/gdas1.t06z.prepbufr.nr
>
http://nomads.ncdc.noaa.gov/data/gdas/201010/20101008/gdas1.t12z.prepbufr.nr
>
http://nomads.ncdc.noaa.gov/data/gdas/201010/20101008/gdas1.t18z.prepbufr.nr
>
> The listing above gives the name of the 4 PREPBUFR files for the day
20101008. Each of these files contains 6 hours of global observations
(A LOT OF DATA). You can use wget to retrieve these files.
> I'd suggest creating a text file that contains a list of these file
names, and then use wget with the "-i" option to retrieve all the
files you have listed in that text file.
>
> For example,
> wget -i pb_files.txt
> where pb_files.txt contains the for URLs listed above.
>
> Lastly, just write some sort of shell script to create a list of all
the PREPBUFR file names you want. Just loop through all the days and
populate the list of the file names. The follow shell script
> written for the C-shell would generate a list of all the files for
2010:
>
> ---------------------------------------------------
>
> #!/bin/csh
>
> # Remove the pb_files.txt if it exists
> rm -f pb_files.txt
>
> # Set the starting and ending dates
> set BEG_DATE=20100101
> set END_DATE=20110101
>
> # Convert those dates to unixtime
> set BEG_UT=`date -ud ${BEG_DATE} +%s`
> set END_UT=`date -ud ${END_DATE} +%s`
> set CUR_UT=${BEG_UT}
>
> # Loop through the time, one day at a time
> while ( ${CUR_UT} <= ${END_UT} )
>
> # Get the current year, month, and day
> set YY=`date -ud '1970-01-01 UTC '${CUR_UT}' seconds' +%Y`
> set MM=`date -ud '1970-01-01 UTC '${CUR_UT}' seconds' +%m`
> set DD=`date -ud '1970-01-01 UTC '${CUR_UT}' seconds' +%d`
>
> # Write out the URLs for the current day
> echo
"http://nomads.ncdc.noaa.gov/data/gdas/${YY}${MM}/${YY}${MM}${DD}/gdas1.t00z.prepbufr.nr"
>> pb_files.txt
> echo
"http://nomads.ncdc.noaa.gov/data/gdas/${YY}${MM}/${YY}${MM}${DD}/gdas1.t06z.prepbufr.nr"
>> pb_files.txt
> echo
"http://nomads.ncdc.noaa.gov/data/gdas/${YY}${MM}/${YY}${MM}${DD}/gdas1.t12z.prepbufr.nr"
>> pb_files.txt
> echo
"http://nomads.ncdc.noaa.gov/data/gdas/${YY}${MM}/${YY}${MM}${DD}/gdas1.t18z.prepbufr.nr"
>> pb_files.txt
>
> # Increment to the next day by adding 86400 seconds
> set CUR_UT=`expr ${CUR_UT} + 86400`
>
> end
>
> ----------------------------------------------
>
> Hope that helps.
>
> John Halley Gotway
> met_help at ucar.edu
>
> On 01/28/2011 01:54 PM, RAL HelpDesk {for Bright Dornblaser} wrote:
>>
>> Fri Jan 28 13:54:19 2011: Request 43935 was acted upon.
>> Transaction: Ticket created by Bright.Dornblaser at tceq.texas.gov
>> Queue: met_help
>> Subject: June 2006 data?
>> Owner: Nobody
>> Requestors: Bright.Dornblaser at tceq.texas.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=43935 >
>>
>>
>> Hello Help Desk,
>>
>> I would like to locate and download as much data as possible to
validate modeling between May 28, 2006 and July 3, 2006. From the MET
observational datasets page I downloaded Stage IV precipitation data.
I wanted to download the PrepBufr for GDAS (Global) - archive starting
January 2006: http://nomads.ncdc.noaa.gov/data/gdas without have to
click link after link. Is there a mirror ftp site that I can reach
with wget?
>>
>> If you can point me to other archived observational datasets for
this time period I would greatly appreciate it.
>>
>> Regards,
>>
>> Bright Dornblaser
>>
>>
>> Dr. Bright Dornblaser
>> Technical Analysis Division MC-164
>> Texas Commission on Environmental Quality
>> P.O. Box 13087
>> Austin, TX 78711-3087
>> (512) 239-1978 (512) 239-1500 fax
>
------------------------------------------------
More information about the Met_help
mailing list