[Met_help] [rt.rap.ucar.edu #90042] History for Reading Precip Climo File
John Halley Gotway via RT
met_help at ucar.edu
Fri May 3 12:07:33 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
John, now that you fixed my CF 1.7 climo file issue, I am trying to read a MET type climo file (I believe). The files are by month and in the file are variables for several precip thresholds at the four cycles (0,6,12 18). When I try to find the climo file variable valid for 00Z, I get the following error:
DEBUG 2: Reading data for PROB(APCP_06>0.100)/A006/PROB.
99 DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
100 DEBUG 3: Applying conversion function.
101 DEBUG 3: Skipping climatology field since time offset exceeds the specified "time_step" (64800 >= 21600) from file: /h/data/global/WXQC/data/climo/cfsr/converted/cfsr_freq_p6_worl d_mo_3.nc
102 DEBUG 3: Found 0 climatology fields.
103 DEBUG 2: For PROB(APCP_06>0.100)/A006/PROB found 1 forecast levels, 0 climatology mean levels, and 0 climatology standard deviation levels.
If I try to read the clmo file variable valid for 06Z, I get:
Reading data for PROB(APCP_06>0.010)/A006/PROB.
315 DEBUG 3: Use the forecast grid.
316 DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 720 Ny: 361 lat_ll: -90.000 lon_ll: -0.000 delta_lat: 0.500 delta_lon: 0.500
317 DEBUG 2: Processing masking regions.
318 DEBUG 3: Processing grid mask: FULL
319 DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
320 DEBUG 3: Applying conversion function.
321 DEBUG 3: Skipping climatology field since time offset exceeds the specified "time_step" (43200 >= 21600) from file: /h/data/global/WXQC/data/climo/cfsr/converted/cfsr_freq_p6_worl d_mo_3.nc
322 DEBUG 3: Found 0 climatology fields.
323 DEBUG 2: For PROB(APCP_06>0.010)/A006/PROB found 1 forecast levels, 0 climatology mean levels, and 0 climatology standard deviation levels.
For 12Z the time step is 21600>=21600 and it doesn't read but the 18Z variable does read:
DEBUG 2: Reading data for PROB(APCP_06>0.010)/A006/PROB.
773 DEBUG 3: Use the forecast grid.
774 DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 720 Ny: 361 lat_ll: -90.000 lon_ll: -0.000 delta_lat: 0.500 delta_lon: 0.500
775 DEBUG 2: Processing masking regions.
776 DEBUG 3: Processing grid mask: FULL
777 DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
778 DEBUG 3: Applying conversion function.
779 DEBUG 1: Regridding climatology CFSR_APCP_T01_00(*,*) to the verification grid.
780 DEBUG 3: For forecast valid at 20190314_180000, found 1 climatology field(s) valid at 20190314_180000.
781 DEBUG 3: Found 1 climatology fields.
782 DEBUG 2: For PROB(APCP_06>0.010)/A006/PROB found 1 forecast levels, 1 climatology mean levels, and 0 climatology standard deviation levels.
783 DEBUG 2:
I updated the variable's attributes to what I thought they should be but I am still missing something.
I sent climo file to you via AMRSAFE.
Thanks
Bob
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Reading Precip Climo File
From: John Halley Gotway
Time: Thu May 02 15:50:11 2019
Bob,
Thanks for sending the sample file via armdec. I grabbed it and am
taking
a look. I see that you've created a single NetCDF file which contains
different variable names for 4 different times of the day: 00, 06, 12,
18.
When we initially developed the climatology logic, we did so using
GRIB
files. And those GRIB files contain separate records for the 4
different
times of day: 00, 06, 12, and 18. But the data was encoded as the
*SAME*
variable. For example, looking at 2m temperature, we have the
following 4
records:
15:1491706:d=79070500:TMP:kpds5=11:kpds6=105:kpds7=2:TR=51:P1=0:P2=0:TimeU=1:2
m above gnd:0-0hr product:ave at 1yr:NAve=31
35:3562210:d=79070506:TMP:kpds5=11:kpds6=105:kpds7=2:TR=51:P1=0:P2=0:TimeU=1:2
m above gnd:0-0hr product:ave at 1yr:NAve=31
55:5632714:d=79070512:TMP:kpds5=11:kpds6=105:kpds7=2:TR=51:P1=0:P2=0:TimeU=1:2
m above gnd:0-0hr product:ave at 1yr:NAve=31
75:7719506:d=79070518:TMP:kpds5=11:kpds6=105:kpds7=2:TR=51:P1=0:P2=0:TimeU=1:2
m above gnd:0-0hr product:ave at 1yr:NAve=31
So if we configure the climo section to look for 'name="TMP";
level="Z2";'
and the get 4 matching records. We use the timing info to pick the
records
we should use. And that's where specifying the "time_step = 21600;"
is
important. We only want to use TMP records when they are "close to"
the
time of day of the forecast we're evaluating. And by "close to", I
mean <=
time_step. For example, if our forecast is valid at 09Z, we want to
find
the data for 06Z and 12Z and interpolate in time to 09Z.
Now, the way you've organized this data is different. The variable
name
differs for each time of day. So the code won't be able to find
multiple
matching records and interpolate in the time. If you think about the
variable named "CFSR_APCP_T100_18", that data is valid at 18Z (not 00,
06,
or 12). So specifying a time_step = 21600 isn't really correct.
Instead,
you could increase it to 86400 (24 hours * 3600 seconds/hour).
I see 2 options here...
(1) Keep your data stored as-is, set time_step = 86400, and realize
that
Grid-Stat won't be able to do the time interpolation for climatology
data.
(2) Or you could restructure your data into 4 separate output files...
one
each of 00, 06, 12, and 18. In those files, remove the _00, _06, _12,
and
_18 from the variable names. So that way we have exactly the same
variable
names used across the 4 files, but the timing info in the variable
attributes differ. Keep the time_step = 21600. In the climo_mean
section,
instead of setting "file_name' equal to 1 file, set it to a list of
all 4
files. Grid-Stat will search through the list, consider the variable
requested from each file, and apply the time_step logic to support
time
interpolation.
The logic of the second option is pretty similar to the logic of using
GRIB
data.
Hope this helps.
Thanks,
John
On Thu, May 2, 2019 at 10:33 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Thu May 02 10:33:26 2019: Request 90042 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Reading Precip Climo File
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90042 >
>
>
> John, now that you fixed my CF 1.7 climo file issue, I am trying to
read a
> MET type climo file (I believe). The files are by month and in
the file
> are variables for several precip thresholds at the four cycles
(0,6,12
> 18). When I try to find the climo file variable valid for 00Z, I
get the
> following error:
>
> DEBUG 2: Reading data for PROB(APCP_06>0.100)/A006/PROB.
> 99 DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> 100 DEBUG 3: Applying conversion function.
> 101 DEBUG 3: Skipping climatology field since time offset exceeds
the
> specified "time_step" (64800 >= 21600) from file:
> /h/data/global/WXQC/data/climo/cfsr/converted/cfsr_freq_p6_worl
> d_mo_3.nc
> 102 DEBUG 3: Found 0 climatology fields.
> 103 DEBUG 2: For PROB(APCP_06>0.100)/A006/PROB found 1 forecast
levels, 0
> climatology mean levels, and 0 climatology standard deviation
levels.
>
> If I try to read the clmo file variable valid for 06Z, I get:
>
> Reading data for PROB(APCP_06>0.010)/A006/PROB.
> 315 DEBUG 3: Use the forecast grid.
> 316 DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 720 Ny: 361
lat_ll:
> -90.000 lon_ll: -0.000 delta_lat: 0.500 delta_lon: 0.500
> 317 DEBUG 2: Processing masking regions.
> 318 DEBUG 3: Processing grid mask: FULL
> 319 DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> 320 DEBUG 3: Applying conversion function.
> 321 DEBUG 3: Skipping climatology field since time offset exceeds
the
> specified "time_step" (43200 >= 21600) from file:
> /h/data/global/WXQC/data/climo/cfsr/converted/cfsr_freq_p6_worl
> d_mo_3.nc
> 322 DEBUG 3: Found 0 climatology fields.
> 323 DEBUG 2: For PROB(APCP_06>0.010)/A006/PROB found 1 forecast
levels, 0
> climatology mean levels, and 0 climatology standard deviation
levels.
>
> For 12Z the time step is 21600>=21600 and it doesn't read but the
18Z
> variable does read:
>
> DEBUG 2: Reading data for PROB(APCP_06>0.010)/A006/PROB.
> 773 DEBUG 3: Use the forecast grid.
> 774 DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 720 Ny: 361
lat_ll:
> -90.000 lon_ll: -0.000 delta_lat: 0.500 delta_lon: 0.500
> 775 DEBUG 2: Processing masking regions.
> 776 DEBUG 3: Processing grid mask: FULL
> 777 DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> 778 DEBUG 3: Applying conversion function.
> 779 DEBUG 1: Regridding climatology CFSR_APCP_T01_00(*,*) to the
> verification grid.
> 780 DEBUG 3: For forecast valid at 20190314_180000, found 1
climatology
> field(s) valid at 20190314_180000.
> 781 DEBUG 3: Found 1 climatology fields.
> 782 DEBUG 2: For PROB(APCP_06>0.010)/A006/PROB found 1 forecast
levels, 1
> climatology mean levels, and 0 climatology standard deviation
levels.
> 783 DEBUG 2:
>
> I updated the variable's attributes to what I thought they should be
but I
> am still missing something.
>
> I sent climo file to you via AMRSAFE.
>
> Thanks
> Bob
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #90042] Reading Precip Climo File
From: robert.craig.2 at us.af.mil
Time: Fri May 03 11:41:58 2019
John, I went with option 2 and it is working. Thanks for the help.
Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, May 2, 2019 4:50 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL
<matthew.sittel.ctr at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90042] Reading Precip
Climo File
Bob,
Thanks for sending the sample file via armdec. I grabbed it and am
taking a look. I see that you've created a single NetCDF file which
contains different variable names for 4 different times of the day:
00, 06, 12, 18.
When we initially developed the climatology logic, we did so using
GRIB files. And those GRIB files contain separate records for the 4
different times of day: 00, 06, 12, and 18. But the data was encoded
as the *SAME* variable. For example, looking at 2m temperature, we
have the following 4
records:
15:1491706:d=79070500:TMP:kpds5=11:kpds6=105:kpds7=2:TR=51:P1=0:P2=0:TimeU=1:2
m above gnd:0-0hr product:ave at 1yr:NAve=31
35:3562210:d=79070506:TMP:kpds5=11:kpds6=105:kpds7=2:TR=51:P1=0:P2=0:TimeU=1:2
m above gnd:0-0hr product:ave at 1yr:NAve=31
55:5632714:d=79070512:TMP:kpds5=11:kpds6=105:kpds7=2:TR=51:P1=0:P2=0:TimeU=1:2
m above gnd:0-0hr product:ave at 1yr:NAve=31
75:7719506:d=79070518:TMP:kpds5=11:kpds6=105:kpds7=2:TR=51:P1=0:P2=0:TimeU=1:2
m above gnd:0-0hr product:ave at 1yr:NAve=31
So if we configure the climo section to look for 'name="TMP";
level="Z2";'
and the get 4 matching records. We use the timing info to pick the
records we should use. And that's where specifying the "time_step =
21600;" is important. We only want to use TMP records when they are
"close to" the time of day of the forecast we're evaluating. And by
"close to", I mean <= time_step. For example, if our forecast is
valid at 09Z, we want to find the data for 06Z and 12Z and interpolate
in time to 09Z.
Now, the way you've organized this data is different. The variable
name differs for each time of day. So the code won't be able to find
multiple matching records and interpolate in the time. If you think
about the variable named "CFSR_APCP_T100_18", that data is valid at
18Z (not 00, 06, or 12). So specifying a time_step = 21600 isn't
really correct. Instead, you could increase it to 86400 (24 hours *
3600 seconds/hour).
I see 2 options here...
(1) Keep your data stored as-is, set time_step = 86400, and realize
that Grid-Stat won't be able to do the time interpolation for
climatology data.
(2) Or you could restructure your data into 4 separate output files...
one each of 00, 06, 12, and 18. In those files, remove the _00, _06,
_12, and
_18 from the variable names. So that way we have exactly the same
variable names used across the 4 files, but the timing info in the
variable attributes differ. Keep the time_step = 21600. In the
climo_mean section, instead of setting "file_name' equal to 1 file,
set it to a list of all 4 files. Grid-Stat will search through the
list, consider the variable requested from each file, and apply the
time_step logic to support time interpolation.
The logic of the second option is pretty similar to the logic of using
GRIB data.
Hope this helps.
Thanks,
John
On Thu, May 2, 2019 at 10:33 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Thu May 02 10:33:26 2019: Request 90042 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Reading Precip Climo File
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
>
https://usg01.safelinks.protection.office365.us/?url=https%3A%2F%2Frt.
>
rap.ucar.edu%2Frt%2FTicket%2FDisplay.html%3Fid%3D90042&data=02%7C0
>
1%7Crobert.craig.2%40us.af.mil%7Cf2f9db98cbca4398606508d6cf485984%7C83
>
31b18d2d8748efa35fac8818ebf9b4%7C1%7C1%7C636924306759651930&sdata=
> aE4s5Cs8Wkh8LjJQlK5DaRae79m22d4EXnRQMz74g0Y%3D&reserved=0 >
>
>
> John, now that you fixed my CF 1.7 climo file issue, I am trying to
read a
> MET type climo file (I believe). The files are by month and in
the file
> are variables for several precip thresholds at the four cycles
(0,6,12
> 18). When I try to find the climo file variable valid for 00Z, I
get the
> following error:
>
> DEBUG 2: Reading data for PROB(APCP_06>0.100)/A006/PROB.
> 99 DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> 100 DEBUG 3: Applying conversion function.
> 101 DEBUG 3: Skipping climatology field since time offset exceeds
the
> specified "time_step" (64800 >= 21600) from file:
> /h/data/global/WXQC/data/climo/cfsr/converted/cfsr_freq_p6_worl
> d_mo_3.nc
> 102 DEBUG 3: Found 0 climatology fields.
> 103 DEBUG 2: For PROB(APCP_06>0.100)/A006/PROB found 1 forecast
> levels, 0 climatology mean levels, and 0 climatology standard
deviation levels.
>
> If I try to read the clmo file variable valid for 06Z, I get:
>
> Reading data for PROB(APCP_06>0.010)/A006/PROB.
> 315 DEBUG 3: Use the forecast grid.
> 316 DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 720 Ny: 361
lat_ll:
> -90.000 lon_ll: -0.000 delta_lat: 0.500 delta_lon: 0.500
> 317 DEBUG 2: Processing masking regions.
> 318 DEBUG 3: Processing grid mask: FULL
> 319 DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> 320 DEBUG 3: Applying conversion function.
> 321 DEBUG 3: Skipping climatology field since time offset exceeds
the
> specified "time_step" (43200 >= 21600) from file:
> /h/data/global/WXQC/data/climo/cfsr/converted/cfsr_freq_p6_worl
> d_mo_3.nc
> 322 DEBUG 3: Found 0 climatology fields.
> 323 DEBUG 2: For PROB(APCP_06>0.010)/A006/PROB found 1 forecast
> levels, 0 climatology mean levels, and 0 climatology standard
deviation levels.
>
> For 12Z the time step is 21600>=21600 and it doesn't read but the
18Z
> variable does read:
>
> DEBUG 2: Reading data for PROB(APCP_06>0.010)/A006/PROB.
> 773 DEBUG 3: Use the forecast grid.
> 774 DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 720 Ny: 361
lat_ll:
> -90.000 lon_ll: -0.000 delta_lat: 0.500 delta_lon: 0.500
> 775 DEBUG 2: Processing masking regions.
> 776 DEBUG 3: Processing grid mask: FULL
> 777 DEBUG 3: Rescaling probabilistic field from [0,100] to [0,1].
> 778 DEBUG 3: Applying conversion function.
> 779 DEBUG 1: Regridding climatology CFSR_APCP_T01_00(*,*) to the
> verification grid.
> 780 DEBUG 3: For forecast valid at 20190314_180000, found 1
> climatology
> field(s) valid at 20190314_180000.
> 781 DEBUG 3: Found 1 climatology fields.
> 782 DEBUG 2: For PROB(APCP_06>0.010)/A006/PROB found 1 forecast
> levels, 1 climatology mean levels, and 0 climatology standard
deviation levels.
> 783 DEBUG 2:
>
> I updated the variable's attributes to what I thought they should be
> but I am still missing something.
>
> I sent climo file to you via AMRSAFE.
>
> Thanks
> Bob
>
>
------------------------------------------------
More information about the Met_help
mailing list