[Met_help] [rt.rap.ucar.edu #88898] History for pcp_combine With Sub-hourly Data
John Halley Gotway via RT
met_help at ucar.edu
Fri Feb 22 09:39:11 MST 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello there,
Can pcp_combine be used to sum sub-hourly data (e.g. 5-minute data), to
create a sub-hourly total (e.g. 15-minute data) using the -sum option?
If not, I can always code in my results using the -add option, but I wanted
to ask first since the code is already in place.
Thank you once again!
Mike
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/NCEP/Weather Prediction Center
Phone: 301-683-1546
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: pcp_combine With Sub-hourly Data
From: John Halley Gotway
Time: Wed Feb 13 10:34:27 2019
Mike,
Yes, it *should* be able to handle that. As a caveat, I haven't done
so
for projects in recent memory. Below, I've listed the usage statement
for
pcp_combine and highlighted the *in_accum* and *out_accum* options.
They
can both be formatted as HH[MMSS]... where the square brackets mean
optional. Let's say you want to accumulation 12 5-minute intervals
into a
1-hour accumulation, you'd set:
in_accum = 000500 (for 5 minutes)
out_accum = 01 (for 1 hour... or 010000 works too)
Please let me know if you find that it doesn't work as expected.
Thanks,
John
* SUM_ARGS: init_time
in_accum valid_time out_accum
out_file [-pcpdir path] [-pcprx
reg_exp] where "init_time" indicates the
initialization
time of the input data files in YYYYMMDD[_HH[MMSS]] format
(required). "in_accum" indicates the
accumulation
interval of the input data files in HH[MMSS] format
(required). "valid_time" indicates the desired
valid
time in YYYYMMDD[_HH[MMSS]] format (required).
"out_accum" indicates the desired accumulation interval for the output
NetCDF file in HH[MMSS] format (required).
"out_file" indicates the name of the output NetCDF file to be written
consisting of the sum of the accumulation intervals
(required). "-pcpdir path" overrides the
default
precipitation directory (.) (optional). "-pcprx
reg_exp" overrides the default regular expression for precipitation
file
naming convention (.*) (optional). Note: Set
init_time to 00000000_000000 when summing observation files.*
On Wed, Feb 13, 2019 at 8:18 AM Michael Erickson - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> Wed Feb 13 08:17:29 2019: Request 88898 was acted upon.
> Transaction: Ticket created by michael.j.erickson at noaa.gov
> Queue: met_help
> Subject: pcp_combine With Sub-hourly Data
> Owner: Nobody
> Requestors: michael.j.erickson at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898 >
>
>
> Hello there,
>
> Can pcp_combine be used to sum sub-hourly data (e.g. 5-minute data),
to
> create a sub-hourly total (e.g. 15-minute data) using the -sum
option?
>
> If not, I can always code in my results using the -add option, but I
wanted
> to ask first since the code is already in place.
>
> Thank you once again!
>
> Mike
>
> --
> Michael J. Erickson
>
> Research Scientist
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> NOAA/NWS/NCEP/Weather Prediction Center
> Phone: 301-683-1546
>
>
------------------------------------------------
Subject: pcp_combine With Sub-hourly Data
From: Michael Erickson - NOAA Affiliate
Time: Wed Feb 13 12:17:44 2019
Thank you again John!
I think the issue here is related to the way the netCDF file is
created.
The data contains 15 minute NEWSe precipitation and was given to me by
the
developers.
It looks like this netCDF file is not in a proper format to be read in
by
MET. I have tried:
/opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf test.ps
'name="RadarOnly_QPE_01H"; level="(*,*)";'
/opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf test.ps
'name="RadarOnly_QPE_01H"; level="R1";'
with the error:
ERROR :
ERROR : plot_data_plane -> file "20180720-181500.netcdf" not a valid
data
file
ERROR :
I have loaded the file here:
https://ftp.wpc.ncep.noaa.gov/erickson/dtc/
If there is no way to read in this data with MET, I can probably write
some
code to create a workaround.
Thanks!
Mike
On Wed, Feb 13, 2019 at 5:34 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Mike,
>
> Yes, it *should* be able to handle that. As a caveat, I haven't
done so
> for projects in recent memory. Below, I've listed the usage
statement for
> pcp_combine and highlighted the *in_accum* and *out_accum* options.
They
> can both be formatted as HH[MMSS]... where the square brackets mean
> optional. Let's say you want to accumulation 12 5-minute intervals
into a
> 1-hour accumulation, you'd set:
> in_accum = 000500 (for 5 minutes)
> out_accum = 01 (for 1 hour... or 010000 works too)
>
> Please let me know if you find that it doesn't work as expected.
>
> Thanks,
> John
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> * SUM_ARGS: init_time
> in_accum valid_time out_accum
> out_file [-pcpdir path] [-pcprx
> reg_exp] where "init_time" indicates the
initialization
> time of the input data files in YYYYMMDD[_HH[MMSS]] format
> (required). "in_accum" indicates the
accumulation
> interval of the input data files in HH[MMSS] format
> (required). "valid_time" indicates the
desired valid
> time in YYYYMMDD[_HH[MMSS]] format (required).
> "out_accum" indicates the desired accumulation interval for the
output
> NetCDF file in HH[MMSS] format (required).
> "out_file" indicates the name of the output NetCDF file to be
written
> consisting of the sum of the accumulation intervals
> (required). "-pcpdir path" overrides the
default
> precipitation directory (.) (optional). "-
pcprx
> reg_exp" overrides the default regular expression for precipitation
file
> naming convention (.*) (optional). Note: Set
> init_time to 00000000_000000 when summing observation files.*
>
>
>
> On Wed, Feb 13, 2019 at 8:18 AM Michael Erickson - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Feb 13 08:17:29 2019: Request 88898 was acted upon.
> > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > Queue: met_help
> > Subject: pcp_combine With Sub-hourly Data
> > Owner: Nobody
> > Requestors: michael.j.erickson at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898 >
> >
> >
> > Hello there,
> >
> > Can pcp_combine be used to sum sub-hourly data (e.g. 5-minute
data), to
> > create a sub-hourly total (e.g. 15-minute data) using the -sum
option?
> >
> > If not, I can always code in my results using the -add option, but
I
> wanted
> > to ask first since the code is already in place.
> >
> > Thank you once again!
> >
> > Mike
> >
> > --
> > Michael J. Erickson
> >
> > Research Scientist
> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > NOAA/NWS/NCEP/Weather Prediction Center
> > Phone: 301-683-1546
> >
> >
>
>
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/NCEP/Weather Prediction Center
Phone: 301-683-1546
------------------------------------------------
Subject: pcp_combine With Sub-hourly Data
From: John Halley Gotway
Time: Thu Feb 14 15:25:57 2019
Mike,
Thanks for posting the sample file. Yes, I agree, MET is not able to
sensibly read data from this NetCDF file. I've included the header
dump
below. I see that it contains the timing and projection information
that
MET needs, but it just doesn't follow any existing conventions that
MET
knows.
I see that you're using met-6.0 currently. If you're able to switch
to
using met-8.0, this would be a great time to use the new Python
interface
functionality. We could write a Python script to read this data,
parse the
metadata, and hand it to MET in memory.
We've posted several examples of this to the MET website here:
https://dtcenter.org/met/users/downloads/analysis_scripts.php
Is running met-8.0 an option for you?
Thanks,
John
netcdf \20180720-181500 {
dimensions:
Lat = 3500 ;
Lon = 7000 ;
variables:
float RadarOnly_QPE_01H(Lat, Lon) ;
RadarOnly_QPE_01H:Units = "mm" ;
// global attributes:
:TypeName = "RadarOnly_QPE_01H" ;
:DataType = "LatLonGrid" ;
:Time = 1532110440 ;
:FractionalTime = 0.f ;
:MissingData = -99900.f ;
:RangeFolded = -99901.f ;
:Latitude = 55.f ;
:Longitude = -130.f ;
:Height = 0.f ;
:LatGridSpacing = 0.01f ;
:LonGridSpacing = 0.01f ;
:attributes = "" ;
}
On Wed, Feb 13, 2019 at 12:18 PM Michael Erickson - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898 >
>
> Thank you again John!
>
> I think the issue here is related to the way the netCDF file is
created.
> The data contains 15 minute NEWSe precipitation and was given to me
by the
> developers.
>
> It looks like this netCDF file is not in a proper format to be read
in by
> MET. I have tried:
>
> /opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf test.ps
> 'name="RadarOnly_QPE_01H"; level="(*,*)";'
> /opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf test.ps
> 'name="RadarOnly_QPE_01H"; level="R1";'
>
> with the error:
>
> ERROR :
> ERROR : plot_data_plane -> file "20180720-181500.netcdf" not a
valid data
> file
> ERROR :
>
> I have loaded the file here:
https://ftp.wpc.ncep.noaa.gov/erickson/dtc/
>
> If there is no way to read in this data with MET, I can probably
write some
> code to create a workaround.
>
> Thanks!
>
> Mike
>
> On Wed, Feb 13, 2019 at 5:34 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Mike,
> >
> > Yes, it *should* be able to handle that. As a caveat, I haven't
done so
> > for projects in recent memory. Below, I've listed the usage
statement
> for
> > pcp_combine and highlighted the *in_accum* and *out_accum*
options. They
> > can both be formatted as HH[MMSS]... where the square brackets
mean
> > optional. Let's say you want to accumulation 12 5-minute
intervals into
> a
> > 1-hour accumulation, you'd set:
> > in_accum = 000500 (for 5 minutes)
> > out_accum = 01 (for 1 hour... or 010000 works too)
> >
> > Please let me know if you find that it doesn't work as expected.
> >
> > Thanks,
> > John
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > * SUM_ARGS: init_time
> > in_accum valid_time out_accum
> > out_file [-pcpdir path] [-pcprx
> > reg_exp] where "init_time" indicates the
initialization
> > time of the input data files in YYYYMMDD[_HH[MMSS]] format
> > (required). "in_accum" indicates the
accumulation
> > interval of the input data files in HH[MMSS] format
> > (required). "valid_time" indicates the
desired
> valid
> > time in YYYYMMDD[_HH[MMSS]] format (required).
> > "out_accum" indicates the desired accumulation interval for the
output
> > NetCDF file in HH[MMSS] format (required).
> > "out_file" indicates the name of the output NetCDF file to be
written
> > consisting of the sum of the accumulation intervals
> > (required). "-pcpdir path" overrides the
default
> > precipitation directory (.) (optional). "-
pcprx
> > reg_exp" overrides the default regular expression for
precipitation file
> > naming convention (.*) (optional). Note:
Set
> > init_time to 00000000_000000 when summing observation files.*
> >
> >
> >
> > On Wed, Feb 13, 2019 at 8:18 AM Michael Erickson - NOAA Affiliate
via RT
> <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Feb 13 08:17:29 2019: Request 88898 was acted upon.
> > > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > > Queue: met_help
> > > Subject: pcp_combine With Sub-hourly Data
> > > Owner: Nobody
> > > Requestors: michael.j.erickson at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898
> >
> > >
> > >
> > > Hello there,
> > >
> > > Can pcp_combine be used to sum sub-hourly data (e.g. 5-minute
data), to
> > > create a sub-hourly total (e.g. 15-minute data) using the -sum
option?
> > >
> > > If not, I can always code in my results using the -add option,
but I
> > wanted
> > > to ask first since the code is already in place.
> > >
> > > Thank you once again!
> > >
> > > Mike
> > >
> > > --
> > > Michael J. Erickson
> > >
> > > Research Scientist
> > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > NOAA/NWS/NCEP/Weather Prediction Center
> > > Phone: 301-683-1546
> > >
> > >
> >
> >
>
> --
> Michael J. Erickson
>
> Research Scientist
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> NOAA/NWS/NCEP/Weather Prediction Center
> Phone: 301-683-1546
>
>
------------------------------------------------
Subject: pcp_combine With Sub-hourly Data
From: Michael Erickson - NOAA Affiliate
Time: Thu Feb 21 11:30:29 2019
Thank you John and sorry for the delay. I haven't been able to work on
this
for a few days, and then I needed to convert the data to a format
readable
by MET.
I am still struggling with this so I apologize for asking a simple
question. I would like pcp_combine to sum a single 15 minute
precipitation
file (i.e. rename the file in a convoluted way). This may sound
strange,
but eventually I will sum greater intervals. The file I created is
here:
https://ftp.wpc.ncep.noaa.gov/erickson/dtc/A001500_s2018072019_e2018072019_vhr45.nc.
This is 15 minute precipitation between 19:30 and 19:45 on 20 July
2018.
Ignore the 'vhr' string on the file, it should technically be minute
rather
than hour.
My pcp_combine call is:
/opt/MET6/met6.0/bin/pcp_combine -sum 00000000_000000 001500
20180720_194500 001500 ./cow.nc -pcpdir ./
ERROR :
ERROR : sum_data_files() -> Cannot find a file with a valid time of
20180720_194500 and accumulation time of 001500 matching the regular
expression ".*"
ERROR :
I created this netcdf file (A001500_s2018072019_e2018072019_vhr45.nc)
from
a crude python program, so if there is an issue with the metadata, I
can
probably fix it. I am not sure though what the particular issue is. A
guess
may be with the variable name of 'A001500.'
Thank you again!
Mike
On Thu, Feb 14, 2019 at 10:25 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Mike,
>
> Thanks for posting the sample file. Yes, I agree, MET is not able
to
> sensibly read data from this NetCDF file. I've included the header
dump
> below. I see that it contains the timing and projection information
that
> MET needs, but it just doesn't follow any existing conventions that
MET
> knows.
>
> I see that you're using met-6.0 currently. If you're able to switch
to
> using met-8.0, this would be a great time to use the new Python
interface
> functionality. We could write a Python script to read this data,
parse the
> metadata, and hand it to MET in memory.
>
> We've posted several examples of this to the MET website here:
> https://dtcenter.org/met/users/downloads/analysis_scripts.php
>
> Is running met-8.0 an option for you?
>
> Thanks,
> John
>
> netcdf \20180720-181500 {
> dimensions:
> Lat = 3500 ;
> Lon = 7000 ;
> variables:
> float RadarOnly_QPE_01H(Lat, Lon) ;
> RadarOnly_QPE_01H:Units = "mm" ;
>
> // global attributes:
> :TypeName = "RadarOnly_QPE_01H" ;
> :DataType = "LatLonGrid" ;
> :Time = 1532110440 ;
> :FractionalTime = 0.f ;
> :MissingData = -99900.f ;
> :RangeFolded = -99901.f ;
> :Latitude = 55.f ;
> :Longitude = -130.f ;
> :Height = 0.f ;
> :LatGridSpacing = 0.01f ;
> :LonGridSpacing = 0.01f ;
> :attributes = "" ;
> }
>
> On Wed, Feb 13, 2019 at 12:18 PM Michael Erickson - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898 >
> >
> > Thank you again John!
> >
> > I think the issue here is related to the way the netCDF file is
created.
> > The data contains 15 minute NEWSe precipitation and was given to
me by
> the
> > developers.
> >
> > It looks like this netCDF file is not in a proper format to be
read in by
> > MET. I have tried:
> >
> > /opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf
test.ps
> > 'name="RadarOnly_QPE_01H"; level="(*,*)";'
> > /opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf
test.ps
> > 'name="RadarOnly_QPE_01H"; level="R1";'
> >
> > with the error:
> >
> > ERROR :
> > ERROR : plot_data_plane -> file "20180720-181500.netcdf" not a
valid
> data
> > file
> > ERROR :
> >
> > I have loaded the file here:
https://ftp.wpc.ncep.noaa.gov/erickson/dtc/
> >
> > If there is no way to read in this data with MET, I can probably
write
> some
> > code to create a workaround.
> >
> > Thanks!
> >
> > Mike
> >
> > On Wed, Feb 13, 2019 at 5:34 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Mike,
> > >
> > > Yes, it *should* be able to handle that. As a caveat, I haven't
done
> so
> > > for projects in recent memory. Below, I've listed the usage
statement
> > for
> > > pcp_combine and highlighted the *in_accum* and *out_accum*
options.
> They
> > > can both be formatted as HH[MMSS]... where the square brackets
mean
> > > optional. Let's say you want to accumulation 12 5-minute
intervals
> into
> > a
> > > 1-hour accumulation, you'd set:
> > > in_accum = 000500 (for 5 minutes)
> > > out_accum = 01 (for 1 hour... or 010000 works too)
> > >
> > > Please let me know if you find that it doesn't work as expected.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > * SUM_ARGS: init_time
> > > in_accum valid_time out_accum
> > > out_file [-pcpdir path] [-pcprx
> > > reg_exp] where "init_time" indicates the
> initialization
> > > time of the input data files in YYYYMMDD[_HH[MMSS]] format
> > > (required). "in_accum" indicates the
> accumulation
> > > interval of the input data files in HH[MMSS] format
> > > (required). "valid_time" indicates the
desired
> > valid
> > > time in YYYYMMDD[_HH[MMSS]] format (required).
> > > "out_accum" indicates the desired accumulation interval for the
output
> > > NetCDF file in HH[MMSS] format (required).
> > > "out_file" indicates the name of the output NetCDF file to be
written
> > > consisting of the sum of the accumulation intervals
> > > (required). "-pcpdir path" overrides the
default
> > > precipitation directory (.) (optional).
"-pcprx
> > > reg_exp" overrides the default regular expression for
precipitation
> file
> > > naming convention (.*) (optional). Note:
Set
> > > init_time to 00000000_000000 when summing observation files.*
> > >
> > >
> > >
> > > On Wed, Feb 13, 2019 at 8:18 AM Michael Erickson - NOAA
Affiliate via
> RT
> > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Feb 13 08:17:29 2019: Request 88898 was acted upon.
> > > > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > > > Queue: met_help
> > > > Subject: pcp_combine With Sub-hourly Data
> > > > Owner: Nobody
> > > > Requestors: michael.j.erickson at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898
> > >
> > > >
> > > >
> > > > Hello there,
> > > >
> > > > Can pcp_combine be used to sum sub-hourly data (e.g. 5-minute
data),
> to
> > > > create a sub-hourly total (e.g. 15-minute data) using the -sum
> option?
> > > >
> > > > If not, I can always code in my results using the -add option,
but I
> > > wanted
> > > > to ask first since the code is already in place.
> > > >
> > > > Thank you once again!
> > > >
> > > > Mike
> > > >
> > > > --
> > > > Michael J. Erickson
> > > >
> > > > Research Scientist
> > > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > > NOAA/NWS/NCEP/Weather Prediction Center
> > > > Phone: 301-683-1546
> > > >
> > > >
> > >
> > >
> >
> > --
> > Michael J. Erickson
> >
> > Research Scientist
> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > NOAA/NWS/NCEP/Weather Prediction Center
> > Phone: 301-683-1546
> >
> >
>
>
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/NCEP/Weather Prediction Center
Phone: 301-683-1546
------------------------------------------------
Subject: pcp_combine With Sub-hourly Data
From: John Halley Gotway
Time: Thu Feb 21 16:10:27 2019
Mike,
Thanks for sending the sample file. Here are some thoughts about
this:
(1) Missing data value.
One thing I noticed right off the bat is that the data contains a
value of
-99 presumably where there should be missing data. But the variable
attribute lists the fill value as -9999:
* double A001500(lat, lon) ; A001500:FillValue =
"-9999.f" ;*
You can fix this either by changing the -99 data values to -9999 or
just by
updating the FillValue attribute to correctly state -99. Either
option
should work to tell MET what's missing data.
(2) Init and Valid times.
MET reads the init_time_ut and valid_time_ut variable attributes to
determine the timing information. The weird thing is that the init
time *is
later than* the valid time:
Init time:
date -ud '1970-01-01 UTC '1532118600' seconds' +%Y%m%d_%H%M%S
*20180720_203000*
Valid time:
date -ud '1970-01-01 UTC '1532115899' seconds' +%Y%m%d_%H%M%S
*20180720_194459*
Presumably, that's a mistake. So for testing, I used the ncatted
utility
to reset:
init time = 20180720_193000 = 1532115000 (date -ud ''2018-07-20'
UTC
'19:30:00'' +%s)
valid time = 20180720_194500 = 1532115900 (date -ud ''2018-07-20' UTC
'19:45:00'' +%s)
*ncatted \*
* -a init_time_ut,A001500,o,c,"1532115000" \*
* -a valid_time_ut,A001500,o,c,"1532115900" \*
* A001500_s2018072019_e2018072019_vhr45.nc \*
* A001500_s2018072019_e2018072019_vhr45_MOD.nc*
(3) Running pcp_combine.
I used this sample data file to test the pcp_combine -add command:
*met-8.0/bin/pcp_combine \*
* -add A001500_s2018072019_e2018072019_vhr45_MOD.nc \*
* 'name="A001500"; level="(*,*)";' test_add.nc <http://test_add.nc>*
And that works fine. It reads the data from the input file and writes
it
to the output file. The -add command can be run on one or more input
files.
Next, I ran this pcp_combine -sum command using met-8.0:
*met-8.0/bin/pcp_combine \*
* -sum 20180720_193000 001500 20180720_194500 001500 test_sum.nc
<http://test_sum.nc> \*
* -field 'name="A001500"; level="(*,*)";'*
Notice a couple of things...
- I used "-field" to tell pcp_combine what data to search for.
- I had to specify the "init" time string to get it to find matching
timing info.
Since this MRMS data is really an observation, we don't want to have
to
specify an init time. So we'd like to use "00000000_000000" on the
command
line. I ran in the debugger and found why you had a problem. It
looks
like the logic for this when reading GRIB data is slightly different
in the
code than for reading NetCDF data.
So handling this well will require some minor code changes... but
changes
nonetheless.
It'd be good to work these wrinkles out prior to the next release,
met-8.1.
MRMS data is available in GRIB format. Is there a reason you're not
using
that?
Thanks,
John
On Thu, Feb 21, 2019 at 11:31 AM Michael Erickson - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898 >
>
> Thank you John and sorry for the delay. I haven't been able to work
on this
> for a few days, and then I needed to convert the data to a format
readable
> by MET.
>
> I am still struggling with this so I apologize for asking a simple
> question. I would like pcp_combine to sum a single 15 minute
precipitation
> file (i.e. rename the file in a convoluted way). This may sound
strange,
> but eventually I will sum greater intervals. The file I created is
here:
>
>
>
https://ftp.wpc.ncep.noaa.gov/erickson/dtc/A001500_s2018072019_e2018072019_vhr45.nc
> .
> This is 15 minute precipitation between 19:30 and 19:45 on 20 July
2018.
> Ignore the 'vhr' string on the file, it should technically be minute
rather
> than hour.
>
> My pcp_combine call is:
>
> /opt/MET6/met6.0/bin/pcp_combine -sum 00000000_000000 001500
> 20180720_194500 001500 ./cow.nc -pcpdir ./
>
> ERROR :
> ERROR : sum_data_files() -> Cannot find a file with a valid time of
> 20180720_194500 and accumulation time of 001500 matching the regular
> expression ".*"
> ERROR :
>
> I created this netcdf file
(A001500_s2018072019_e2018072019_vhr45.nc) from
> a crude python program, so if there is an issue with the metadata, I
can
> probably fix it. I am not sure though what the particular issue is.
A guess
> may be with the variable name of 'A001500.'
>
> Thank you again!
>
> Mike
>
>
> On Thu, Feb 14, 2019 at 10:25 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Mike,
> >
> > Thanks for posting the sample file. Yes, I agree, MET is not able
to
> > sensibly read data from this NetCDF file. I've included the
header dump
> > below. I see that it contains the timing and projection
information that
> > MET needs, but it just doesn't follow any existing conventions
that MET
> > knows.
> >
> > I see that you're using met-6.0 currently. If you're able to
switch to
> > using met-8.0, this would be a great time to use the new Python
interface
> > functionality. We could write a Python script to read this data,
parse
> the
> > metadata, and hand it to MET in memory.
> >
> > We've posted several examples of this to the MET website here:
> > https://dtcenter.org/met/users/downloads/analysis_scripts.php
> >
> > Is running met-8.0 an option for you?
> >
> > Thanks,
> > John
> >
> > netcdf \20180720-181500 {
> > dimensions:
> > Lat = 3500 ;
> > Lon = 7000 ;
> > variables:
> > float RadarOnly_QPE_01H(Lat, Lon) ;
> > RadarOnly_QPE_01H:Units = "mm" ;
> >
> > // global attributes:
> > :TypeName = "RadarOnly_QPE_01H" ;
> > :DataType = "LatLonGrid" ;
> > :Time = 1532110440 ;
> > :FractionalTime = 0.f ;
> > :MissingData = -99900.f ;
> > :RangeFolded = -99901.f ;
> > :Latitude = 55.f ;
> > :Longitude = -130.f ;
> > :Height = 0.f ;
> > :LatGridSpacing = 0.01f ;
> > :LonGridSpacing = 0.01f ;
> > :attributes = "" ;
> > }
> >
> > On Wed, Feb 13, 2019 at 12:18 PM Michael Erickson - NOAA Affiliate
via
> RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898 >
> > >
> > > Thank you again John!
> > >
> > > I think the issue here is related to the way the netCDF file is
> created.
> > > The data contains 15 minute NEWSe precipitation and was given to
me by
> > the
> > > developers.
> > >
> > > It looks like this netCDF file is not in a proper format to be
read in
> by
> > > MET. I have tried:
> > >
> > > /opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf
test.ps
> > > 'name="RadarOnly_QPE_01H"; level="(*,*)";'
> > > /opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf
test.ps
> > > 'name="RadarOnly_QPE_01H"; level="R1";'
> > >
> > > with the error:
> > >
> > > ERROR :
> > > ERROR : plot_data_plane -> file "20180720-181500.netcdf" not a
valid
> > data
> > > file
> > > ERROR :
> > >
> > > I have loaded the file here:
> https://ftp.wpc.ncep.noaa.gov/erickson/dtc/
> > >
> > > If there is no way to read in this data with MET, I can probably
write
> > some
> > > code to create a workaround.
> > >
> > > Thanks!
> > >
> > > Mike
> > >
> > > On Wed, Feb 13, 2019 at 5:34 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Mike,
> > > >
> > > > Yes, it *should* be able to handle that. As a caveat, I
haven't done
> > so
> > > > for projects in recent memory. Below, I've listed the usage
> statement
> > > for
> > > > pcp_combine and highlighted the *in_accum* and *out_accum*
options.
> > They
> > > > can both be formatted as HH[MMSS]... where the square brackets
mean
> > > > optional. Let's say you want to accumulation 12 5-minute
intervals
> > into
> > > a
> > > > 1-hour accumulation, you'd set:
> > > > in_accum = 000500 (for 5 minutes)
> > > > out_accum = 01 (for 1 hour... or 010000 works too)
> > > >
> > > > Please let me know if you find that it doesn't work as
expected.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > * SUM_ARGS: init_time
> > > > in_accum valid_time out_accum
> > > > out_file [-pcpdir path] [-pcprx
> > > > reg_exp] where "init_time" indicates the
> > initialization
> > > > time of the input data files in YYYYMMDD[_HH[MMSS]] format
> > > > (required). "in_accum" indicates the
> > accumulation
> > > > interval of the input data files in HH[MMSS] format
> > > > (required). "valid_time" indicates the
desired
> > > valid
> > > > time in YYYYMMDD[_HH[MMSS]] format (required).
> > > > "out_accum" indicates the desired accumulation interval for
the
> output
> > > > NetCDF file in HH[MMSS] format (required).
> > > > "out_file" indicates the name of the output NetCDF file to be
written
> > > > consisting of the sum of the accumulation intervals
> > > > (required). "-pcpdir path" overrides
the
> default
> > > > precipitation directory (.) (optional).
> "-pcprx
> > > > reg_exp" overrides the default regular expression for
precipitation
> > file
> > > > naming convention (.*) (optional).
Note: Set
> > > > init_time to 00000000_000000 when summing observation files.*
> > > >
> > > >
> > > >
> > > > On Wed, Feb 13, 2019 at 8:18 AM Michael Erickson - NOAA
Affiliate via
> > RT
> > > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Wed Feb 13 08:17:29 2019: Request 88898 was acted upon.
> > > > > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: pcp_combine With Sub-hourly Data
> > > > > Owner: Nobody
> > > > > Requestors: michael.j.erickson at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898
> > > >
> > > > >
> > > > >
> > > > > Hello there,
> > > > >
> > > > > Can pcp_combine be used to sum sub-hourly data (e.g. 5-
minute
> data),
> > to
> > > > > create a sub-hourly total (e.g. 15-minute data) using the
-sum
> > option?
> > > > >
> > > > > If not, I can always code in my results using the -add
option, but
> I
> > > > wanted
> > > > > to ask first since the code is already in place.
> > > > >
> > > > > Thank you once again!
> > > > >
> > > > > Mike
> > > > >
> > > > > --
> > > > > Michael J. Erickson
> > > > >
> > > > > Research Scientist
> > > > > Cooperative Institute for Research in Environmental Sciences
> (CIRES)
> > > > > NOAA/NWS/NCEP/Weather Prediction Center
> > > > > Phone: 301-683-1546
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > Michael J. Erickson
> > >
> > > Research Scientist
> > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > NOAA/NWS/NCEP/Weather Prediction Center
> > > Phone: 301-683-1546
> > >
> > >
> >
> >
>
> --
> Michael J. Erickson
>
> Research Scientist
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> NOAA/NWS/NCEP/Weather Prediction Center
> Phone: 301-683-1546
>
>
------------------------------------------------
Subject: pcp_combine With Sub-hourly Data
From: John Halley Gotway
Time: Thu Feb 21 16:51:29 2019
Mike,
Using the debugger, I figured out how the logic that's used when
searching
timing info in GRIB files. And I made a one-line change to MET's
vx_data2d_nc_met library to use that same logic. Just needed to add a
check for a bad data value:
*- ( vinfo.lead() && MetNc->lead_time() != vinfo.lead() ) )+
( !is_bad_data(vinfo.lead()) && MetNc->lead_time() != vinfo.lead() )
)*
I committed this to the trunk to be included in met-8.1. This one
line
change enables this pcp_combine command to work on your data:
*pcp_combine -sum 00000000_000000 001500 20180720_194500 001500 \*
* test_sum.nc <http://test_sum.nc> -field 'name="A001500";
level="(*,*)";'*
Thanks for finding this issue!
John
On Thu, Feb 21, 2019 at 4:10 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Mike,
>
> Thanks for sending the sample file. Here are some thoughts about
this:
>
> (1) Missing data value.
>
> One thing I noticed right off the bat is that the data contains a
value of
> -99 presumably where there should be missing data. But the variable
> attribute lists the fill value as -9999:
>
> * double A001500(lat, lon) ; A001500:FillValue
=
> "-9999.f" ;*
> You can fix this either by changing the -99 data values to -9999 or
just
> by updating the FillValue attribute to correctly state -99. Either
option
> should work to tell MET what's missing data.
>
> (2) Init and Valid times.
>
> MET reads the init_time_ut and valid_time_ut variable attributes to
> determine the timing information. The weird thing is that the init
time *is
> later than* the valid time:
>
> Init time:
> date -ud '1970-01-01 UTC '1532118600' seconds' +%Y%m%d_%H%M%S
> *20180720_203000*
>
> Valid time:
> date -ud '1970-01-01 UTC '1532115899' seconds' +%Y%m%d_%H%M%S
> *20180720_194459*
>
> Presumably, that's a mistake. So for testing, I used the ncatted
utility
> to reset:
> init time = 20180720_193000 = 1532115000 (date -ud ''2018-07-20'
UTC
> '19:30:00'' +%s)
> valid time = 20180720_194500 = 1532115900 (date -ud ''2018-07-20'
UTC
> '19:45:00'' +%s)
>
> *ncatted \*
> * -a init_time_ut,A001500,o,c,"1532115000" \*
> * -a valid_time_ut,A001500,o,c,"1532115900" \*
> * A001500_s2018072019_e2018072019_vhr45.nc \*
> * A001500_s2018072019_e2018072019_vhr45_MOD.nc*
>
> (3) Running pcp_combine.
>
> I used this sample data file to test the pcp_combine -add command:
> *met-8.0/bin/pcp_combine \*
> * -add A001500_s2018072019_e2018072019_vhr45_MOD.nc \*
> * 'name="A001500"; level="(*,*)";' test_add.nc
<http://test_add.nc>*
>
> And that works fine. It reads the data from the input file and
writes it
> to the output file. The -add command can be run on one or more
input files.
>
> Next, I ran this pcp_combine -sum command using met-8.0:
> *met-8.0/bin/pcp_combine \*
> * -sum 20180720_193000 001500 20180720_194500 001500 test_sum.nc
> <http://test_sum.nc> \*
> * -field 'name="A001500"; level="(*,*)";'*
>
> Notice a couple of things...
> - I used "-field" to tell pcp_combine what data to search for.
> - I had to specify the "init" time string to get it to find
matching
> timing info.
>
> Since this MRMS data is really an observation, we don't want to have
to
> specify an init time. So we'd like to use "00000000_000000" on the
command
> line. I ran in the debugger and found why you had a problem. It
looks
> like the logic for this when reading GRIB data is slightly different
in the
> code than for reading NetCDF data.
>
> So handling this well will require some minor code changes... but
changes
> nonetheless.
>
> It'd be good to work these wrinkles out prior to the next release,
met-8.1.
>
> MRMS data is available in GRIB format. Is there a reason you're not
using
> that?
>
> Thanks,
> John
>
>
>
>
>
> On Thu, Feb 21, 2019 at 11:31 AM Michael Erickson - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898 >
>>
>> Thank you John and sorry for the delay. I haven't been able to work
on
>> this
>> for a few days, and then I needed to convert the data to a format
readable
>> by MET.
>>
>> I am still struggling with this so I apologize for asking a simple
>> question. I would like pcp_combine to sum a single 15 minute
precipitation
>> file (i.e. rename the file in a convoluted way). This may sound
strange,
>> but eventually I will sum greater intervals. The file I created is
here:
>>
>>
>>
https://ftp.wpc.ncep.noaa.gov/erickson/dtc/A001500_s2018072019_e2018072019_vhr45.nc
>> .
>> This is 15 minute precipitation between 19:30 and 19:45 on 20 July
2018.
>> Ignore the 'vhr' string on the file, it should technically be
minute
>> rather
>> than hour.
>>
>> My pcp_combine call is:
>>
>> /opt/MET6/met6.0/bin/pcp_combine -sum 00000000_000000 001500
>> 20180720_194500 001500 ./cow.nc -pcpdir ./
>>
>> ERROR :
>> ERROR : sum_data_files() -> Cannot find a file with a valid time
of
>> 20180720_194500 and accumulation time of 001500 matching the
regular
>> expression ".*"
>> ERROR :
>>
>> I created this netcdf file
(A001500_s2018072019_e2018072019_vhr45.nc)
>> from
>> a crude python program, so if there is an issue with the metadata,
I can
>> probably fix it. I am not sure though what the particular issue is.
A
>> guess
>> may be with the variable name of 'A001500.'
>>
>> Thank you again!
>>
>> Mike
>>
>>
>> On Thu, Feb 14, 2019 at 10:25 PM John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>> > Mike,
>> >
>> > Thanks for posting the sample file. Yes, I agree, MET is not
able to
>> > sensibly read data from this NetCDF file. I've included the
header dump
>> > below. I see that it contains the timing and projection
information
>> that
>> > MET needs, but it just doesn't follow any existing conventions
that MET
>> > knows.
>> >
>> > I see that you're using met-6.0 currently. If you're able to
switch to
>> > using met-8.0, this would be a great time to use the new Python
>> interface
>> > functionality. We could write a Python script to read this data,
parse
>> the
>> > metadata, and hand it to MET in memory.
>> >
>> > We've posted several examples of this to the MET website here:
>> > https://dtcenter.org/met/users/downloads/analysis_scripts.php
>> >
>> > Is running met-8.0 an option for you?
>> >
>> > Thanks,
>> > John
>> >
>> > netcdf \20180720-181500 {
>> > dimensions:
>> > Lat = 3500 ;
>> > Lon = 7000 ;
>> > variables:
>> > float RadarOnly_QPE_01H(Lat, Lon) ;
>> > RadarOnly_QPE_01H:Units = "mm" ;
>> >
>> > // global attributes:
>> > :TypeName = "RadarOnly_QPE_01H" ;
>> > :DataType = "LatLonGrid" ;
>> > :Time = 1532110440 ;
>> > :FractionalTime = 0.f ;
>> > :MissingData = -99900.f ;
>> > :RangeFolded = -99901.f ;
>> > :Latitude = 55.f ;
>> > :Longitude = -130.f ;
>> > :Height = 0.f ;
>> > :LatGridSpacing = 0.01f ;
>> > :LonGridSpacing = 0.01f ;
>> > :attributes = "" ;
>> > }
>> >
>> > On Wed, Feb 13, 2019 at 12:18 PM Michael Erickson - NOAA
Affiliate via
>> RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898 >
>> > >
>> > > Thank you again John!
>> > >
>> > > I think the issue here is related to the way the netCDF file is
>> created.
>> > > The data contains 15 minute NEWSe precipitation and was given
to me by
>> > the
>> > > developers.
>> > >
>> > > It looks like this netCDF file is not in a proper format to be
read
>> in by
>> > > MET. I have tried:
>> > >
>> > > /opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf
test.ps
>> > > 'name="RadarOnly_QPE_01H"; level="(*,*)";'
>> > > /opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf
test.ps
>> > > 'name="RadarOnly_QPE_01H"; level="R1";'
>> > >
>> > > with the error:
>> > >
>> > > ERROR :
>> > > ERROR : plot_data_plane -> file "20180720-181500.netcdf" not a
valid
>> > data
>> > > file
>> > > ERROR :
>> > >
>> > > I have loaded the file here:
>> https://ftp.wpc.ncep.noaa.gov/erickson/dtc/
>> > >
>> > > If there is no way to read in this data with MET, I can
probably write
>> > some
>> > > code to create a workaround.
>> > >
>> > > Thanks!
>> > >
>> > > Mike
>> > >
>> > > On Wed, Feb 13, 2019 at 5:34 PM John Halley Gotway via RT <
>> > > met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > > Mike,
>> > > >
>> > > > Yes, it *should* be able to handle that. As a caveat, I
haven't
>> done
>> > so
>> > > > for projects in recent memory. Below, I've listed the usage
>> statement
>> > > for
>> > > > pcp_combine and highlighted the *in_accum* and *out_accum*
options.
>> > They
>> > > > can both be formatted as HH[MMSS]... where the square
brackets mean
>> > > > optional. Let's say you want to accumulation 12 5-minute
intervals
>> > into
>> > > a
>> > > > 1-hour accumulation, you'd set:
>> > > > in_accum = 000500 (for 5 minutes)
>> > > > out_accum = 01 (for 1 hour... or 010000 works too)
>> > > >
>> > > > Please let me know if you find that it doesn't work as
expected.
>> > > >
>> > > > Thanks,
>> > > > John
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > * SUM_ARGS: init_time
>> > > > in_accum valid_time out_accum
>> > > > out_file [-pcpdir path] [-pcprx
>> > > > reg_exp] where "init_time" indicates the
>> > initialization
>> > > > time of the input data files in YYYYMMDD[_HH[MMSS]] format
>> > > > (required). "in_accum" indicates the
>> > accumulation
>> > > > interval of the input data files in HH[MMSS] format
>> > > > (required). "valid_time" indicates the
>> desired
>> > > valid
>> > > > time in YYYYMMDD[_HH[MMSS]] format (required).
>> > > > "out_accum" indicates the desired accumulation interval for
the
>> output
>> > > > NetCDF file in HH[MMSS] format (required).
>> > > > "out_file" indicates the name of the output NetCDF file to be
>> written
>> > > > consisting of the sum of the accumulation intervals
>> > > > (required). "-pcpdir path" overrides
the
>> default
>> > > > precipitation directory (.) (optional).
>> "-pcprx
>> > > > reg_exp" overrides the default regular expression for
precipitation
>> > file
>> > > > naming convention (.*) (optional).
Note: Set
>> > > > init_time to 00000000_000000 when summing observation files.*
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Feb 13, 2019 at 8:18 AM Michael Erickson - NOAA
Affiliate
>> via
>> > RT
>> > > <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > Wed Feb 13 08:17:29 2019: Request 88898 was acted upon.
>> > > > > Transaction: Ticket created by michael.j.erickson at noaa.gov
>> > > > > Queue: met_help
>> > > > > Subject: pcp_combine With Sub-hourly Data
>> > > > > Owner: Nobody
>> > > > > Requestors: michael.j.erickson at noaa.gov
>> > > > > Status: new
>> > > > > Ticket <URL:
>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898
>> > > >
>> > > > >
>> > > > >
>> > > > > Hello there,
>> > > > >
>> > > > > Can pcp_combine be used to sum sub-hourly data (e.g. 5-
minute
>> data),
>> > to
>> > > > > create a sub-hourly total (e.g. 15-minute data) using the
-sum
>> > option?
>> > > > >
>> > > > > If not, I can always code in my results using the -add
option,
>> but I
>> > > > wanted
>> > > > > to ask first since the code is already in place.
>> > > > >
>> > > > > Thank you once again!
>> > > > >
>> > > > > Mike
>> > > > >
>> > > > > --
>> > > > > Michael J. Erickson
>> > > > >
>> > > > > Research Scientist
>> > > > > Cooperative Institute for Research in Environmental
Sciences
>> (CIRES)
>> > > > > NOAA/NWS/NCEP/Weather Prediction Center
>> > > > > Phone: 301-683-1546
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > > --
>> > > Michael J. Erickson
>> > >
>> > > Research Scientist
>> > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
>> > > NOAA/NWS/NCEP/Weather Prediction Center
>> > > Phone: 301-683-1546
>> > >
>> > >
>> >
>> >
>>
>> --
>> Michael J. Erickson
>>
>> Research Scientist
>> Cooperative Institute for Research in Environmental Sciences
(CIRES)
>> NOAA/NWS/NCEP/Weather Prediction Center
>> Phone: 301-683-1546
>>
>>
------------------------------------------------
Subject: pcp_combine With Sub-hourly Data
From: Michael Erickson - NOAA Affiliate
Time: Fri Feb 22 06:35:07 2019
Hi John,
Thanks for all of your help with this! I'm not using the grib2 MRMS in
this
case since this is older historical data provided to me by NSSL. I
then
modified that netCDF file to contain the appropriate metadata so MET
can
read it, hence that strange error with the init/valid dates. I have
fixed
that metadata issue within the netCDF file and I have been able to use
pcp_combine with the -add command. Thank you! This year I will pull
MRMS
grib2 data live, and this will hopefully result in a simpler series of
scripts.
Glad I could accidently help with the debugging of MET!
Mike
On Thu, Feb 21, 2019 at 11:51 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Mike,
>
> Using the debugger, I figured out how the logic that's used when
searching
> timing info in GRIB files. And I made a one-line change to MET's
> vx_data2d_nc_met library to use that same logic. Just needed to add
a
> check for a bad data value:
>
>
> *- ( vinfo.lead() && MetNc->lead_time() != vinfo.lead() ) )+
> ( !is_bad_data(vinfo.lead()) && MetNc->lead_time() != vinfo.lead()
) )*
>
> I committed this to the trunk to be included in met-8.1. This one
line
> change enables this pcp_combine command to work on your data:
>
> *pcp_combine -sum 00000000_000000 001500 20180720_194500 001500 \*
> * test_sum.nc <http://test_sum.nc> -field 'name="A001500";
> level="(*,*)";'*
>
> Thanks for finding this issue!
>
> John
>
> On Thu, Feb 21, 2019 at 4:10 PM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Mike,
> >
> > Thanks for sending the sample file. Here are some thoughts about
this:
> >
> > (1) Missing data value.
> >
> > One thing I noticed right off the bat is that the data contains a
value
> of
> > -99 presumably where there should be missing data. But the
variable
> > attribute lists the fill value as -9999:
> >
> > * double A001500(lat, lon) ;
A001500:FillValue =
> > "-9999.f" ;*
> > You can fix this either by changing the -99 data values to -9999
or just
> > by updating the FillValue attribute to correctly state -99.
Either
> option
> > should work to tell MET what's missing data.
> >
> > (2) Init and Valid times.
> >
> > MET reads the init_time_ut and valid_time_ut variable attributes
to
> > determine the timing information. The weird thing is that the
init time
> *is
> > later than* the valid time:
> >
> > Init time:
> > date -ud '1970-01-01 UTC '1532118600' seconds' +%Y%m%d_%H%M%S
> > *20180720_203000*
> >
> > Valid time:
> > date -ud '1970-01-01 UTC '1532115899' seconds' +%Y%m%d_%H%M%S
> > *20180720_194459*
> >
> > Presumably, that's a mistake. So for testing, I used the ncatted
utility
> > to reset:
> > init time = 20180720_193000 = 1532115000 (date -ud ''2018-07-
20' UTC
> > '19:30:00'' +%s)
> > valid time = 20180720_194500 = 1532115900 (date -ud ''2018-07-20'
UTC
> > '19:45:00'' +%s)
> >
> > *ncatted \*
> > * -a init_time_ut,A001500,o,c,"1532115000" \*
> > * -a valid_time_ut,A001500,o,c,"1532115900" \*
> > * A001500_s2018072019_e2018072019_vhr45.nc \*
> > * A001500_s2018072019_e2018072019_vhr45_MOD.nc*
> >
> > (3) Running pcp_combine.
> >
> > I used this sample data file to test the pcp_combine -add command:
> > *met-8.0/bin/pcp_combine \*
> > * -add A001500_s2018072019_e2018072019_vhr45_MOD.nc \*
> > * 'name="A001500"; level="(*,*)";' test_add.nc
<http://test_add.nc>*
> >
> > And that works fine. It reads the data from the input file and
writes it
> > to the output file. The -add command can be run on one or more
input
> files.
> >
> > Next, I ran this pcp_combine -sum command using met-8.0:
> > *met-8.0/bin/pcp_combine \*
> > * -sum 20180720_193000 001500 20180720_194500 001500 test_sum.nc
> > <http://test_sum.nc> \*
> > * -field 'name="A001500"; level="(*,*)";'*
> >
> > Notice a couple of things...
> > - I used "-field" to tell pcp_combine what data to search for.
> > - I had to specify the "init" time string to get it to find
matching
> > timing info.
> >
> > Since this MRMS data is really an observation, we don't want to
have to
> > specify an init time. So we'd like to use "00000000_000000" on
the
> command
> > line. I ran in the debugger and found why you had a problem. It
looks
> > like the logic for this when reading GRIB data is slightly
different in
> the
> > code than for reading NetCDF data.
> >
> > So handling this well will require some minor code changes... but
changes
> > nonetheless.
> >
> > It'd be good to work these wrinkles out prior to the next release,
> met-8.1.
> >
> > MRMS data is available in GRIB format. Is there a reason you're
not
> using
> > that?
> >
> > Thanks,
> > John
> >
> >
> >
> >
> >
> > On Thu, Feb 21, 2019 at 11:31 AM Michael Erickson - NOAA Affiliate
via
> RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898 >
> >>
> >> Thank you John and sorry for the delay. I haven't been able to
work on
> >> this
> >> for a few days, and then I needed to convert the data to a format
> readable
> >> by MET.
> >>
> >> I am still struggling with this so I apologize for asking a
simple
> >> question. I would like pcp_combine to sum a single 15 minute
> precipitation
> >> file (i.e. rename the file in a convoluted way). This may sound
strange,
> >> but eventually I will sum greater intervals. The file I created
is here:
> >>
> >>
> >>
>
https://ftp.wpc.ncep.noaa.gov/erickson/dtc/A001500_s2018072019_e2018072019_vhr45.nc
> >> .
> >> This is 15 minute precipitation between 19:30 and 19:45 on 20
July 2018.
> >> Ignore the 'vhr' string on the file, it should technically be
minute
> >> rather
> >> than hour.
> >>
> >> My pcp_combine call is:
> >>
> >> /opt/MET6/met6.0/bin/pcp_combine -sum 00000000_000000 001500
> >> 20180720_194500 001500 ./cow.nc -pcpdir ./
> >>
> >> ERROR :
> >> ERROR : sum_data_files() -> Cannot find a file with a valid time
of
> >> 20180720_194500 and accumulation time of 001500 matching the
regular
> >> expression ".*"
> >> ERROR :
> >>
> >> I created this netcdf file
(A001500_s2018072019_e2018072019_vhr45.nc)
> >> from
> >> a crude python program, so if there is an issue with the
metadata, I can
> >> probably fix it. I am not sure though what the particular issue
is. A
> >> guess
> >> may be with the variable name of 'A001500.'
> >>
> >> Thank you again!
> >>
> >> Mike
> >>
> >>
> >> On Thu, Feb 14, 2019 at 10:25 PM John Halley Gotway via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> > Mike,
> >> >
> >> > Thanks for posting the sample file. Yes, I agree, MET is not
able to
> >> > sensibly read data from this NetCDF file. I've included the
header
> dump
> >> > below. I see that it contains the timing and projection
information
> >> that
> >> > MET needs, but it just doesn't follow any existing conventions
that
> MET
> >> > knows.
> >> >
> >> > I see that you're using met-6.0 currently. If you're able to
switch
> to
> >> > using met-8.0, this would be a great time to use the new Python
> >> interface
> >> > functionality. We could write a Python script to read this
data,
> parse
> >> the
> >> > metadata, and hand it to MET in memory.
> >> >
> >> > We've posted several examples of this to the MET website here:
> >> >
https://dtcenter.org/met/users/downloads/analysis_scripts.php
> >> >
> >> > Is running met-8.0 an option for you?
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> > netcdf \20180720-181500 {
> >> > dimensions:
> >> > Lat = 3500 ;
> >> > Lon = 7000 ;
> >> > variables:
> >> > float RadarOnly_QPE_01H(Lat, Lon) ;
> >> > RadarOnly_QPE_01H:Units = "mm" ;
> >> >
> >> > // global attributes:
> >> > :TypeName = "RadarOnly_QPE_01H" ;
> >> > :DataType = "LatLonGrid" ;
> >> > :Time = 1532110440 ;
> >> > :FractionalTime = 0.f ;
> >> > :MissingData = -99900.f ;
> >> > :RangeFolded = -99901.f ;
> >> > :Latitude = 55.f ;
> >> > :Longitude = -130.f ;
> >> > :Height = 0.f ;
> >> > :LatGridSpacing = 0.01f ;
> >> > :LonGridSpacing = 0.01f ;
> >> > :attributes = "" ;
> >> > }
> >> >
> >> > On Wed, Feb 13, 2019 at 12:18 PM Michael Erickson - NOAA
Affiliate via
> >> RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898
>
> >> > >
> >> > > Thank you again John!
> >> > >
> >> > > I think the issue here is related to the way the netCDF file
is
> >> created.
> >> > > The data contains 15 minute NEWSe precipitation and was given
to me
> by
> >> > the
> >> > > developers.
> >> > >
> >> > > It looks like this netCDF file is not in a proper format to
be read
> >> in by
> >> > > MET. I have tried:
> >> > >
> >> > > /opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf
test.ps
> >> > > 'name="RadarOnly_QPE_01H"; level="(*,*)";'
> >> > > /opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf
test.ps
> >> > > 'name="RadarOnly_QPE_01H"; level="R1";'
> >> > >
> >> > > with the error:
> >> > >
> >> > > ERROR :
> >> > > ERROR : plot_data_plane -> file "20180720-181500.netcdf" not
a
> valid
> >> > data
> >> > > file
> >> > > ERROR :
> >> > >
> >> > > I have loaded the file here:
> >> https://ftp.wpc.ncep.noaa.gov/erickson/dtc/
> >> > >
> >> > > If there is no way to read in this data with MET, I can
probably
> write
> >> > some
> >> > > code to create a workaround.
> >> > >
> >> > > Thanks!
> >> > >
> >> > > Mike
> >> > >
> >> > > On Wed, Feb 13, 2019 at 5:34 PM John Halley Gotway via RT <
> >> > > met_help at ucar.edu>
> >> > > wrote:
> >> > >
> >> > > > Mike,
> >> > > >
> >> > > > Yes, it *should* be able to handle that. As a caveat, I
haven't
> >> done
> >> > so
> >> > > > for projects in recent memory. Below, I've listed the
usage
> >> statement
> >> > > for
> >> > > > pcp_combine and highlighted the *in_accum* and *out_accum*
> options.
> >> > They
> >> > > > can both be formatted as HH[MMSS]... where the square
brackets
> mean
> >> > > > optional. Let's say you want to accumulation 12 5-minute
> intervals
> >> > into
> >> > > a
> >> > > > 1-hour accumulation, you'd set:
> >> > > > in_accum = 000500 (for 5 minutes)
> >> > > > out_accum = 01 (for 1 hour... or 010000 works too)
> >> > > >
> >> > > > Please let me know if you find that it doesn't work as
expected.
> >> > > >
> >> > > > Thanks,
> >> > > > John
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > * SUM_ARGS: init_time
> >> > > > in_accum valid_time out_accum
> >> > > > out_file [-pcpdir path] [-
pcprx
> >> > > > reg_exp] where "init_time" indicates the
> >> > initialization
> >> > > > time of the input data files in YYYYMMDD[_HH[MMSS]] format
> >> > > > (required). "in_accum" indicates the
> >> > accumulation
> >> > > > interval of the input data files in HH[MMSS] format
> >> > > > (required). "valid_time" indicates
the
> >> desired
> >> > > valid
> >> > > > time in YYYYMMDD[_HH[MMSS]] format (required).
> >> > > > "out_accum" indicates the desired accumulation interval for
the
> >> output
> >> > > > NetCDF file in HH[MMSS] format (required).
> >> > > > "out_file" indicates the name of the output NetCDF file to
be
> >> written
> >> > > > consisting of the sum of the accumulation intervals
> >> > > > (required). "-pcpdir path" overrides
the
> >> default
> >> > > > precipitation directory (.) (optional).
> >> "-pcprx
> >> > > > reg_exp" overrides the default regular expression for
> precipitation
> >> > file
> >> > > > naming convention (.*) (optional).
Note:
> Set
> >> > > > init_time to 00000000_000000 when summing observation
files.*
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Wed, Feb 13, 2019 at 8:18 AM Michael Erickson - NOAA
Affiliate
> >> via
> >> > RT
> >> > > <
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > >
> >> > > > > Wed Feb 13 08:17:29 2019: Request 88898 was acted upon.
> >> > > > > Transaction: Ticket created by
michael.j.erickson at noaa.gov
> >> > > > > Queue: met_help
> >> > > > > Subject: pcp_combine With Sub-hourly Data
> >> > > > > Owner: Nobody
> >> > > > > Requestors: michael.j.erickson at noaa.gov
> >> > > > > Status: new
> >> > > > > Ticket <URL:
> >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898
> >> > > >
> >> > > > >
> >> > > > >
> >> > > > > Hello there,
> >> > > > >
> >> > > > > Can pcp_combine be used to sum sub-hourly data (e.g. 5-
minute
> >> data),
> >> > to
> >> > > > > create a sub-hourly total (e.g. 15-minute data) using the
-sum
> >> > option?
> >> > > > >
> >> > > > > If not, I can always code in my results using the -add
option,
> >> but I
> >> > > > wanted
> >> > > > > to ask first since the code is already in place.
> >> > > > >
> >> > > > > Thank you once again!
> >> > > > >
> >> > > > > Mike
> >> > > > >
> >> > > > > --
> >> > > > > Michael J. Erickson
> >> > > > >
> >> > > > > Research Scientist
> >> > > > > Cooperative Institute for Research in Environmental
Sciences
> >> (CIRES)
> >> > > > > NOAA/NWS/NCEP/Weather Prediction Center
> >> > > > > Phone: 301-683-1546
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > > --
> >> > > Michael J. Erickson
> >> > >
> >> > > Research Scientist
> >> > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> >> > > NOAA/NWS/NCEP/Weather Prediction Center
> >> > > Phone: 301-683-1546
> >> > >
> >> > >
> >> >
> >> >
> >>
> >> --
> >> Michael J. Erickson
> >>
> >> Research Scientist
> >> Cooperative Institute for Research in Environmental Sciences
(CIRES)
> >> NOAA/NWS/NCEP/Weather Prediction Center
> >> Phone: 301-683-1546
> >>
> >>
>
>
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/NCEP/Weather Prediction Center
Phone: 301-683-1546
------------------------------------------------
Subject: pcp_combine With Sub-hourly Data
From: John Halley Gotway
Time: Fri Feb 22 09:38:50 2019
Mike,
Great, glad you're all set. Since you already have the script to
reformat
the data, going that route is fine.
But I wanted to suggest that if you run into a similar situation in
the
future, you could consider writing a Python script to read the data
into
memory and then hand it off to MET. That avoids the step of having to
write the reformatted version of the file to disk. Practically
speaking,
the Python script wouldn't be much different from what you've already
written. It still needs to load up data and define metadata in a
particular way. But if the input dataset is very large or contains
many
variables to be processed, it may be impractical or inconvenient to
reformat all of that data ahead of time. In that case, you may find
using
a Python script with met-8.0 or later to be useful.
We've posted several examples of doing so to the MET website:
https://dtcenter.org/met/users/downloads/analysis_scripts.php
Just another option to consider. I'll go ahead and resolve this
issue.
Thanks,
John
On Fri, Feb 22, 2019 at 6:35 AM Michael Erickson - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898 >
>
> Hi John,
>
> Thanks for all of your help with this! I'm not using the grib2 MRMS
in this
> case since this is older historical data provided to me by NSSL. I
then
> modified that netCDF file to contain the appropriate metadata so MET
can
> read it, hence that strange error with the init/valid dates. I have
fixed
> that metadata issue within the netCDF file and I have been able to
use
> pcp_combine with the -add command. Thank you! This year I will pull
MRMS
> grib2 data live, and this will hopefully result in a simpler series
of
> scripts.
>
> Glad I could accidently help with the debugging of MET!
>
> Mike
>
>
>
> On Thu, Feb 21, 2019 at 11:51 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Mike,
> >
> > Using the debugger, I figured out how the logic that's used when
> searching
> > timing info in GRIB files. And I made a one-line change to MET's
> > vx_data2d_nc_met library to use that same logic. Just needed to
add a
> > check for a bad data value:
> >
> >
> > *- ( vinfo.lead() && MetNc->lead_time() != vinfo.lead() )
)+
> > ( !is_bad_data(vinfo.lead()) && MetNc->lead_time() != vinfo.lead()
) )*
> >
> > I committed this to the trunk to be included in met-8.1. This one
line
> > change enables this pcp_combine command to work on your data:
> >
> > *pcp_combine -sum 00000000_000000 001500 20180720_194500 001500 \*
> > * test_sum.nc <http://test_sum.nc> -field 'name="A001500";
> > level="(*,*)";'*
> >
> > Thanks for finding this issue!
> >
> > John
> >
> > On Thu, Feb 21, 2019 at 4:10 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Mike,
> > >
> > > Thanks for sending the sample file. Here are some thoughts
about this:
> > >
> > > (1) Missing data value.
> > >
> > > One thing I noticed right off the bat is that the data contains
a value
> > of
> > > -99 presumably where there should be missing data. But the
variable
> > > attribute lists the fill value as -9999:
> > >
> > > * double A001500(lat, lon) ;
A001500:FillValue =
> > > "-9999.f" ;*
> > > You can fix this either by changing the -99 data values to -9999
or
> just
> > > by updating the FillValue attribute to correctly state -99.
Either
> > option
> > > should work to tell MET what's missing data.
> > >
> > > (2) Init and Valid times.
> > >
> > > MET reads the init_time_ut and valid_time_ut variable attributes
to
> > > determine the timing information. The weird thing is that the
init
> time
> > *is
> > > later than* the valid time:
> > >
> > > Init time:
> > > date -ud '1970-01-01 UTC '1532118600' seconds' +%Y%m%d_%H%M%S
> > > *20180720_203000*
> > >
> > > Valid time:
> > > date -ud '1970-01-01 UTC '1532115899' seconds' +%Y%m%d_%H%M%S
> > > *20180720_194459*
> > >
> > > Presumably, that's a mistake. So for testing, I used the
ncatted
> utility
> > > to reset:
> > > init time = 20180720_193000 = 1532115000 (date -ud ''2018-07-
20' UTC
> > > '19:30:00'' +%s)
> > > valid time = 20180720_194500 = 1532115900 (date -ud ''2018-07-
20' UTC
> > > '19:45:00'' +%s)
> > >
> > > *ncatted \*
> > > * -a init_time_ut,A001500,o,c,"1532115000" \*
> > > * -a valid_time_ut,A001500,o,c,"1532115900" \*
> > > * A001500_s2018072019_e2018072019_vhr45.nc \*
> > > * A001500_s2018072019_e2018072019_vhr45_MOD.nc*
> > >
> > > (3) Running pcp_combine.
> > >
> > > I used this sample data file to test the pcp_combine -add
command:
> > > *met-8.0/bin/pcp_combine \*
> > > * -add A001500_s2018072019_e2018072019_vhr45_MOD.nc \*
> > > * 'name="A001500"; level="(*,*)";' test_add.nc
<http://test_add.nc>*
> > >
> > > And that works fine. It reads the data from the input file and
writes
> it
> > > to the output file. The -add command can be run on one or more
input
> > files.
> > >
> > > Next, I ran this pcp_combine -sum command using met-8.0:
> > > *met-8.0/bin/pcp_combine \*
> > > * -sum 20180720_193000 001500 20180720_194500 001500
test_sum.nc
> > > <http://test_sum.nc> \*
> > > * -field 'name="A001500"; level="(*,*)";'*
> > >
> > > Notice a couple of things...
> > > - I used "-field" to tell pcp_combine what data to search for.
> > > - I had to specify the "init" time string to get it to find
matching
> > > timing info.
> > >
> > > Since this MRMS data is really an observation, we don't want to
have to
> > > specify an init time. So we'd like to use "00000000_000000" on
the
> > command
> > > line. I ran in the debugger and found why you had a problem.
It looks
> > > like the logic for this when reading GRIB data is slightly
different in
> > the
> > > code than for reading NetCDF data.
> > >
> > > So handling this well will require some minor code changes...
but
> changes
> > > nonetheless.
> > >
> > > It'd be good to work these wrinkles out prior to the next
release,
> > met-8.1.
> > >
> > > MRMS data is available in GRIB format. Is there a reason you're
not
> > using
> > > that?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Feb 21, 2019 at 11:31 AM Michael Erickson - NOAA
Affiliate via
> > RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898 >
> > >>
> > >> Thank you John and sorry for the delay. I haven't been able to
work on
> > >> this
> > >> for a few days, and then I needed to convert the data to a
format
> > readable
> > >> by MET.
> > >>
> > >> I am still struggling with this so I apologize for asking a
simple
> > >> question. I would like pcp_combine to sum a single 15 minute
> > precipitation
> > >> file (i.e. rename the file in a convoluted way). This may sound
> strange,
> > >> but eventually I will sum greater intervals. The file I created
is
> here:
> > >>
> > >>
> > >>
> >
>
https://ftp.wpc.ncep.noaa.gov/erickson/dtc/A001500_s2018072019_e2018072019_vhr45.nc
> > >> .
> > >> This is 15 minute precipitation between 19:30 and 19:45 on 20
July
> 2018.
> > >> Ignore the 'vhr' string on the file, it should technically be
minute
> > >> rather
> > >> than hour.
> > >>
> > >> My pcp_combine call is:
> > >>
> > >> /opt/MET6/met6.0/bin/pcp_combine -sum 00000000_000000 001500
> > >> 20180720_194500 001500 ./cow.nc -pcpdir ./
> > >>
> > >> ERROR :
> > >> ERROR : sum_data_files() -> Cannot find a file with a valid
time of
> > >> 20180720_194500 and accumulation time of 001500 matching the
regular
> > >> expression ".*"
> > >> ERROR :
> > >>
> > >> I created this netcdf file
(A001500_s2018072019_e2018072019_vhr45.nc)
> > >> from
> > >> a crude python program, so if there is an issue with the
metadata, I
> can
> > >> probably fix it. I am not sure though what the particular issue
is. A
> > >> guess
> > >> may be with the variable name of 'A001500.'
> > >>
> > >> Thank you again!
> > >>
> > >> Mike
> > >>
> > >>
> > >> On Thu, Feb 14, 2019 at 10:25 PM John Halley Gotway via RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> > Mike,
> > >> >
> > >> > Thanks for posting the sample file. Yes, I agree, MET is not
able
> to
> > >> > sensibly read data from this NetCDF file. I've included the
header
> > dump
> > >> > below. I see that it contains the timing and projection
information
> > >> that
> > >> > MET needs, but it just doesn't follow any existing
conventions that
> > MET
> > >> > knows.
> > >> >
> > >> > I see that you're using met-6.0 currently. If you're able to
switch
> > to
> > >> > using met-8.0, this would be a great time to use the new
Python
> > >> interface
> > >> > functionality. We could write a Python script to read this
data,
> > parse
> > >> the
> > >> > metadata, and hand it to MET in memory.
> > >> >
> > >> > We've posted several examples of this to the MET website
here:
> > >> >
https://dtcenter.org/met/users/downloads/analysis_scripts.php
> > >> >
> > >> > Is running met-8.0 an option for you?
> > >> >
> > >> > Thanks,
> > >> > John
> > >> >
> > >> > netcdf \20180720-181500 {
> > >> > dimensions:
> > >> > Lat = 3500 ;
> > >> > Lon = 7000 ;
> > >> > variables:
> > >> > float RadarOnly_QPE_01H(Lat, Lon) ;
> > >> > RadarOnly_QPE_01H:Units = "mm" ;
> > >> >
> > >> > // global attributes:
> > >> > :TypeName = "RadarOnly_QPE_01H" ;
> > >> > :DataType = "LatLonGrid" ;
> > >> > :Time = 1532110440 ;
> > >> > :FractionalTime = 0.f ;
> > >> > :MissingData = -99900.f ;
> > >> > :RangeFolded = -99901.f ;
> > >> > :Latitude = 55.f ;
> > >> > :Longitude = -130.f ;
> > >> > :Height = 0.f ;
> > >> > :LatGridSpacing = 0.01f ;
> > >> > :LonGridSpacing = 0.01f ;
> > >> > :attributes = "" ;
> > >> > }
> > >> >
> > >> > On Wed, Feb 13, 2019 at 12:18 PM Michael Erickson - NOAA
Affiliate
> via
> > >> RT <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> > >
> > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898 >
> > >> > >
> > >> > > Thank you again John!
> > >> > >
> > >> > > I think the issue here is related to the way the netCDF
file is
> > >> created.
> > >> > > The data contains 15 minute NEWSe precipitation and was
given to
> me
> > by
> > >> > the
> > >> > > developers.
> > >> > >
> > >> > > It looks like this netCDF file is not in a proper format to
be
> read
> > >> in by
> > >> > > MET. I have tried:
> > >> > >
> > >> > > /opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf
> test.ps
> > >> > > 'name="RadarOnly_QPE_01H"; level="(*,*)";'
> > >> > > /opt/MET6/met6.0/bin/plot_data_plane 20180720-181500.netcdf
> test.ps
> > >> > > 'name="RadarOnly_QPE_01H"; level="R1";'
> > >> > >
> > >> > > with the error:
> > >> > >
> > >> > > ERROR :
> > >> > > ERROR : plot_data_plane -> file "20180720-181500.netcdf"
not a
> > valid
> > >> > data
> > >> > > file
> > >> > > ERROR :
> > >> > >
> > >> > > I have loaded the file here:
> > >> https://ftp.wpc.ncep.noaa.gov/erickson/dtc/
> > >> > >
> > >> > > If there is no way to read in this data with MET, I can
probably
> > write
> > >> > some
> > >> > > code to create a workaround.
> > >> > >
> > >> > > Thanks!
> > >> > >
> > >> > > Mike
> > >> > >
> > >> > > On Wed, Feb 13, 2019 at 5:34 PM John Halley Gotway via RT <
> > >> > > met_help at ucar.edu>
> > >> > > wrote:
> > >> > >
> > >> > > > Mike,
> > >> > > >
> > >> > > > Yes, it *should* be able to handle that. As a caveat, I
haven't
> > >> done
> > >> > so
> > >> > > > for projects in recent memory. Below, I've listed the
usage
> > >> statement
> > >> > > for
> > >> > > > pcp_combine and highlighted the *in_accum* and
*out_accum*
> > options.
> > >> > They
> > >> > > > can both be formatted as HH[MMSS]... where the square
brackets
> > mean
> > >> > > > optional. Let's say you want to accumulation 12 5-minute
> > intervals
> > >> > into
> > >> > > a
> > >> > > > 1-hour accumulation, you'd set:
> > >> > > > in_accum = 000500 (for 5 minutes)
> > >> > > > out_accum = 01 (for 1 hour... or 010000 works too)
> > >> > > >
> > >> > > > Please let me know if you find that it doesn't work as
expected.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > John
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > * SUM_ARGS: init_time
> > >> > > > in_accum valid_time
out_accum
> > >> > > > out_file [-pcpdir path] [-
pcprx
> > >> > > > reg_exp] where "init_time" indicates the
> > >> > initialization
> > >> > > > time of the input data files in YYYYMMDD[_HH[MMSS]]
format
> > >> > > > (required). "in_accum" indicates
the
> > >> > accumulation
> > >> > > > interval of the input data files in HH[MMSS] format
> > >> > > > (required). "valid_time" indicates
the
> > >> desired
> > >> > > valid
> > >> > > > time in YYYYMMDD[_HH[MMSS]] format (required).
> > >> > > > "out_accum" indicates the desired accumulation interval
for the
> > >> output
> > >> > > > NetCDF file in HH[MMSS] format (required).
> > >> > > > "out_file" indicates the name of the output NetCDF file
to be
> > >> written
> > >> > > > consisting of the sum of the accumulation intervals
> > >> > > > (required). "-pcpdir path"
overrides the
> > >> default
> > >> > > > precipitation directory (.) (optional).
> > >> "-pcprx
> > >> > > > reg_exp" overrides the default regular expression for
> > precipitation
> > >> > file
> > >> > > > naming convention (.*) (optional).
Note:
> > Set
> > >> > > > init_time to 00000000_000000 when summing observation
files.*
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Feb 13, 2019 at 8:18 AM Michael Erickson - NOAA
> Affiliate
> > >> via
> > >> > RT
> > >> > > <
> > >> > > > met_help at ucar.edu> wrote:
> > >> > > >
> > >> > > > >
> > >> > > > > Wed Feb 13 08:17:29 2019: Request 88898 was acted upon.
> > >> > > > > Transaction: Ticket created by
michael.j.erickson at noaa.gov
> > >> > > > > Queue: met_help
> > >> > > > > Subject: pcp_combine With Sub-hourly Data
> > >> > > > > Owner: Nobody
> > >> > > > > Requestors: michael.j.erickson at noaa.gov
> > >> > > > > Status: new
> > >> > > > > Ticket <URL:
> > >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88898
> > >> > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > Hello there,
> > >> > > > >
> > >> > > > > Can pcp_combine be used to sum sub-hourly data (e.g. 5-
minute
> > >> data),
> > >> > to
> > >> > > > > create a sub-hourly total (e.g. 15-minute data) using
the -sum
> > >> > option?
> > >> > > > >
> > >> > > > > If not, I can always code in my results using the -add
option,
> > >> but I
> > >> > > > wanted
> > >> > > > > to ask first since the code is already in place.
> > >> > > > >
> > >> > > > > Thank you once again!
> > >> > > > >
> > >> > > > > Mike
> > >> > > > >
> > >> > > > > --
> > >> > > > > Michael J. Erickson
> > >> > > > >
> > >> > > > > Research Scientist
> > >> > > > > Cooperative Institute for Research in Environmental
Sciences
> > >> (CIRES)
> > >> > > > > NOAA/NWS/NCEP/Weather Prediction Center
> > >> > > > > Phone: 301-683-1546
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> > > --
> > >> > > Michael J. Erickson
> > >> > >
> > >> > > Research Scientist
> > >> > > Cooperative Institute for Research in Environmental
Sciences
> (CIRES)
> > >> > > NOAA/NWS/NCEP/Weather Prediction Center
> > >> > > Phone: 301-683-1546
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >> --
> > >> Michael J. Erickson
> > >>
> > >> Research Scientist
> > >> Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > >> NOAA/NWS/NCEP/Weather Prediction Center
> > >> Phone: 301-683-1546
> > >>
> > >>
> >
> >
>
> --
> Michael J. Erickson
>
> Research Scientist
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> NOAA/NWS/NCEP/Weather Prediction Center
> Phone: 301-683-1546
>
>
------------------------------------------------
More information about the Met_help
mailing list