[Met_help] [rt.rap.ucar.edu #84581] History for Re: question about netcdf format files, dataset 337.0
John Halley Gotway via RT
met_help at ucar.edu
Tue Jul 9 12:03:35 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
hi tom,
ok, thanks for the info. do you know who
we would check with at the DTC about options
for this tool? (or since met_help is cc'd on this
thread, perhaps they know?)
thanks, and we'll check out the ascii format.
if you get some time to add the obs error to the
ascii files, please do let us know. that would
be great.
again, thanks.
nancy
On 3/28/18 3:19 PM, Thomas Cram wrote:
> Hi Nancy,
> according to the MET documentation, the prepbufr report type and observation error values cannot be added to the pb2nc output. I
> am copying MET support in this message, as they can take a note of this and possibly consider it for inclusion in a future release.
>
> In the meantime, you can choose ASCII output from the ds337.0 subsetting form, and this will include the PB report type in the
> output. It currently does not include the observation error values, however, but I will see if I can modify the code to include
> this sometime in the near future.
>
> - Tom
>
> On Wed, Mar 28, 2018 at 2:45 PM, nancy collins <nancy at ucar.edu <mailto:nancy at ucar.edu>> wrote:
>
> tom,
>
> thanks for checking into this. that would be great news
> if these items were possible to add to the output files.
>
> n.
>
>
> On 3/28/18 9:13 AM, Thomas Cram wrote:
>> Hi Nancy,
>>
>> I use the pb2nc tool in the MET software to produce the NetCDF files (see https://dtcenter.org/met/users/index.php
>> <https://dtcenter.org/met/users/index.php>), and the output is defined by a configuration file passed to pb2nc. I'll take a
>> look at the documentation and see if it's possible to include the error values and PB report types in the output.
>>
>> - Tom
>>
>> On Tue, Mar 27, 2018 at 3:57 PM, Nancy Collins <nancy at ucar.edu <mailto:nancy at ucar.edu>> wrote:
>>
>> hello thomas,
>>
>> i'm looking at RDA dataset number 337.0 (NCEP prepbufr observation files).
>> this data is now also available in netcdf format, which is great.
>>
>> we downloaded an example file and most things we need are there.
>> however, the netcdf file seems to be missing the observation error
>> values (present in the prepbufr files) and also the prepbufr report
>> types (again, present in the prepbufr files). while the netcdf files
>> do have 'grib codes' and 'message types', they don't map uniquely
>> to prepbufr report types, and we see no way to get the obs errors.
>> (note that this is not the quality control - it is the error in the same
>> units as the observation value. like a temperature obs error might
>> be 1 or 2 degrees K.)
>>
>> was this an intentional choice? can you tell us if it's possible to
>> get netcdf format files with these two additional fields? are there
>> options on the conversion tool that could be set?
>>
>> thanks for any information on how these files are created and
>> who made the choices for what data is converted into the
>> output netcdf files.
>>
>> nancy
>>
>>
>>
>>
>> --
>> Thomas Cram
>> NCAR / CISL / DSS
>> tcram at ucar.edu <mailto:tcram at ucar.edu>
>> rda.ucar.edu <https://rda.ucar.edu>
>
>
>
>
> --
> Thomas Cram
> NCAR / CISL / DSS
> tcram at ucar.edu <mailto:tcram at ucar.edu>
> rda.ucar.edu <https://rda.ucar.edu>
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: question about netcdf format files, dataset 337.0
From: John Halley Gotway
Time: Wed Mar 28 16:07:00 2018
Hi Tom and Nancy,
You are correct, pb2nc doesn't currently include those quantities in
the
output file. However, as luck would have it, we will be including the
prepbufr report type, input report type, and instrument type in the
pb2nc
output for the next version of met, met-7.1.
We have some funding to enhance the ensemble-stat tool in MET to do
apply
observation error. However, the observation error can be different
for
each report type and instrument type. So we need to pass that info
through
pb2nc for ensemble-stat to use in it's application of observation
error.
Since we're editing the pb2nc file, we're looking for ways to simplify
it's
structure and avoid repetition of strings whenever possible. And we
have
some good ideas on how to do that.
Can you tell me more about the "obs error" from the PREPBUFR file
itself?
I'm not familiar with that field. Adding it during our development
for
met-7.1 would definitely be the best time to do it!
Thanks,
John Halley Gotway
On Wed, Mar 28, 2018 at 3:35 PM, nancy at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> Wed Mar 28 15:35:25 2018: Request 84581 was acted upon.
> Transaction: Ticket created by nancy at ucar.edu
> Queue: met_help
> Subject: Re: question about netcdf format files, dataset 337.0
> Owner: Nobody
> Requestors: nancy at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581 >
>
>
> hi tom,
>
> ok, thanks for the info. do you know who
> we would check with at the DTC about options
> for this tool? (or since met_help is cc'd on this
> thread, perhaps they know?)
>
> thanks, and we'll check out the ascii format.
> if you get some time to add the obs error to the
> ascii files, please do let us know. that would
> be great.
>
> again, thanks.
> nancy
>
> On 3/28/18 3:19 PM, Thomas Cram wrote:
> > Hi Nancy,
> > according to the MET documentation, the prepbufr report type and
> observation error values cannot be added to the pb2nc output. I
> > am copying MET support in this message, as they can take a note of
this
> and possibly consider it for inclusion in a future release.
> >
> > In the meantime, you can choose ASCII output from the ds337.0
subsetting
> form, and this will include the PB report type in the
> > output. It currently does not include the observation error
values,
> however, but I will see if I can modify the code to include
> > this sometime in the near future.
> >
> > - Tom
> >
> > On Wed, Mar 28, 2018 at 2:45 PM, nancy collins <nancy at ucar.edu
<mailto:
> nancy at ucar.edu>> wrote:
> >
> > tom,
> >
> > thanks for checking into this. that would be great news
> > if these items were possible to add to the output files.
> >
> > n.
> >
> >
> > On 3/28/18 9:13 AM, Thomas Cram wrote:
> >> Hi Nancy,
> >>
> >> I use the pb2nc tool in the MET software to produce the
NetCDF
> files (see https://dtcenter.org/met/users/index.php
> >> <https://dtcenter.org/met/users/index.php>), and the output
is
> defined by a configuration file passed to pb2nc. I'll take a
> >> look at the documentation and see if it's possible to include
the
> error values and PB report types in the output.
> >>
> >> - Tom
> >>
> >> On Tue, Mar 27, 2018 at 3:57 PM, Nancy Collins
<nancy at ucar.edu
> <mailto:nancy at ucar.edu>> wrote:
> >>
> >> hello thomas,
> >>
> >> i'm looking at RDA dataset number 337.0 (NCEP prepbufr
> observation files).
> >> this data is now also available in netcdf format, which
is
> great.
> >>
> >> we downloaded an example file and most things we need are
there.
> >> however, the netcdf file seems to be missing the
observation
> error
> >> values (present in the prepbufr files) and also the
prepbufr
> report
> >> types (again, present in the prepbufr files). while the
netcdf
> files
> >> do have 'grib codes' and 'message types', they don't map
> uniquely
> >> to prepbufr report types, and we see no way to get the
obs
> errors.
> >> (note that this is not the quality control - it is the
error in
> the same
> >> units as the observation value. like a temperature obs
error
> might
> >> be 1 or 2 degrees K.)
> >>
> >> was this an intentional choice? can you tell us if it's
> possible to
> >> get netcdf format files with these two additional fields?
are
> there
> >> options on the conversion tool that could be set?
> >>
> >> thanks for any information on how these files are created
and
> >> who made the choices for what data is converted into the
> >> output netcdf files.
> >>
> >> nancy
> >>
> >>
> >>
> >>
> >> --
> >> Thomas Cram
> >> NCAR / CISL / DSS
> >> tcram at ucar.edu <mailto:tcram at ucar.edu>
> >> rda.ucar.edu <https://rda.ucar.edu>
> >
> >
> >
> >
> > --
> > Thomas Cram
> > NCAR / CISL / DSS
> > tcram at ucar.edu <mailto:tcram at ucar.edu>
> > rda.ucar.edu <https://rda.ucar.edu>
>
>
>
------------------------------------------------
Subject: Re: question about netcdf format files, dataset 337.0
From: syha at ucar.edu
Time: Wed Mar 28 16:21:43 2018
Dear John Halley Gotway,
This is Soyoung Ha from MMM. I'm the one who is actually using the
data and
suggested to use the netcdf data converted thru MET. So I quickly
looked at
met-7.0 source code for the part.
In the READPB subroutine in met-7.0/src/tools/other/pb2nc/readpb.f, I
grabbed these comments:
C* The 4-D array of data, EVNS ( ii, lv, jj, kk ), is indexed
C* as follows:
C*
C* "ii" indexes the event data types; these consist of:
C* 1) OBservation
C* 2) Quality Mark
C* 3) Program Code
C* 4) Reason Code
C* 5) ForeCast value
C* 6) ANalysed value
C* 7) office note CATegory
C* "lv" indexes the levels of the report
C* "jj" indexes the event stacks
C* "kk" indexes the variable types (p,q,t,z,u,v)
And the obs error we want here is 7) office note CATegory.
It looks like the code is reading all 7 "ii" indices (which means obs
error
is also read), but it is just not stored in the output netcdf file.
The output netcdf file Tom kindly converted for me using MET5.0 has
the
output variable "obs_arr" which seems to be extensible, would it be
easy
enough to pass this info as well?
Thanks a lot,
Soyoung
On Wed, Mar 28, 2018 at 4:07 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Hi Tom and Nancy,
>
> You are correct, pb2nc doesn't currently include those quantities in
the
> output file. However, as luck would have it, we will be including
the
> prepbufr report type, input report type, and instrument type in the
pb2nc
> output for the next version of met, met-7.1.
>
> We have some funding to enhance the ensemble-stat tool in MET to do
apply
> observation error. However, the observation error can be different
for
> each report type and instrument type. So we need to pass that info
through
> pb2nc for ensemble-stat to use in it's application of observation
error.
>
> Since we're editing the pb2nc file, we're looking for ways to
simplify it's
> structure and avoid repetition of strings whenever possible. And we
have
> some good ideas on how to do that.
>
> Can you tell me more about the "obs error" from the PREPBUFR file
itself?
> I'm not familiar with that field. Adding it during our development
for
> met-7.1 would definitely be the best time to do it!
>
> Thanks,
> John Halley Gotway
>
>
> On Wed, Mar 28, 2018 at 3:35 PM, nancy at ucar.edu via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Wed Mar 28 15:35:25 2018: Request 84581 was acted upon.
> > Transaction: Ticket created by nancy at ucar.edu
> > Queue: met_help
> > Subject: Re: question about netcdf format files, dataset
337.0
> > Owner: Nobody
> > Requestors: nancy at ucar.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581 >
> >
> >
> > hi tom,
> >
> > ok, thanks for the info. do you know who
> > we would check with at the DTC about options
> > for this tool? (or since met_help is cc'd on this
> > thread, perhaps they know?)
> >
> > thanks, and we'll check out the ascii format.
> > if you get some time to add the obs error to the
> > ascii files, please do let us know. that would
> > be great.
> >
> > again, thanks.
> > nancy
> >
> > On 3/28/18 3:19 PM, Thomas Cram wrote:
> > > Hi Nancy,
> > > according to the MET documentation, the prepbufr report type and
> > observation error values cannot be added to the pb2nc output. I
> > > am copying MET support in this message, as they can take a note
of this
> > and possibly consider it for inclusion in a future release.
> > >
> > > In the meantime, you can choose ASCII output from the ds337.0
> subsetting
> > form, and this will include the PB report type in the
> > > output. It currently does not include the observation error
values,
> > however, but I will see if I can modify the code to include
> > > this sometime in the near future.
> > >
> > > - Tom
> > >
> > > On Wed, Mar 28, 2018 at 2:45 PM, nancy collins <nancy at ucar.edu
> <mailto:
> > nancy at ucar.edu>> wrote:
> > >
> > > tom,
> > >
> > > thanks for checking into this. that would be great news
> > > if these items were possible to add to the output files.
> > >
> > > n.
> > >
> > >
> > > On 3/28/18 9:13 AM, Thomas Cram wrote:
> > >> Hi Nancy,
> > >>
> > >> I use the pb2nc tool in the MET software to produce the
NetCDF
> > files (see https://dtcenter.org/met/users/index.php
> > >> <https://dtcenter.org/met/users/index.php>), and the output
is
> > defined by a configuration file passed to pb2nc. I'll take a
> > >> look at the documentation and see if it's possible to
include the
> > error values and PB report types in the output.
> > >>
> > >> - Tom
> > >>
> > >> On Tue, Mar 27, 2018 at 3:57 PM, Nancy Collins
<nancy at ucar.edu
> > <mailto:nancy at ucar.edu>> wrote:
> > >>
> > >> hello thomas,
> > >>
> > >> i'm looking at RDA dataset number 337.0 (NCEP prepbufr
> > observation files).
> > >> this data is now also available in netcdf format, which
is
> > great.
> > >>
> > >> we downloaded an example file and most things we need
are
> there.
> > >> however, the netcdf file seems to be missing the
observation
> > error
> > >> values (present in the prepbufr files) and also the
prepbufr
> > report
> > >> types (again, present in the prepbufr files). while
the
> netcdf
> > files
> > >> do have 'grib codes' and 'message types', they don't
map
> > uniquely
> > >> to prepbufr report types, and we see no way to get the
obs
> > errors.
> > >> (note that this is not the quality control - it is the
error
> in
> > the same
> > >> units as the observation value. like a temperature obs
error
> > might
> > >> be 1 or 2 degrees K.)
> > >>
> > >> was this an intentional choice? can you tell us if
it's
> > possible to
> > >> get netcdf format files with these two additional
fields? are
> > there
> > >> options on the conversion tool that could be set?
> > >>
> > >> thanks for any information on how these files are
created and
> > >> who made the choices for what data is converted into
the
> > >> output netcdf files.
> > >>
> > >> nancy
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> Thomas Cram
> > >> NCAR / CISL / DSS
> > >> tcram at ucar.edu <mailto:tcram at ucar.edu>
> > >> rda.ucar.edu <https://rda.ucar.edu>
> > >
> > >
> > >
> > >
> > > --
> > > Thomas Cram
> > > NCAR / CISL / DSS
> > > tcram at ucar.edu <mailto:tcram at ucar.edu>
> > > rda.ucar.edu <https://rda.ucar.edu>
> >
> >
> >
>
>
------------------------------------------------
Subject: Re: question about netcdf format files, dataset 337.0
From: Thomas Cram
Time: Wed Mar 28 16:28:22 2018
Hi Soyoung & John,
my interpretation of "office note CATegory" is that it's the Data
Level
Category (denoted by the mnemonic CAT), not the observation error:
http://www.emc.ncep.noaa.gov/mmb/data_processing/table_local_await-
val.htm#0-08-193
The observation error is new to me, too, but it looks like they are
defined
by the Table B entries for Report Level data. My reference for this
is the
prepbufr Table 1 mnemonics:
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
Scroll down to "Table B entries for the report level data". This is
about
1/3 down the page. You will find observation error mnemonics for each
variable. For example, pressure observation error is coded as "POE",
temperature observation error is "TOE", etc.
- Tom
On Wed, Mar 28, 2018 at 4:21 PM, Soyoung Ha <syha at ucar.edu> wrote:
> Dear John Halley Gotway,
>
> This is Soyoung Ha from MMM. I'm the one who is actually using the
data
> and suggested to use the netcdf data converted thru MET. So I
quickly
> looked at met-7.0 source code for the part.
>
> In the READPB subroutine in met-7.0/src/tools/other/pb2nc/readpb.f,
I
> grabbed these comments:
>
> C* The 4-D array of data, EVNS ( ii, lv, jj, kk ), is indexed
> C* as follows:
> C*
> C* "ii" indexes the event data types; these consist of:
> C* 1) OBservation
> C* 2) Quality Mark
> C* 3) Program Code
> C* 4) Reason Code
> C* 5) ForeCast value
> C* 6) ANalysed value
> C* 7) office note CATegory
> C* "lv" indexes the levels of the report
> C* "jj" indexes the event stacks
> C* "kk" indexes the variable types (p,q,t,z,u,v)
>
> And the obs error we want here is 7) office note CATegory.
> It looks like the code is reading all 7 "ii" indices (which means
obs
> error is also read), but it is just not stored in the output netcdf
file.
> The output netcdf file Tom kindly converted for me using MET5.0 has
the
> output variable "obs_arr" which seems to be extensible, would it be
easy
> enough to pass this info as well?
> Thanks a lot,
>
> Soyoung
>
>
>
>
>
> On Wed, Mar 28, 2018 at 4:07 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Hi Tom and Nancy,
>>
>> You are correct, pb2nc doesn't currently include those quantities
in the
>> output file. However, as luck would have it, we will be including
the
>> prepbufr report type, input report type, and instrument type in the
pb2nc
>> output for the next version of met, met-7.1.
>>
>> We have some funding to enhance the ensemble-stat tool in MET to do
apply
>> observation error. However, the observation error can be different
for
>> each report type and instrument type. So we need to pass that info
>> through
>> pb2nc for ensemble-stat to use in it's application of observation
error.
>>
>> Since we're editing the pb2nc file, we're looking for ways to
simplify
>> it's
>> structure and avoid repetition of strings whenever possible. And
we have
>> some good ideas on how to do that.
>>
>> Can you tell me more about the "obs error" from the PREPBUFR file
itself?
>> I'm not familiar with that field. Adding it during our development
for
>> met-7.1 would definitely be the best time to do it!
>>
>> Thanks,
>> John Halley Gotway
>>
>>
>> On Wed, Mar 28, 2018 at 3:35 PM, nancy at ucar.edu via RT
<met_help at ucar.edu
>> >
>> wrote:
>>
>> >
>> > Wed Mar 28 15:35:25 2018: Request 84581 was acted upon.
>> > Transaction: Ticket created by nancy at ucar.edu
>> > Queue: met_help
>> > Subject: Re: question about netcdf format files, dataset
337.0
>> > Owner: Nobody
>> > Requestors: nancy at ucar.edu
>> > Status: new
>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581 >
>> >
>> >
>> > hi tom,
>> >
>> > ok, thanks for the info. do you know who
>> > we would check with at the DTC about options
>> > for this tool? (or since met_help is cc'd on this
>> > thread, perhaps they know?)
>> >
>> > thanks, and we'll check out the ascii format.
>> > if you get some time to add the obs error to the
>> > ascii files, please do let us know. that would
>> > be great.
>> >
>> > again, thanks.
>> > nancy
>> >
>> > On 3/28/18 3:19 PM, Thomas Cram wrote:
>> > > Hi Nancy,
>> > > according to the MET documentation, the prepbufr report type
and
>> > observation error values cannot be added to the pb2nc output. I
>> > > am copying MET support in this message, as they can take a note
of
>> this
>> > and possibly consider it for inclusion in a future release.
>> > >
>> > > In the meantime, you can choose ASCII output from the ds337.0
>> subsetting
>> > form, and this will include the PB report type in the
>> > > output. It currently does not include the observation error
values,
>> > however, but I will see if I can modify the code to include
>> > > this sometime in the near future.
>> > >
>> > > - Tom
>> > >
>> > > On Wed, Mar 28, 2018 at 2:45 PM, nancy collins <nancy at ucar.edu
>> <mailto:
>> > nancy at ucar.edu>> wrote:
>> > >
>> > > tom,
>> > >
>> > > thanks for checking into this. that would be great news
>> > > if these items were possible to add to the output files.
>> > >
>> > > n.
>> > >
>> > >
>> > > On 3/28/18 9:13 AM, Thomas Cram wrote:
>> > >> Hi Nancy,
>> > >>
>> > >> I use the pb2nc tool in the MET software to produce the
NetCDF
>> > files (see https://dtcenter.org/met/users/index.php
>> > >> <https://dtcenter.org/met/users/index.php>), and the
output is
>> > defined by a configuration file passed to pb2nc. I'll take a
>> > >> look at the documentation and see if it's possible to
include the
>> > error values and PB report types in the output.
>> > >>
>> > >> - Tom
>> > >>
>> > >> On Tue, Mar 27, 2018 at 3:57 PM, Nancy Collins
<nancy at ucar.edu
>> > <mailto:nancy at ucar.edu>> wrote:
>> > >>
>> > >> hello thomas,
>> > >>
>> > >> i'm looking at RDA dataset number 337.0 (NCEP prepbufr
>> > observation files).
>> > >> this data is now also available in netcdf format,
which is
>> > great.
>> > >>
>> > >> we downloaded an example file and most things we need
are
>> there.
>> > >> however, the netcdf file seems to be missing the
observation
>> > error
>> > >> values (present in the prepbufr files) and also the
prepbufr
>> > report
>> > >> types (again, present in the prepbufr files). while
the
>> netcdf
>> > files
>> > >> do have 'grib codes' and 'message types', they don't
map
>> > uniquely
>> > >> to prepbufr report types, and we see no way to get the
obs
>> > errors.
>> > >> (note that this is not the quality control - it is the
error
>> in
>> > the same
>> > >> units as the observation value. like a temperature
obs error
>> > might
>> > >> be 1 or 2 degrees K.)
>> > >>
>> > >> was this an intentional choice? can you tell us if
it's
>> > possible to
>> > >> get netcdf format files with these two additional
fields?
>> are
>> > there
>> > >> options on the conversion tool that could be set?
>> > >>
>> > >> thanks for any information on how these files are
created and
>> > >> who made the choices for what data is converted into
the
>> > >> output netcdf files.
>> > >>
>> > >> nancy
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Thomas Cram
>> > >> NCAR / CISL / DSS
>> > >> tcram at ucar.edu <mailto:tcram at ucar.edu>
>> > >> rda.ucar.edu <https://rda.ucar.edu>
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > > Thomas Cram
>> > > NCAR / CISL / DSS
>> > > tcram at ucar.edu <mailto:tcram at ucar.edu>
>> > > rda.ucar.edu <https://rda.ucar.edu>
>> >
>> >
>> >
>>
>>
>
--
Thomas Cram
NCAR / CISL / DSS
tcram at ucar.edu
rda.ucar.edu
------------------------------------------------
Subject: Re: question about netcdf format files, dataset 337.0
From: nancy at ucar.edu
Time: Wed Mar 28 16:45:17 2018
hi john,
i see i'm slow to this email thread. as soyoung already said,
we access the obs error with ii=7. i have no idea why the docs
say 'office note CATegory'.
the converter code i'm looking at started life as a prepbufr dump
program, and it called event type index 7 the observational error,
and in fact it has lines like:
toe = evns(7, lv, j2t, 3)
qoe = evns(7, lv, j2q, 2)
poe = evns(7, lv, 1, 1)
from inspecting the values we get when we run the program
they appear to be reasonable error values. but it's a little
disquieting that it's not documented as such.
n.
On 3/28/18 4:28 PM, Thomas Cram wrote:
> Hi Soyoung & John,
>
> my interpretation of "office note CATegory" is that it's the Data
Level Category (denoted by the mnemonic CAT), not the
> observation error:
> http://www.emc.ncep.noaa.gov/mmb/data_processing/table_local_await-
val.htm#0-08-193
>
> The observation error is new to me, too, but it looks like they are
defined by the Table B entries for Report Level data. My
> reference for this is the prepbufr Table 1 mnemonics:
>
>
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
>
<http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm>
>
> Scroll down to "Table B entries for the report level data". This is
about 1/3 down the page. You will find observation error
> mnemonics for each variable. For example, pressure observation
error is coded as "POE", temperature observation error is "TOE", etc.
>
> - Tom
>
>
>
> On Wed, Mar 28, 2018 at 4:21 PM, Soyoung Ha <syha at ucar.edu
<mailto:syha at ucar.edu>> wrote:
>
> Dear John Halley Gotway,
>
> This is Soyoung Ha from MMM. I'm the one who is actually using
the data and suggested to use the netcdf data converted thru
> MET. So I quickly looked at met-7.0 source code for the part.
>
> In the READPB subroutine in met-
7.0/src/tools/other/pb2nc/readpb.f, I grabbed these comments:
>
> C* The 4-D array of data, EVNS ( ii, lv, jj, kk ), is
indexed
> C* as follows:
> C*
> C* "ii" indexes the event data types; these consist of:
> C* 1) OBservation
> C* 2) Quality Mark
> C* 3) Program Code
> C* 4) Reason Code
> C* 5) ForeCast value
> C* 6) ANalysed value
> C* 7) office note CATegory
> C* "lv" indexes the levels of the report
> C* "jj" indexes the event stacks
> C* "kk" indexes the variable types (p,q,t,z,u,v)
>
> And the obs error we want here is 7) office note CATegory.
> It looks like the code is reading all 7 "ii" indices (which
means obs error is also read), but it is just not stored in the
> output netcdf file.
> The output netcdf file Tom kindly converted for me using MET5.0
has the output variable "obs_arr" which seems to be
> extensible, would it be easy enough to pass this info as well?
> Thanks a lot,
>
> Soyoung
>
>
>
>
>
> On Wed, Mar 28, 2018 at 4:07 PM, John Halley Gotway via RT
<met_help at ucar.edu <mailto:met_help at ucar.edu>> wrote:
>
> Hi Tom and Nancy,
>
> You are correct, pb2nc doesn't currently include those
quantities in the
> output file. However, as luck would have it, we will be
including the
> prepbufr report type, input report type, and instrument type
in the pb2nc
> output for the next version of met, met-7.1.
>
> We have some funding to enhance the ensemble-stat tool in
MET to do apply
> observation error. However, the observation error can be
different for
> each report type and instrument type. So we need to pass
that info through
> pb2nc for ensemble-stat to use in it's application of
observation error.
>
> Since we're editing the pb2nc file, we're looking for ways
to simplify it's
> structure and avoid repetition of strings whenever
possible. And we have
> some good ideas on how to do that.
>
> Can you tell me more about the "obs error" from the PREPBUFR
file itself?
> I'm not familiar with that field. Adding it during our
development for
> met-7.1 would definitely be the best time to do it!
>
> Thanks,
> John Halley Gotway
>
>
> On Wed, Mar 28, 2018 at 3:35 PM, nancy at ucar.edu
<mailto:nancy at ucar.edu> via RT <met_help at ucar.edu
<mailto:met_help at ucar.edu>>
> wrote:
>
> >
> > Wed Mar 28 15:35:25 2018: Request 84581 was acted upon.
> > Transaction: Ticket created by nancy at ucar.edu
<mailto:nancy at ucar.edu>
> > Queue: met_help
> > Subject: Re: question about netcdf format files,
dataset 337.0
> > Owner: Nobody
> > Requestors: nancy at ucar.edu <mailto:nancy at ucar.edu>
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581
> <https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581> >
> >
> >
> > hi tom,
> >
> > ok, thanks for the info. do you know who
> > we would check with at the DTC about options
> > for this tool? (or since met_help is cc'd on this
> > thread, perhaps they know?)
> >
> > thanks, and we'll check out the ascii format.
> > if you get some time to add the obs error to the
> > ascii files, please do let us know. that would
> > be great.
> >
> > again, thanks.
> > nancy
> >
> > On 3/28/18 3:19 PM, Thomas Cram wrote:
> > > Hi Nancy,
> > > according to the MET documentation, the prepbufr report
type and
> > observation error values cannot be added to the pb2nc
output. I
> > > am copying MET support in this message, as they can take
a note of this
> > and possibly consider it for inclusion in a future
release.
> > >
> > > In the meantime, you can choose ASCII output from the
ds337.0 subsetting
> > form, and this will include the PB report type in the
> > > output. It currently does not include the observation
error values,
> > however, but I will see if I can modify the code to
include
> > > this sometime in the near future.
> > >
> > > - Tom
> > >
> > > On Wed, Mar 28, 2018 at 2:45 PM, nancy collins
<nancy at ucar.edu <mailto:nancy at ucar.edu> <mailto:
> > nancy at ucar.edu <mailto:nancy at ucar.edu>>> wrote:
> > >
> > > tom,
> > >
> > > thanks for checking into this. that would be great
news
> > > if these items were possible to add to the output
files.
> > >
> > > n.
> > >
> > >
> > > On 3/28/18 9:13 AM, Thomas Cram wrote:
> > >> Hi Nancy,
> > >>
> > >> I use the pb2nc tool in the MET software to produce
the NetCDF
> > files (see https://dtcenter.org/met/users/index.php
<https://dtcenter.org/met/users/index.php>
> > >> <https://dtcenter.org/met/users/index.php
<https://dtcenter.org/met/users/index.php>>), and the output is
> > defined by a configuration file passed to pb2nc. I'll
take a
> > >> look at the documentation and see if it's possible
to include the
> > error values and PB report types in the output.
> > >>
> > >> - Tom
> > >>
> > >> On Tue, Mar 27, 2018 at 3:57 PM, Nancy Collins
<nancy at ucar.edu <mailto:nancy at ucar.edu>
> > <mailto:nancy at ucar.edu <mailto:nancy at ucar.edu>>> wrote:
> > >>
> > >> hello thomas,
> > >>
> > >> i'm looking at RDA dataset number 337.0 (NCEP
prepbufr
> > observation files).
> > >> this data is now also available in netcdf
format, which is
> > great.
> > >>
> > >> we downloaded an example file and most things
we need are there.
> > >> however, the netcdf file seems to be missing
the observation
> > error
> > >> values (present in the prepbufr files) and also
the prepbufr
> > report
> > >> types (again, present in the prepbufr files).
while the netcdf
> > files
> > >> do have 'grib codes' and 'message types', they
don't map
> > uniquely
> > >> to prepbufr report types, and we see no way to
get the obs
> > errors.
> > >> (note that this is not the quality control - it
is the error in
> > the same
> > >> units as the observation value. like a
temperature obs error
> > might
> > >> be 1 or 2 degrees K.)
> > >>
> > >> was this an intentional choice? can you tell
us if it's
> > possible to
> > >> get netcdf format files with these two
additional fields? are
> > there
> > >> options on the conversion tool that could be
set?
> > >>
> > >> thanks for any information on how these files
are created and
> > >> who made the choices for what data is converted
into the
> > >> output netcdf files.
> > >>
> > >> nancy
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> Thomas Cram
> > >> NCAR / CISL / DSS
> > >> tcram at ucar.edu <mailto:tcram at ucar.edu>
<mailto:tcram at ucar.edu <mailto:tcram at ucar.edu>>
> > >> rda.ucar.edu <http://rda.ucar.edu>
<https://rda.ucar.edu>
> > >
> > >
> > >
> > >
> > > --
> > > Thomas Cram
> > > NCAR / CISL / DSS
> > > tcram at ucar.edu <mailto:tcram at ucar.edu>
<mailto:tcram at ucar.edu <mailto:tcram at ucar.edu>>
> > > rda.ucar.edu <http://rda.ucar.edu>
<https://rda.ucar.edu>
> >
> >
> >
>
>
>
>
>
> --
> Thomas Cram
> NCAR / CISL / DSS
> tcram at ucar.edu <mailto:tcram at ucar.edu>
> rda.ucar.edu <https://rda.ucar.edu>
------------------------------------------------
Subject: Re: question about netcdf format files, dataset 337.0
From: syha at ucar.edu
Time: Wed Mar 28 17:02:35 2018
Tom,
As Nancy pointed out, we've used the 7th index as obs error, which
might be
adjusted later when needed.
Even in the GSI prepbufr_decoding program, they define the indices as
below.
character(80):: obstr='POB QOB TOB ZOB UOB VOB PWO CAT PRSS'
character(80):: qcstr='PQM QQM TQM ZQM WQM NUL PWQ '
character(80):: oestr='POE QOE TOE NUL WOE NUL PWE '
...
call ufbint(unit_in,obs,mxmn,mxlv,iret,obstr)
call ufbint(unit_in,oer,mxmn,mxlv,iret,oestr)
call ufbint(unit_in,qcf,mxmn,mxlv,iret,qcstr)
...
The corresponding string in the READPB module in MET (lines 62-68 in
pb2nc/readpb.f) looks as below.
CHARACTER*(MXSTRL) ostr ( MXR8VT )
+ / 'POB PQM PPC PRC PFC PAN CAT',
+ 'QOB QQM QPC QRC QFC QAN CAT',
+ 'TOB TQM TPC TRC TFC TAN CAT',
+ 'ZOB ZQM ZPC ZRC ZFC ZAN CAT',
+ 'UOB WQM WPC WRC UFC UAN CAT',
+ 'VOB WQM WPC WRC VFC VAN CAT' /
Comparing these two, we can figure out that the last column of "ostr"
(which is all "CAT") can be replaced with "oestr" that I listed in the
GSI
program above. So this column is considered to be obs error - POE,
QOE,
TOE, etc.
Again, the error can be adjusted later (before getting into the
assimilation system), but we want to read it as a reference or a
starting
value.
Thanks,
Soyoung
On Wed, Mar 28, 2018 at 4:28 PM, Thomas Cram <tcram at ucar.edu> wrote:
> Hi Soyoung & John,
>
> my interpretation of "office note CATegory" is that it's the Data
Level
> Category (denoted by the mnemonic CAT), not the observation error:
> http://www.emc.ncep.noaa.gov/mmb/data_processing/table_
> local_await-val.htm#0-08-193
>
> The observation error is new to me, too, but it looks like they are
> defined by the Table B entries for Report Level data. My reference
for
> this is the prepbufr Table 1 mnemonics:
>
>
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
>
> Scroll down to "Table B entries for the report level data". This is
about
> 1/3 down the page. You will find observation error mnemonics for
each
> variable. For example, pressure observation error is coded as
"POE",
> temperature observation error is "TOE", etc.
>
> - Tom
>
>
>
> On Wed, Mar 28, 2018 at 4:21 PM, Soyoung Ha <syha at ucar.edu> wrote:
>
>> Dear John Halley Gotway,
>>
>> This is Soyoung Ha from MMM. I'm the one who is actually using the
data
>> and suggested to use the netcdf data converted thru MET. So I
quickly
>> looked at met-7.0 source code for the part.
>>
>> In the READPB subroutine in met-7.0/src/tools/other/pb2nc/readpb.f,
I
>> grabbed these comments:
>>
>> C* The 4-D array of data, EVNS ( ii, lv, jj, kk ), is indexed
>> C* as follows:
>> C*
>> C* "ii" indexes the event data types; these consist of:
>> C* 1) OBservation
>> C* 2) Quality Mark
>> C* 3) Program Code
>> C* 4) Reason Code
>> C* 5) ForeCast value
>> C* 6) ANalysed value
>> C* 7) office note CATegory
>> C* "lv" indexes the levels of the report
>> C* "jj" indexes the event stacks
>> C* "kk" indexes the variable types (p,q,t,z,u,v)
>>
>> And the obs error we want here is 7) office note CATegory.
>> It looks like the code is reading all 7 "ii" indices (which means
obs
>> error is also read), but it is just not stored in the output netcdf
file.
>> The output netcdf file Tom kindly converted for me using MET5.0 has
the
>> output variable "obs_arr" which seems to be extensible, would it be
easy
>> enough to pass this info as well?
>> Thanks a lot,
>>
>> Soyoung
>>
>>
>>
>>
>>
>> On Wed, Mar 28, 2018 at 4:07 PM, John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>>> Hi Tom and Nancy,
>>>
>>> You are correct, pb2nc doesn't currently include those quantities
in the
>>> output file. However, as luck would have it, we will be including
the
>>> prepbufr report type, input report type, and instrument type in
the pb2nc
>>> output for the next version of met, met-7.1.
>>>
>>> We have some funding to enhance the ensemble-stat tool in MET to
do apply
>>> observation error. However, the observation error can be
different for
>>> each report type and instrument type. So we need to pass that
info
>>> through
>>> pb2nc for ensemble-stat to use in it's application of observation
error.
>>>
>>> Since we're editing the pb2nc file, we're looking for ways to
simplify
>>> it's
>>> structure and avoid repetition of strings whenever possible. And
we have
>>> some good ideas on how to do that.
>>>
>>> Can you tell me more about the "obs error" from the PREPBUFR file
itself?
>>> I'm not familiar with that field. Adding it during our
development for
>>> met-7.1 would definitely be the best time to do it!
>>>
>>> Thanks,
>>> John Halley Gotway
>>>
>>>
>>> On Wed, Mar 28, 2018 at 3:35 PM, nancy at ucar.edu via RT <
>>> met_help at ucar.edu>
>>> wrote:
>>>
>>> >
>>> > Wed Mar 28 15:35:25 2018: Request 84581 was acted upon.
>>> > Transaction: Ticket created by nancy at ucar.edu
>>> > Queue: met_help
>>> > Subject: Re: question about netcdf format files, dataset
337.0
>>> > Owner: Nobody
>>> > Requestors: nancy at ucar.edu
>>> > Status: new
>>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581
>>> >
>>> >
>>> >
>>> > hi tom,
>>> >
>>> > ok, thanks for the info. do you know who
>>> > we would check with at the DTC about options
>>> > for this tool? (or since met_help is cc'd on this
>>> > thread, perhaps they know?)
>>> >
>>> > thanks, and we'll check out the ascii format.
>>> > if you get some time to add the obs error to the
>>> > ascii files, please do let us know. that would
>>> > be great.
>>> >
>>> > again, thanks.
>>> > nancy
>>> >
>>> > On 3/28/18 3:19 PM, Thomas Cram wrote:
>>> > > Hi Nancy,
>>> > > according to the MET documentation, the prepbufr report type
and
>>> > observation error values cannot be added to the pb2nc output. I
>>> > > am copying MET support in this message, as they can take a
note of
>>> this
>>> > and possibly consider it for inclusion in a future release.
>>> > >
>>> > > In the meantime, you can choose ASCII output from the ds337.0
>>> subsetting
>>> > form, and this will include the PB report type in the
>>> > > output. It currently does not include the observation error
values,
>>> > however, but I will see if I can modify the code to include
>>> > > this sometime in the near future.
>>> > >
>>> > > - Tom
>>> > >
>>> > > On Wed, Mar 28, 2018 at 2:45 PM, nancy collins <nancy at ucar.edu
>>> <mailto:
>>> > nancy at ucar.edu>> wrote:
>>> > >
>>> > > tom,
>>> > >
>>> > > thanks for checking into this. that would be great news
>>> > > if these items were possible to add to the output files.
>>> > >
>>> > > n.
>>> > >
>>> > >
>>> > > On 3/28/18 9:13 AM, Thomas Cram wrote:
>>> > >> Hi Nancy,
>>> > >>
>>> > >> I use the pb2nc tool in the MET software to produce the
NetCDF
>>> > files (see https://dtcenter.org/met/users/index.php
>>> > >> <https://dtcenter.org/met/users/index.php>), and the
output is
>>> > defined by a configuration file passed to pb2nc. I'll take a
>>> > >> look at the documentation and see if it's possible to
include
>>> the
>>> > error values and PB report types in the output.
>>> > >>
>>> > >> - Tom
>>> > >>
>>> > >> On Tue, Mar 27, 2018 at 3:57 PM, Nancy Collins
<nancy at ucar.edu
>>> > <mailto:nancy at ucar.edu>> wrote:
>>> > >>
>>> > >> hello thomas,
>>> > >>
>>> > >> i'm looking at RDA dataset number 337.0 (NCEP
prepbufr
>>> > observation files).
>>> > >> this data is now also available in netcdf format,
which is
>>> > great.
>>> > >>
>>> > >> we downloaded an example file and most things we need
are
>>> there.
>>> > >> however, the netcdf file seems to be missing the
observation
>>> > error
>>> > >> values (present in the prepbufr files) and also the
prepbufr
>>> > report
>>> > >> types (again, present in the prepbufr files). while
the
>>> netcdf
>>> > files
>>> > >> do have 'grib codes' and 'message types', they don't
map
>>> > uniquely
>>> > >> to prepbufr report types, and we see no way to get
the obs
>>> > errors.
>>> > >> (note that this is not the quality control - it is
the
>>> error in
>>> > the same
>>> > >> units as the observation value. like a temperature
obs
>>> error
>>> > might
>>> > >> be 1 or 2 degrees K.)
>>> > >>
>>> > >> was this an intentional choice? can you tell us if
it's
>>> > possible to
>>> > >> get netcdf format files with these two additional
fields?
>>> are
>>> > there
>>> > >> options on the conversion tool that could be set?
>>> > >>
>>> > >> thanks for any information on how these files are
created
>>> and
>>> > >> who made the choices for what data is converted into
the
>>> > >> output netcdf files.
>>> > >>
>>> > >> nancy
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> > >> --
>>> > >> Thomas Cram
>>> > >> NCAR / CISL / DSS
>>> > >> tcram at ucar.edu <mailto:tcram at ucar.edu>
>>> > >> rda.ucar.edu <https://rda.ucar.edu>
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Thomas Cram
>>> > > NCAR / CISL / DSS
>>> > > tcram at ucar.edu <mailto:tcram at ucar.edu>
>>> > > rda.ucar.edu <https://rda.ucar.edu>
>>> >
>>> >
>>> >
>>>
>>>
>>
>
>
> --
> Thomas Cram
> NCAR / CISL / DSS
> tcram at ucar.edu
> rda.ucar.edu
>
------------------------------------------------
Subject: Re: question about netcdf format files, dataset 337.0
From: John Halley Gotway
Time: Wed Mar 28 17:02:43 2018
Nancy and Soyoung,
I’m out of the office the rest of this week and then will be attending
the
NCAR SEA conference next week.
You are of course welcome to modify the source code however you’d like
and
recompile.
But I do think it’d be a good idea for us to meet eventually and
review how
we’re planning on modifying the pb2nc file format at some point. I’d
like
to get your input to see if the changes would be useful to you.
Thanks
John
On Wed, Mar 28, 2018 at 4:45 PM nancy at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581 >
>
> hi john,
>
> i see i'm slow to this email thread. as soyoung already said,
> we access the obs error with ii=7. i have no idea why the docs
> say 'office note CATegory'.
>
> the converter code i'm looking at started life as a prepbufr dump
> program, and it called event type index 7 the observational error,
> and in fact it has lines like:
>
> toe = evns(7, lv, j2t, 3)
> qoe = evns(7, lv, j2q, 2)
> poe = evns(7, lv, 1, 1)
>
> from inspecting the values we get when we run the program
> they appear to be reasonable error values. but it's a little
> disquieting that it's not documented as such.
>
> n.
>
> On 3/28/18 4:28 PM, Thomas Cram wrote:
> > Hi Soyoung & John,
> >
> > my interpretation of "office note CATegory" is that it's the Data
Level
> Category (denoted by the mnemonic CAT), not the
> > observation error:
> >
> http://www.emc.ncep.noaa.gov/mmb/data_processing/table_local_await-
val.htm#0-08-193
> >
> > The observation error is new to me, too, but it looks like they
are
> defined by the Table B entries for Report Level data. My
> > reference for this is the prepbufr Table 1 mnemonics:
> >
> >
>
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
> > <
>
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm>
> >
> > Scroll down to "Table B entries for the report level data". This
is
> about 1/3 down the page. You will find observation error
> > mnemonics for each variable. For example, pressure observation
error is
> coded as "POE", temperature observation error is "TOE", etc.
> >
> > - Tom
> >
> >
> >
> > On Wed, Mar 28, 2018 at 4:21 PM, Soyoung Ha <syha at ucar.edu
<mailto:
> syha at ucar.edu>> wrote:
> >
> > Dear John Halley Gotway,
> >
> > This is Soyoung Ha from MMM. I'm the one who is actually using
the
> data and suggested to use the netcdf data converted thru
> > MET. So I quickly looked at met-7.0 source code for the part.
> >
> > In the READPB subroutine in met-
7.0/src/tools/other/pb2nc/readpb.f,
> I grabbed these comments:
> >
> > C* The 4-D array of data, EVNS ( ii, lv, jj, kk ), is
indexed
> > C* as follows:
> > C*
> > C* "ii" indexes the event data types; these consist of:
> > C* 1) OBservation
> > C* 2) Quality Mark
> > C* 3) Program Code
> > C* 4) Reason Code
> > C* 5) ForeCast value
> > C* 6) ANalysed value
> > C* 7) office note CATegory
> > C* "lv" indexes the levels of the report
> > C* "jj" indexes the event stacks
> > C* "kk" indexes the variable types (p,q,t,z,u,v)
> >
> > And the obs error we want here is 7) office note CATegory.
> > It looks like the code is reading all 7 "ii" indices (which
means
> obs error is also read), but it is just not stored in the
> > output netcdf file.
> > The output netcdf file Tom kindly converted for me using
MET5.0 has
> the output variable "obs_arr" which seems to be
> > extensible, would it be easy enough to pass this info as well?
> > Thanks a lot,
> >
> > Soyoung
> >
> >
> >
> >
> >
> > On Wed, Mar 28, 2018 at 4:07 PM, John Halley Gotway via RT <
> met_help at ucar.edu <mailto:met_help at ucar.edu>> wrote:
> >
> > Hi Tom and Nancy,
> >
> > You are correct, pb2nc doesn't currently include those
> quantities in the
> > output file. However, as luck would have it, we will be
> including the
> > prepbufr report type, input report type, and instrument
type in
> the pb2nc
> > output for the next version of met, met-7.1.
> >
> > We have some funding to enhance the ensemble-stat tool in
MET to
> do apply
> > observation error. However, the observation error can be
> different for
> > each report type and instrument type. So we need to pass
that
> info through
> > pb2nc for ensemble-stat to use in it's application of
> observation error.
> >
> > Since we're editing the pb2nc file, we're looking for ways
to
> simplify it's
> > structure and avoid repetition of strings whenever
possible.
> And we have
> > some good ideas on how to do that.
> >
> > Can you tell me more about the "obs error" from the
PREPBUFR
> file itself?
> > I'm not familiar with that field. Adding it during our
> development for
> > met-7.1 would definitely be the best time to do it!
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> > On Wed, Mar 28, 2018 at 3:35 PM, nancy at ucar.edu <mailto:
> nancy at ucar.edu> via RT <met_help at ucar.edu
<mailto:met_help at ucar.edu>>
> > wrote:
> >
> > >
> > > Wed Mar 28 15:35:25 2018: Request 84581 was acted upon.
> > > Transaction: Ticket created by nancy at ucar.edu <mailto:
> nancy at ucar.edu>
> > > Queue: met_help
> > > Subject: Re: question about netcdf format files,
dataset
> 337.0
> > > Owner: Nobody
> > > Requestors: nancy at ucar.edu <mailto:nancy at ucar.edu>
> > > Status: new
> > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581
> > <https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581>
>
> > >
> > >
> > > hi tom,
> > >
> > > ok, thanks for the info. do you know who
> > > we would check with at the DTC about options
> > > for this tool? (or since met_help is cc'd on this
> > > thread, perhaps they know?)
> > >
> > > thanks, and we'll check out the ascii format.
> > > if you get some time to add the obs error to the
> > > ascii files, please do let us know. that would
> > > be great.
> > >
> > > again, thanks.
> > > nancy
> > >
> > > On 3/28/18 3:19 PM, Thomas Cram wrote:
> > > > Hi Nancy,
> > > > according to the MET documentation, the prepbufr
report type
> and
> > > observation error values cannot be added to the pb2nc
output.
> I
> > > > am copying MET support in this message, as they can
take a
> note of this
> > > and possibly consider it for inclusion in a future
release.
> > > >
> > > > In the meantime, you can choose ASCII output from the
> ds337.0 subsetting
> > > form, and this will include the PB report type in the
> > > > output. It currently does not include the observation
error
> values,
> > > however, but I will see if I can modify the code to
include
> > > > this sometime in the near future.
> > > >
> > > > - Tom
> > > >
> > > > On Wed, Mar 28, 2018 at 2:45 PM, nancy collins <
> nancy at ucar.edu <mailto:nancy at ucar.edu> <mailto:
> > > nancy at ucar.edu <mailto:nancy at ucar.edu>>> wrote:
> > > >
> > > > tom,
> > > >
> > > > thanks for checking into this. that would be great
news
> > > > if these items were possible to add to the output
files.
> > > >
> > > > n.
> > > >
> > > >
> > > > On 3/28/18 9:13 AM, Thomas Cram wrote:
> > > >> Hi Nancy,
> > > >>
> > > >> I use the pb2nc tool in the MET software to
produce the
> NetCDF
> > > files (see https://dtcenter.org/met/users/index.php <
> https://dtcenter.org/met/users/index.php>
> > > >> <https://dtcenter.org/met/users/index.php <
> https://dtcenter.org/met/users/index.php>>), and the output is
> > > defined by a configuration file passed to pb2nc. I'll
take a
> > > >> look at the documentation and see if it's
possible to
> include the
> > > error values and PB report types in the output.
> > > >>
> > > >> - Tom
> > > >>
> > > >> On Tue, Mar 27, 2018 at 3:57 PM, Nancy Collins <
> nancy at ucar.edu <mailto:nancy at ucar.edu>
> > > <mailto:nancy at ucar.edu <mailto:nancy at ucar.edu>>> wrote:
> > > >>
> > > >> hello thomas,
> > > >>
> > > >> i'm looking at RDA dataset number 337.0 (NCEP
> prepbufr
> > > observation files).
> > > >> this data is now also available in netcdf
format,
> which is
> > > great.
> > > >>
> > > >> we downloaded an example file and most things
we
> need are there.
> > > >> however, the netcdf file seems to be missing
the
> observation
> > > error
> > > >> values (present in the prepbufr files) and
also the
> prepbufr
> > > report
> > > >> types (again, present in the prepbufr files).
> while the netcdf
> > > files
> > > >> do have 'grib codes' and 'message types',
they
> don't map
> > > uniquely
> > > >> to prepbufr report types, and we see no way
to get
> the obs
> > > errors.
> > > >> (note that this is not the quality control -
it is
> the error in
> > > the same
> > > >> units as the observation value. like a
temperature
> obs error
> > > might
> > > >> be 1 or 2 degrees K.)
> > > >>
> > > >> was this an intentional choice? can you tell
us if
> it's
> > > possible to
> > > >> get netcdf format files with these two
additional
> fields? are
> > > there
> > > >> options on the conversion tool that could be
set?
> > > >>
> > > >> thanks for any information on how these files
are
> created and
> > > >> who made the choices for what data is
converted
> into the
> > > >> output netcdf files.
> > > >>
> > > >> nancy
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Thomas Cram
> > > >> NCAR / CISL / DSS
> > > >> tcram at ucar.edu <mailto:tcram at ucar.edu> <mailto:
> tcram at ucar.edu <mailto:tcram at ucar.edu>>
> > > >> rda.ucar.edu <http://rda.ucar.edu>
<https://rda.ucar.edu>
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thomas Cram
> > > > NCAR / CISL / DSS
> > > > tcram at ucar.edu <mailto:tcram at ucar.edu> <mailto:
> tcram at ucar.edu <mailto:tcram at ucar.edu>>
> > > > rda.ucar.edu <http://rda.ucar.edu>
<https://rda.ucar.edu>
> > >
> > >
> > >
> >
> >
> >
> >
> >
> > --
> > Thomas Cram
> > NCAR / CISL / DSS
> > tcram at ucar.edu <mailto:tcram at ucar.edu>
> > rda.ucar.edu <https://rda.ucar.edu>
>
>
>
------------------------------------------------
Subject: Re: question about netcdf format files, dataset 337.0
From: Thomas Cram
Time: Wed Mar 28 17:08:53 2018
Hi Soyoung,
thanks for the clarification. The decoder I'm working with only uses
the
variable obstr, but I'll see if I can include oestr in my ascii
output.
- Tom
On Wed, Mar 28, 2018 at 5:02 PM, Soyoung Ha <syha at ucar.edu> wrote:
> Tom,
>
> As Nancy pointed out, we've used the 7th index as obs error, which
might
> be adjusted later when needed.
> Even in the GSI prepbufr_decoding program, they define the indices
as
> below.
>
> character(80):: obstr='POB QOB TOB ZOB UOB VOB PWO CAT PRSS'
> character(80):: qcstr='PQM QQM TQM ZQM WQM NUL PWQ '
> character(80):: oestr='POE QOE TOE NUL WOE NUL PWE '
> ...
> call ufbint(unit_in,obs,mxmn,mxlv,iret,obstr)
> call ufbint(unit_in,oer,mxmn,mxlv,iret,oestr)
> call ufbint(unit_in,qcf,mxmn,mxlv,iret,qcstr)
> ...
>
> The corresponding string in the READPB module in MET (lines 62-68 in
> pb2nc/readpb.f) looks as below.
>
> CHARACTER*(MXSTRL) ostr ( MXR8VT )
> + / 'POB PQM PPC PRC PFC PAN CAT',
> + 'QOB QQM QPC QRC QFC QAN CAT',
> + 'TOB TQM TPC TRC TFC TAN CAT',
> + 'ZOB ZQM ZPC ZRC ZFC ZAN CAT',
> + 'UOB WQM WPC WRC UFC UAN CAT',
> + 'VOB WQM WPC WRC VFC VAN CAT' /
>
> Comparing these two, we can figure out that the last column of
"ostr"
> (which is all "CAT") can be replaced with "oestr" that I listed in
the GSI
> program above. So this column is considered to be obs error - POE,
QOE,
> TOE, etc.
> Again, the error can be adjusted later (before getting into the
> assimilation system), but we want to read it as a reference or a
starting
> value.
> Thanks,
>
> Soyoung
>
>
>
>
> On Wed, Mar 28, 2018 at 4:28 PM, Thomas Cram <tcram at ucar.edu> wrote:
>
>> Hi Soyoung & John,
>>
>> my interpretation of "office note CATegory" is that it's the Data
Level
>> Category (denoted by the mnemonic CAT), not the observation error:
>> http://www.emc.ncep.noaa.gov/mmb/data_processing/table_local
>> _await-val.htm#0-08-193
>>
>> The observation error is new to me, too, but it looks like they are
>> defined by the Table B entries for Report Level data. My reference
for
>> this is the prepbufr Table 1 mnemonics:
>>
>>
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
>>
>> Scroll down to "Table B entries for the report level data". This
is
>> about 1/3 down the page. You will find observation error mnemonics
for
>> each variable. For example, pressure observation error is coded as
"POE",
>> temperature observation error is "TOE", etc.
>>
>> - Tom
>>
>>
>>
>> On Wed, Mar 28, 2018 at 4:21 PM, Soyoung Ha <syha at ucar.edu> wrote:
>>
>>> Dear John Halley Gotway,
>>>
>>> This is Soyoung Ha from MMM. I'm the one who is actually using the
data
>>> and suggested to use the netcdf data converted thru MET. So I
quickly
>>> looked at met-7.0 source code for the part.
>>>
>>> In the READPB subroutine in met-
7.0/src/tools/other/pb2nc/readpb.f, I
>>> grabbed these comments:
>>>
>>> C* The 4-D array of data, EVNS ( ii, lv, jj, kk ), is indexed
>>> C* as follows:
>>> C*
>>> C* "ii" indexes the event data types; these consist of:
>>> C* 1) OBservation
>>> C* 2) Quality Mark
>>> C* 3) Program Code
>>> C* 4) Reason Code
>>> C* 5) ForeCast value
>>> C* 6) ANalysed value
>>> C* 7) office note CATegory
>>> C* "lv" indexes the levels of the report
>>> C* "jj" indexes the event stacks
>>> C* "kk" indexes the variable types (p,q,t,z,u,v)
>>>
>>> And the obs error we want here is 7) office note CATegory.
>>> It looks like the code is reading all 7 "ii" indices (which means
obs
>>> error is also read), but it is just not stored in the output
netcdf file.
>>> The output netcdf file Tom kindly converted for me using MET5.0
has the
>>> output variable "obs_arr" which seems to be extensible, would it
be easy
>>> enough to pass this info as well?
>>> Thanks a lot,
>>>
>>> Soyoung
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Mar 28, 2018 at 4:07 PM, John Halley Gotway via RT <
>>> met_help at ucar.edu> wrote:
>>>
>>>> Hi Tom and Nancy,
>>>>
>>>> You are correct, pb2nc doesn't currently include those quantities
in the
>>>> output file. However, as luck would have it, we will be
including the
>>>> prepbufr report type, input report type, and instrument type in
the
>>>> pb2nc
>>>> output for the next version of met, met-7.1.
>>>>
>>>> We have some funding to enhance the ensemble-stat tool in MET to
do
>>>> apply
>>>> observation error. However, the observation error can be
different for
>>>> each report type and instrument type. So we need to pass that
info
>>>> through
>>>> pb2nc for ensemble-stat to use in it's application of observation
error.
>>>>
>>>> Since we're editing the pb2nc file, we're looking for ways to
simplify
>>>> it's
>>>> structure and avoid repetition of strings whenever possible. And
we
>>>> have
>>>> some good ideas on how to do that.
>>>>
>>>> Can you tell me more about the "obs error" from the PREPBUFR file
>>>> itself?
>>>> I'm not familiar with that field. Adding it during our
development for
>>>> met-7.1 would definitely be the best time to do it!
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>>
>>>>
>>>> On Wed, Mar 28, 2018 at 3:35 PM, nancy at ucar.edu via RT <
>>>> met_help at ucar.edu>
>>>> wrote:
>>>>
>>>> >
>>>> > Wed Mar 28 15:35:25 2018: Request 84581 was acted upon.
>>>> > Transaction: Ticket created by nancy at ucar.edu
>>>> > Queue: met_help
>>>> > Subject: Re: question about netcdf format files, dataset
337.0
>>>> > Owner: Nobody
>>>> > Requestors: nancy at ucar.edu
>>>> > Status: new
>>>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581
>>>> >
>>>> >
>>>> >
>>>> > hi tom,
>>>> >
>>>> > ok, thanks for the info. do you know who
>>>> > we would check with at the DTC about options
>>>> > for this tool? (or since met_help is cc'd on this
>>>> > thread, perhaps they know?)
>>>> >
>>>> > thanks, and we'll check out the ascii format.
>>>> > if you get some time to add the obs error to the
>>>> > ascii files, please do let us know. that would
>>>> > be great.
>>>> >
>>>> > again, thanks.
>>>> > nancy
>>>> >
>>>> > On 3/28/18 3:19 PM, Thomas Cram wrote:
>>>> > > Hi Nancy,
>>>> > > according to the MET documentation, the prepbufr report type
and
>>>> > observation error values cannot be added to the pb2nc output.
I
>>>> > > am copying MET support in this message, as they can take a
note of
>>>> this
>>>> > and possibly consider it for inclusion in a future release.
>>>> > >
>>>> > > In the meantime, you can choose ASCII output from the ds337.0
>>>> subsetting
>>>> > form, and this will include the PB report type in the
>>>> > > output. It currently does not include the observation error
values,
>>>> > however, but I will see if I can modify the code to include
>>>> > > this sometime in the near future.
>>>> > >
>>>> > > - Tom
>>>> > >
>>>> > > On Wed, Mar 28, 2018 at 2:45 PM, nancy collins
<nancy at ucar.edu
>>>> <mailto:
>>>> > nancy at ucar.edu>> wrote:
>>>> > >
>>>> > > tom,
>>>> > >
>>>> > > thanks for checking into this. that would be great news
>>>> > > if these items were possible to add to the output files.
>>>> > >
>>>> > > n.
>>>> > >
>>>> > >
>>>> > > On 3/28/18 9:13 AM, Thomas Cram wrote:
>>>> > >> Hi Nancy,
>>>> > >>
>>>> > >> I use the pb2nc tool in the MET software to produce the
NetCDF
>>>> > files (see https://dtcenter.org/met/users/index.php
>>>> > >> <https://dtcenter.org/met/users/index.php>), and the
output is
>>>> > defined by a configuration file passed to pb2nc. I'll take a
>>>> > >> look at the documentation and see if it's possible to
include
>>>> the
>>>> > error values and PB report types in the output.
>>>> > >>
>>>> > >> - Tom
>>>> > >>
>>>> > >> On Tue, Mar 27, 2018 at 3:57 PM, Nancy Collins
<nancy at ucar.edu
>>>> > <mailto:nancy at ucar.edu>> wrote:
>>>> > >>
>>>> > >> hello thomas,
>>>> > >>
>>>> > >> i'm looking at RDA dataset number 337.0 (NCEP
prepbufr
>>>> > observation files).
>>>> > >> this data is now also available in netcdf format,
which is
>>>> > great.
>>>> > >>
>>>> > >> we downloaded an example file and most things we
need are
>>>> there.
>>>> > >> however, the netcdf file seems to be missing the
>>>> observation
>>>> > error
>>>> > >> values (present in the prepbufr files) and also the
>>>> prepbufr
>>>> > report
>>>> > >> types (again, present in the prepbufr files). while
the
>>>> netcdf
>>>> > files
>>>> > >> do have 'grib codes' and 'message types', they don't
map
>>>> > uniquely
>>>> > >> to prepbufr report types, and we see no way to get
the obs
>>>> > errors.
>>>> > >> (note that this is not the quality control - it is
the
>>>> error in
>>>> > the same
>>>> > >> units as the observation value. like a temperature
obs
>>>> error
>>>> > might
>>>> > >> be 1 or 2 degrees K.)
>>>> > >>
>>>> > >> was this an intentional choice? can you tell us if
it's
>>>> > possible to
>>>> > >> get netcdf format files with these two additional
fields?
>>>> are
>>>> > there
>>>> > >> options on the conversion tool that could be set?
>>>> > >>
>>>> > >> thanks for any information on how these files are
created
>>>> and
>>>> > >> who made the choices for what data is converted into
the
>>>> > >> output netcdf files.
>>>> > >>
>>>> > >> nancy
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >> --
>>>> > >> Thomas Cram
>>>> > >> NCAR / CISL / DSS
>>>> > >> tcram at ucar.edu <mailto:tcram at ucar.edu>
>>>> > >> rda.ucar.edu <https://rda.ucar.edu>
>>>> > >
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Thomas Cram
>>>> > > NCAR / CISL / DSS
>>>> > > tcram at ucar.edu <mailto:tcram at ucar.edu>
>>>> > > rda.ucar.edu <https://rda.ucar.edu>
>>>> >
>>>> >
>>>> >
>>>>
>>>>
>>>
>>
>>
>> --
>> Thomas Cram
>> NCAR / CISL / DSS
>> tcram at ucar.edu
>> rda.ucar.edu
>>
>
>
--
Thomas Cram
NCAR / CISL / DSS
tcram at ucar.edu
rda.ucar.edu
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #84581] Re: question about netcdf format files, dataset 337.0
From: Howard Soh
Time: Wed Mar 28 18:05:06 2018
The output variables for BUFR input are configurable and defined as
"obs_bufr_var" at the PB2NCConfig. The same configuration is used to
add/remove the output variables for PREPBURF input.
The CAT (PREPBUFR DATA LEVEL CATEGORY) variable will be added into the
pb2nc output with the PB2NC configuration.
The default output variable list at PB2NCConfig_default:
obs_bufr_var = [ "QOB", "TOB", "ZOB", "UOB", "VOB" ];
- Add "CAT" to obs_bufr_var
obs_bufr_var = [ "QOB", "TOB", "ZOB", "UOB", "VOB", "CAT" ];
Cheers,
Howard
On 3/28/2018 5:08 PM, Thomas Cram via RT wrote:
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581 >
>
> Hi Soyoung,
> thanks for the clarification. The decoder I'm working with only
uses the
> variable obstr, but I'll see if I can include oestr in my ascii
output.
>
> - Tom
>
> On Wed, Mar 28, 2018 at 5:02 PM, Soyoung Ha <syha at ucar.edu> wrote:
>
>> Tom,
>>
>> As Nancy pointed out, we've used the 7th index as obs error, which
might
>> be adjusted later when needed.
>> Even in the GSI prepbufr_decoding program, they define the indices
as
>> below.
>>
>> character(80):: obstr='POB QOB TOB ZOB UOB VOB PWO CAT PRSS'
>> character(80):: qcstr='PQM QQM TQM ZQM WQM NUL PWQ '
>> character(80):: oestr='POE QOE TOE NUL WOE NUL PWE '
>> ...
>> call ufbint(unit_in,obs,mxmn,mxlv,iret,obstr)
>> call ufbint(unit_in,oer,mxmn,mxlv,iret,oestr)
>> call ufbint(unit_in,qcf,mxmn,mxlv,iret,qcstr)
>> ...
>>
>> The corresponding string in the READPB module in MET (lines 62-68
in
>> pb2nc/readpb.f) looks as below.
>>
>> CHARACTER*(MXSTRL) ostr ( MXR8VT )
>> + / 'POB PQM PPC PRC PFC PAN CAT',
>> + 'QOB QQM QPC QRC QFC QAN CAT',
>> + 'TOB TQM TPC TRC TFC TAN CAT',
>> + 'ZOB ZQM ZPC ZRC ZFC ZAN CAT',
>> + 'UOB WQM WPC WRC UFC UAN CAT',
>> + 'VOB WQM WPC WRC VFC VAN CAT' /
>>
>> Comparing these two, we can figure out that the last column of
"ostr"
>> (which is all "CAT") can be replaced with "oestr" that I listed in
the GSI
>> program above. So this column is considered to be obs error - POE,
QOE,
>> TOE, etc.
>> Again, the error can be adjusted later (before getting into the
>> assimilation system), but we want to read it as a reference or a
starting
>> value.
>> Thanks,
>>
>> Soyoung
>>
>>
>>
>>
>> On Wed, Mar 28, 2018 at 4:28 PM, Thomas Cram <tcram at ucar.edu>
wrote:
>>
>>> Hi Soyoung & John,
>>>
>>> my interpretation of "office note CATegory" is that it's the Data
Level
>>> Category (denoted by the mnemonic CAT), not the observation error:
>>> http://www.emc.ncep.noaa.gov/mmb/data_processing/table_local
>>> _await-val.htm#0-08-193
>>>
>>> The observation error is new to me, too, but it looks like they
are
>>> defined by the Table B entries for Report Level data. My
reference for
>>> this is the prepbufr Table 1 mnemonics:
>>>
>>>
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
>>>
>>> Scroll down to "Table B entries for the report level data". This
is
>>> about 1/3 down the page. You will find observation error
mnemonics for
>>> each variable. For example, pressure observation error is coded
as "POE",
>>> temperature observation error is "TOE", etc.
>>>
>>> - Tom
>>>
>>>
>>>
>>> On Wed, Mar 28, 2018 at 4:21 PM, Soyoung Ha <syha at ucar.edu> wrote:
>>>
>>>> Dear John Halley Gotway,
>>>>
>>>> This is Soyoung Ha from MMM. I'm the one who is actually using
the data
>>>> and suggested to use the netcdf data converted thru MET. So I
quickly
>>>> looked at met-7.0 source code for the part.
>>>>
>>>> In the READPB subroutine in met-
7.0/src/tools/other/pb2nc/readpb.f, I
>>>> grabbed these comments:
>>>>
>>>> C* The 4-D array of data, EVNS ( ii, lv, jj, kk ), is
indexed
>>>> C* as follows:
>>>> C*
>>>> C* "ii" indexes the event data types; these consist of:
>>>> C* 1) OBservation
>>>> C* 2) Quality Mark
>>>> C* 3) Program Code
>>>> C* 4) Reason Code
>>>> C* 5) ForeCast value
>>>> C* 6) ANalysed value
>>>> C* 7) office note CATegory
>>>> C* "lv" indexes the levels of the report
>>>> C* "jj" indexes the event stacks
>>>> C* "kk" indexes the variable types (p,q,t,z,u,v)
>>>>
>>>> And the obs error we want here is 7) office note CATegory.
>>>> It looks like the code is reading all 7 "ii" indices (which means
obs
>>>> error is also read), but it is just not stored in the output
netcdf file.
>>>> The output netcdf file Tom kindly converted for me using MET5.0
has the
>>>> output variable "obs_arr" which seems to be extensible, would it
be easy
>>>> enough to pass this info as well?
>>>> Thanks a lot,
>>>>
>>>> Soyoung
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Mar 28, 2018 at 4:07 PM, John Halley Gotway via RT <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>>> Hi Tom and Nancy,
>>>>>
>>>>> You are correct, pb2nc doesn't currently include those
quantities in the
>>>>> output file. However, as luck would have it, we will be
including the
>>>>> prepbufr report type, input report type, and instrument type in
the
>>>>> pb2nc
>>>>> output for the next version of met, met-7.1.
>>>>>
>>>>> We have some funding to enhance the ensemble-stat tool in MET to
do
>>>>> apply
>>>>> observation error. However, the observation error can be
different for
>>>>> each report type and instrument type. So we need to pass that
info
>>>>> through
>>>>> pb2nc for ensemble-stat to use in it's application of
observation error.
>>>>>
>>>>> Since we're editing the pb2nc file, we're looking for ways to
simplify
>>>>> it's
>>>>> structure and avoid repetition of strings whenever possible.
And we
>>>>> have
>>>>> some good ideas on how to do that.
>>>>>
>>>>> Can you tell me more about the "obs error" from the PREPBUFR
file
>>>>> itself?
>>>>> I'm not familiar with that field. Adding it during our
development for
>>>>> met-7.1 would definitely be the best time to do it!
>>>>>
>>>>> Thanks,
>>>>> John Halley Gotway
>>>>>
>>>>>
>>>>> On Wed, Mar 28, 2018 at 3:35 PM, nancy at ucar.edu via RT <
>>>>> met_help at ucar.edu>
>>>>> wrote:
>>>>>
>>>>>> Wed Mar 28 15:35:25 2018: Request 84581 was acted upon.
>>>>>> Transaction: Ticket created by nancy at ucar.edu
>>>>>> Queue: met_help
>>>>>> Subject: Re: question about netcdf format files, dataset
337.0
>>>>>> Owner: Nobody
>>>>>> Requestors: nancy at ucar.edu
>>>>>> Status: new
>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581
>>>>>>
>>>>>>
>>>>>>
>>>>>> hi tom,
>>>>>>
>>>>>> ok, thanks for the info. do you know who
>>>>>> we would check with at the DTC about options
>>>>>> for this tool? (or since met_help is cc'd on this
>>>>>> thread, perhaps they know?)
>>>>>>
>>>>>> thanks, and we'll check out the ascii format.
>>>>>> if you get some time to add the obs error to the
>>>>>> ascii files, please do let us know. that would
>>>>>> be great.
>>>>>>
>>>>>> again, thanks.
>>>>>> nancy
>>>>>>
>>>>>> On 3/28/18 3:19 PM, Thomas Cram wrote:
>>>>>>> Hi Nancy,
>>>>>>> according to the MET documentation, the prepbufr report type
and
>>>>>> observation error values cannot be added to the pb2nc output.
I
>>>>>>> am copying MET support in this message, as they can take a
note of
>>>>> this
>>>>>> and possibly consider it for inclusion in a future release.
>>>>>>> In the meantime, you can choose ASCII output from the ds337.0
>>>>> subsetting
>>>>>> form, and this will include the PB report type in the
>>>>>>> output. It currently does not include the observation error
values,
>>>>>> however, but I will see if I can modify the code to include
>>>>>>> this sometime in the near future.
>>>>>>>
>>>>>>> - Tom
>>>>>>>
>>>>>>> On Wed, Mar 28, 2018 at 2:45 PM, nancy collins <nancy at ucar.edu
>>>>> <mailto:
>>>>>> nancy at ucar.edu>> wrote:
>>>>>>> tom,
>>>>>>>
>>>>>>> thanks for checking into this. that would be great news
>>>>>>> if these items were possible to add to the output files.
>>>>>>>
>>>>>>> n.
>>>>>>>
>>>>>>>
>>>>>>> On 3/28/18 9:13 AM, Thomas Cram wrote:
>>>>>>>> Hi Nancy,
>>>>>>>>
>>>>>>>> I use the pb2nc tool in the MET software to produce the
NetCDF
>>>>>> files (see https://dtcenter.org/met/users/index.php
>>>>>>>> <https://dtcenter.org/met/users/index.php>), and the
output is
>>>>>> defined by a configuration file passed to pb2nc. I'll take a
>>>>>>>> look at the documentation and see if it's possible to
include
>>>>> the
>>>>>> error values and PB report types in the output.
>>>>>>>> - Tom
>>>>>>>>
>>>>>>>> On Tue, Mar 27, 2018 at 3:57 PM, Nancy Collins
<nancy at ucar.edu
>>>>>> <mailto:nancy at ucar.edu>> wrote:
>>>>>>>> hello thomas,
>>>>>>>>
>>>>>>>> i'm looking at RDA dataset number 337.0 (NCEP
prepbufr
>>>>>> observation files).
>>>>>>>> this data is now also available in netcdf format,
which is
>>>>>> great.
>>>>>>>> we downloaded an example file and most things we
need are
>>>>> there.
>>>>>>>> however, the netcdf file seems to be missing the
>>>>> observation
>>>>>> error
>>>>>>>> values (present in the prepbufr files) and also the
>>>>> prepbufr
>>>>>> report
>>>>>>>> types (again, present in the prepbufr files). while
the
>>>>> netcdf
>>>>>> files
>>>>>>>> do have 'grib codes' and 'message types', they don't
map
>>>>>> uniquely
>>>>>>>> to prepbufr report types, and we see no way to get
the obs
>>>>>> errors.
>>>>>>>> (note that this is not the quality control - it is
the
>>>>> error in
>>>>>> the same
>>>>>>>> units as the observation value. like a temperature
obs
>>>>> error
>>>>>> might
>>>>>>>> be 1 or 2 degrees K.)
>>>>>>>>
>>>>>>>> was this an intentional choice? can you tell us if
it's
>>>>>> possible to
>>>>>>>> get netcdf format files with these two additional
fields?
>>>>> are
>>>>>> there
>>>>>>>> options on the conversion tool that could be set?
>>>>>>>>
>>>>>>>> thanks for any information on how these files are
created
>>>>> and
>>>>>>>> who made the choices for what data is converted into
the
>>>>>>>> output netcdf files.
>>>>>>>>
>>>>>>>> nancy
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thomas Cram
>>>>>>>> NCAR / CISL / DSS
>>>>>>>> tcram at ucar.edu <mailto:tcram at ucar.edu>
>>>>>>>> rda.ucar.edu <https://rda.ucar.edu>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thomas Cram
>>>>>>> NCAR / CISL / DSS
>>>>>>> tcram at ucar.edu <mailto:tcram at ucar.edu>
>>>>>>> rda.ucar.edu <https://rda.ucar.edu>
>>>>>>
>>>>>>
>>>>>
>>>
>>> --
>>> Thomas Cram
>>> NCAR / CISL / DSS
>>> tcram at ucar.edu
>>> rda.ucar.edu
>>>
>>
>
------------------------------------------------
Subject: Re: question about netcdf format files, dataset 337.0
From: Thomas Cram
Time: Wed May 23 13:59:07 2018
Hi Nancy,
as I mentioned to you at the seminar this morning, I have updated my
PrepBUFR decoder to include observation error in the ASCII output.
The
source code and instructions are available in the following public
Github
repository:
https://github.com/NCAR/rda-prepbufr-decode
Adding this to the output was really quite simple. All I needed to do
was
add 'POE, QOE, TOE, etc.' to the 'ostr' character string in the readpb
subroutine, and accommodate this change in the EVNS array dimensions.
Sample output from this program is available in my scratch directory:
/glade2/scratch2/tcram/readpb
I'm working on deploying this code to the ds337.0 subsetting service,
and
hope to have it in operation within the next couple of weeks.
Let me know if you have additional questions,
- Tom
On Thu, Mar 29, 2018 at 8:34 AM, Nancy Collins <nancy at ucar.edu> wrote:
> howard, john, tom --
>
> this is great news. i'd like to say a very sincere thank you
> for your quick responses to our requests.
>
> the DART data assimilation software has its own custom
> prepbufr reader, but it's older code written by someone
> who is no longer with the project. prepbufr format is
> completely impenetrable to most of us. we've wanted
> a clearer, simpler way to convert observations for ingestion
> into our software for a long time.
>
> if these changes are indeed made to the pb2nc converter
> we can easily read netcdf (that, i understand!) and write
> the format we need. this would make a much easier
> workflow for converting observations for our users
> (and for me!!).
>
> soyoung and i would be happy to be alpha or beta testers
> of anything you would like us to try out.
>
> again, thanks.
> n.
>
>
> On 3/28/18 6:10 PM, Howard Soh via RT wrote:
>
> The output variables for BUFR input are configurable and defined as
> "obs_bufr_var" at the PB2NCConfig. The same configuration is used to
> add/remove the output variables for PREPBURF input.
> The CAT (PREPBUFR DATA LEVEL CATEGORY) variable will be added into
the
> pb2nc output with the PB2NC configuration.
>
> The default output variable list at PB2NCConfig_default:
>
> obs_bufr_var = [ "QOB", "TOB", "ZOB", "UOB", "VOB" ];
>
> - Add "CAT" to obs_bufr_var
>
> obs_bufr_var = [ "QOB", "TOB", "ZOB", "UOB", "VOB", "CAT" ];
>
> Cheers,
> Howard
>
>
> On 3/28/2018 5:08 PM, Thomas Cram via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581 >
>
> Hi Soyoung,
> thanks for the clarification. The decoder I'm working with only
uses the
> variable obstr, but I'll see if I can include oestr in my ascii
output.
>
> - Tom
>
> On Wed, Mar 28, 2018 at 5:02 PM, Soyoung Ha <syha at ucar.edu>
<syha at ucar.edu> wrote:
>
>
> Tom,
>
> As Nancy pointed out, we've used the 7th index as obs error, which
might
> be adjusted later when needed.
> Even in the GSI prepbufr_decoding program, they define the indices
as
> below.
>
> character(80):: obstr='POB QOB TOB ZOB UOB VOB PWO CAT PRSS'
> character(80):: qcstr='PQM QQM TQM ZQM WQM NUL PWQ '
> character(80):: oestr='POE QOE TOE NUL WOE NUL PWE '
> ...
> call ufbint(unit_in,obs,mxmn,mxlv,iret,obstr)
> call ufbint(unit_in,oer,mxmn,mxlv,iret,oestr)
> call ufbint(unit_in,qcf,mxmn,mxlv,iret,qcstr)
> ...
>
> The corresponding string in the READPB module in MET (lines 62-68 in
> pb2nc/readpb.f) looks as below.
>
> CHARACTER*(MXSTRL) ostr ( MXR8VT )
> + / 'POB PQM PPC PRC PFC PAN CAT',
> + 'QOB QQM QPC QRC QFC QAN CAT',
> + 'TOB TQM TPC TRC TFC TAN CAT',
> + 'ZOB ZQM ZPC ZRC ZFC ZAN CAT',
> + 'UOB WQM WPC WRC UFC UAN CAT',
> + 'VOB WQM WPC WRC VFC VAN CAT' /
>
> Comparing these two, we can figure out that the last column of
"ostr"
> (which is all "CAT") can be replaced with "oestr" that I listed in
the GSI
> program above. So this column is considered to be obs error - POE,
QOE,
> TOE, etc.
> Again, the error can be adjusted later (before getting into the
> assimilation system), but we want to read it as a reference or a
starting
> value.
> Thanks,
>
> Soyoung
>
>
>
>
> On Wed, Mar 28, 2018 at 4:28 PM, Thomas Cram <tcram at ucar.edu>
<tcram at ucar.edu> wrote:
>
>
> Hi Soyoung & John,
>
> my interpretation of "office note CATegory" is that it's the Data
Level
> Category (denoted by the mnemonic CAT), not the observation
error:http://www.emc.ncep.noaa.gov/mmb/data_processing/table_local
> _await-val.htm#0-08-193
>
> The observation error is new to me, too, but it looks like they are
> defined by the Table B entries for Report Level data. My reference
for
> this is the prepbufr Table 1 mnemonics:
>
http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_1.htm
>
> Scroll down to "Table B entries for the report level data". This is
> about 1/3 down the page. You will find observation error mnemonics
for
> each variable. For example, pressure observation error is coded as
"POE",
> temperature observation error is "TOE", etc.
>
> - Tom
>
>
>
> On Wed, Mar 28, 2018 at 4:21 PM, Soyoung Ha <syha at ucar.edu>
<syha at ucar.edu> wrote:
>
>
> Dear John Halley Gotway,
>
> This is Soyoung Ha from MMM. I'm the one who is actually using the
data
> and suggested to use the netcdf data converted thru MET. So I
quickly
> looked at met-7.0 source code for the part.
>
> In the READPB subroutine in met-7.0/src/tools/other/pb2nc/readpb.f,
I
> grabbed these comments:
>
> C* The 4-D array of data, EVNS ( ii, lv, jj, kk ), is indexed
> C* as follows:
> C*
> C* "ii" indexes the event data types; these consist of:
> C* 1) OBservation
> C* 2) Quality Mark
> C* 3) Program Code
> C* 4) Reason Code
> C* 5) ForeCast value
> C* 6) ANalysed value
> C* 7) office note CATegory
> C* "lv" indexes the levels of the report
> C* "jj" indexes the event stacks
> C* "kk" indexes the variable types (p,q,t,z,u,v)
>
> And the obs error we want here is 7) office note CATegory.
> It looks like the code is reading all 7 "ii" indices (which means
obs
> error is also read), but it is just not stored in the output netcdf
file.
> The output netcdf file Tom kindly converted for me using MET5.0 has
the
> output variable "obs_arr" which seems to be extensible, would it be
easy
> enough to pass this info as well?
> Thanks a lot,
>
> Soyoung
>
>
>
>
>
> On Wed, Mar 28, 2018 at 4:07 PM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
>
> Hi Tom and Nancy,
>
> You are correct, pb2nc doesn't currently include those quantities in
the
> output file. However, as luck would have it, we will be including
the
> prepbufr report type, input report type, and instrument type in the
> pb2nc
> output for the next version of met, met-7.1.
>
> We have some funding to enhance the ensemble-stat tool in MET to do
> apply
> observation error. However, the observation error can be different
for
> each report type and instrument type. So we need to pass that info
> through
> pb2nc for ensemble-stat to use in it's application of observation
error.
>
> Since we're editing the pb2nc file, we're looking for ways to
simplify
> it's
> structure and avoid repetition of strings whenever possible. And we
> have
> some good ideas on how to do that.
>
> Can you tell me more about the "obs error" from the PREPBUFR file
> itself?
> I'm not familiar with that field. Adding it during our development
for
> met-7.1 would definitely be the best time to do it!
>
> Thanks,
> John Halley Gotway
>
>
> On Wed, Mar 28, 2018 at 3:35 PM, nancy at ucar.edu via RT
<met_help at ucar.edu>
> wrote:
>
>
> Wed Mar 28 15:35:25 2018: Request 84581 was acted upon.
> Transaction: Ticket created by nancy at ucar.edu
> Queue: met_help
> Subject: Re: question about netcdf format files, dataset 337.0
> Owner: Nobody
> Requestors: nancy at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84581
>
>
>
> hi tom,
>
> ok, thanks for the info. do you know who
> we would check with at the DTC about options
> for this tool? (or since met_help is cc'd on this
> thread, perhaps they know?)
>
> thanks, and we'll check out the ascii format.
> if you get some time to add the obs error to the
> ascii files, please do let us know. that would
> be great.
>
> again, thanks.
> nancy
>
> On 3/28/18 3:19 PM, Thomas Cram wrote:
>
> Hi Nancy,
> according to the MET documentation, the prepbufr report type and
>
> observation error values cannot be added to the pb2nc output. I
>
> am copying MET support in this message, as they can take a note of
>
> this
>
> and possibly consider it for inclusion in a future release.
>
> In the meantime, you can choose ASCII output from the ds337.0
>
> subsetting
>
> form, and this will include the PB report type in the
>
> output. It currently does not include the observation error values,
>
> however, but I will see if I can modify the code to include
>
> this sometime in the near future.
>
> - Tom
>
> On Wed, Mar 28, 2018 at 2:45 PM, nancy collins <nancy at ucar.edu
>
> <mailto:
>
> nancy at ucar.edu>> wrote:
>
> tom,
>
> thanks for checking into this. that would be great news
> if these items were possible to add to the output files.
>
> n.
>
>
> On 3/28/18 9:13 AM, Thomas Cram wrote:
>
> Hi Nancy,
>
> I use the pb2nc tool in the MET software to produce the NetCDF
>
> files (see https://dtcenter.org/met/users/index.php
>
> <https://dtcenter.org/met/users/index.php>
<https://dtcenter.org/met/users/index.php>), and the output is
>
> defined by a configuration file passed to pb2nc. I'll take a
>
> look at the documentation and see if it's possible to include
>
> the
>
> error values and PB report types in the output.
>
> - Tom
>
> On Tue, Mar 27, 2018 at 3:57 PM, Nancy Collins <nancy at ucar.edu
>
> <mailto:nancy at ucar.edu> <nancy at ucar.edu>> wrote:
>
> hello thomas,
>
> i'm looking at RDA dataset number 337.0 (NCEP prepbufr
>
> observation files).
>
> this data is now also available in netcdf format, which is
>
> great.
>
> we downloaded an example file and most things we need are
>
> there.
>
> however, the netcdf file seems to be missing the
>
> observation
>
> error
>
> values (present in the prepbufr files) and also the
>
> prepbufr
>
> report
>
> types (again, present in the prepbufr files). while the
>
> netcdf
>
> files
>
> do have 'grib codes' and 'message types', they don't map
>
> uniquely
>
> to prepbufr report types, and we see no way to get the obs
>
> errors.
>
> (note that this is not the quality control - it is the
>
> error in
>
> the same
>
> units as the observation value. like a temperature obs
>
> error
>
> might
>
> be 1 or 2 degrees K.)
>
> was this an intentional choice? can you tell us if it's
>
> possible to
>
> get netcdf format files with these two additional fields?
>
> are
>
> there
>
> options on the conversion tool that could be set?
>
> thanks for any information on how these files are created
>
> and
>
> who made the choices for what data is converted into the
> output netcdf files.
>
> nancy
>
>
>
>
> --
> Thomas Cram
> NCAR / CISL / DSS
> tcram at ucar.edu <mailto:tcram at ucar.edu> <tcram at ucar.edu>
> rda.ucar.edu <https://rda.ucar.edu> <https://rda.ucar.edu>
>
>
>
> --
> Thomas Cram
> NCAR / CISL / DSStcram at ucar.edu <mailto:tcram at ucar.edu>
<tcram at ucar.edu>rda.ucar.edu <https://rda.ucar.edu>
<https://rda.ucar.edu>
>
> --
> Thomas Cram
> NCAR / CISL / DSStcram at ucar.edurda.ucar.edu
>
>
>
--
Thomas Cram
NCAR / CISL / DSS
tcram at ucar.edu
rda.ucar.edu
------------------------------------------------
More information about the Met_help
mailing list