[Met_help] [rt.rap.ucar.edu #96825] History for reader for NRL "innovation files"
John Halley Gotway via RT
met_help at ucar.edu
Thu Oct 1 14:21:23 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi John,
We had discussed previously the prospect of creating a read tool for the NRL
innovation files (similar to pb2nc). The innovation files are basically our
version of prepBUFR.
The innovation files are ascii files with a 75 line header. The column
description is below. If you all could provide a reader to get these files
into MET that would be great! I'm happy to provide a sample data set and
any additional information that you may need.
Thanks,
Liz
1. n: Observation number for indexing obs.
2. ob: Observation value.
3. bk_ob: Background (short-term forecast) value, interpolated in space and
time to the observation location/time.
4. t_bk_ob: Supplemental information required for assimilating the
observation.
5. xiv_ob: Innovation, or difference between the observation and the
background value (e.g., ob - bk_ob)
6. err_ob: Observation error standard deviation
7. etc_ob: Supplemental information required for assimilating the
observation.
8. lat_ob: latitude of the observation.
9. lon_ob: longitude of the observation.
10. p_ob: pressure level of observation; except for radiances.
11. jvarty: Jvarty_ob in the code, this is the variable type code.
12. insty: Insty_ob in the code; instrument type code.
13. nvp: Number of observations in the "profile"
14. ichk: Quality indicator.
15. idt: Time difference in seconds from the center of the time window. For
a 6-hr time window, the values range from -10800 to +10800.
16. c_pf_ob: Observation platform identifier (16 characters). For radiances,
this includes sensor name (AMSU-A), channel number, ascending/descending
flag and assimilate flag (A, M, R - for assimilate, monitor or reject). For
radiosonde, it includes the block/station number.
17. c_db_ob: More platform identifiers (10 characters). For radiances, this
is includes the satellite name (NOAA15), and land/sea/ice flag. For
radiosondes, it includes the retransmission status.
18. idp: NAVDAS computes the pressure change over the assimilation window
and stores here.
19. lsi: Land/sea/ice flag
20. rej: Reject flag
21. bkerr: background error standard deviation, in observation space.
22. cob: analysis solution, in observation space prior to being projected
back into model space
23. resid: residual in observation space, or H(analysis - background).
A read statement would look like:
C=textscan(fileID,
'%*7d%8f%*1c%8f%*1c%*8f%*1c%8f%9f%*1c%*9f%*1c%9f%*1c%9f%*1c%11f%*1c%2d%*1c%3
d%*1c%5d%*1c%*4d%*1c%7d%*2c%16c%*2c%10c%*1c%*4d%*1c%*2d%*1c%3d%*1c%7f%*1c%*7
f%*1c%9f%*1c%9f%*[^\n]','headerlines',75,'whitespace','','delimiter','/n');
fortran = '7s 9s 1x 8s 1x 8s 1x 8s 9s 1x 9s 1x 9s 1x 9s 1x 11s ' +\
'1x 2s 1x 3s 1x 5s 1x 4s 1x 7s 2x 16s 2x 10s 1x 4s 1x 2s '
+\
'1x 3s 1x 7s 1x 7s 1x 9s 1x 9s'
Elizabeth Satterfield
Editor, Monthly Weather Review
Chair, AMS Committee on Probability and Statistics
Meteorologist, Code 7531
Marine Meteorology Division
U.S. Naval Research Laboratory
T 831.656.4056 F 831.656.4026
<http://www.nrl.navy.mil/> www.nrl.navy.mil
__________________________________________________________________
FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
Any misuse or unauthorized disclosure may result in both civil and criminal
penalties.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: reader for NRL "innovation files"
From: John Halley Gotway
Time: Thu Sep 24 15:16:59 2020
Hi Liz,
Sounds great. Please send along at least one sample file. If it’s
small
enough you could send as an email attachment. Otherwise you can post
to our
anonymous ftp site:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-
help-desk#ftp
With this info and that data, we can create a GitHub issue to make
sure we
don’t lose track of this task.
It probably belongs in the METplus repo:
https://github.com/dtcenter/METplus/issues
Thanks,
John
On Wed, Sep 23, 2020 at 7:03 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
>
> Wed Sep 23 19:03:22 2020: Request 96825 was acted upon.
>
> Transaction: Ticket created by Elizabeth.Satterfield at nrlmry.navy.mil
>
> Queue: met_help
>
> Subject: reader for NRL "innovation files"
>
> Owner: Nobody
>
> Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
>
> Status: new
>
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96825 >
>
>
>
>
>
> Hi John,
>
>
>
>
>
>
>
> We had discussed previously the prospect of creating a read tool for
the
> NRL
>
> innovation files (similar to pb2nc). The innovation files are
basically
> our
>
> version of prepBUFR.
>
>
>
>
>
>
>
> The innovation files are ascii files with a 75 line header. The
column
>
> description is below. If you all could provide a reader to get
these files
>
> into MET that would be great! I'm happy to provide a sample data
set and
>
> any additional information that you may need.
>
>
>
>
>
>
>
> Thanks,
>
>
>
> Liz
>
>
>
>
>
>
>
>
>
>
>
> 1. n: Observation number for indexing obs.
>
>
>
>
>
>
>
> 2. ob: Observation value.
>
>
>
>
>
>
>
> 3. bk_ob: Background (short-term forecast) value, interpolated in
space and
>
> time to the observation location/time.
>
>
>
>
>
>
>
> 4. t_bk_ob: Supplemental information required for assimilating the
>
> observation.
>
>
>
>
>
>
>
> 5. xiv_ob: Innovation, or difference between the observation and the
>
> background value (e.g., ob - bk_ob)
>
>
>
>
>
>
>
> 6. err_ob: Observation error standard deviation
>
>
>
>
>
>
>
> 7. etc_ob: Supplemental information required for assimilating the
>
> observation.
>
>
>
>
>
>
>
> 8. lat_ob: latitude of the observation.
>
>
>
>
>
>
>
> 9. lon_ob: longitude of the observation.
>
>
>
>
>
>
>
> 10. p_ob: pressure level of observation; except for radiances.
>
>
>
>
>
>
>
> 11. jvarty: Jvarty_ob in the code, this is the variable type code.
>
>
>
>
>
>
>
> 12. insty: Insty_ob in the code; instrument type code.
>
>
>
>
>
>
>
> 13. nvp: Number of observations in the "profile"
>
>
>
>
>
>
>
> 14. ichk: Quality indicator.
>
>
>
>
>
>
>
> 15. idt: Time difference in seconds from the center of the time
window. For
>
> a 6-hr time window, the values range from -10800 to +10800.
>
>
>
>
>
>
>
> 16. c_pf_ob: Observation platform identifier (16 characters). For
> radiances,
>
> this includes sensor name (AMSU-A), channel number,
ascending/descending
>
> flag and assimilate flag (A, M, R - for assimilate, monitor or
reject). For
>
> radiosonde, it includes the block/station number.
>
>
>
>
>
>
>
> 17. c_db_ob: More platform identifiers (10 characters). For
radiances, this
>
> is includes the satellite name (NOAA15), and land/sea/ice flag. For
>
> radiosondes, it includes the retransmission status.
>
>
>
>
>
>
>
> 18. idp: NAVDAS computes the pressure change over the assimilation
window
>
> and stores here.
>
>
>
>
>
>
>
> 19. lsi: Land/sea/ice flag
>
>
>
>
>
>
>
> 20. rej: Reject flag
>
>
>
>
>
>
>
> 21. bkerr: background error standard deviation, in observation
space.
>
>
>
>
>
>
>
> 22. cob: analysis solution, in observation space prior to being
projected
>
> back into model space
>
>
>
>
>
>
>
> 23. resid: residual in observation space, or H(analysis -
background).
>
>
>
>
>
>
>
> A read statement would look like:
>
>
>
>
>
>
>
> C=textscan(fileID,
>
>
>
'%*7d%8f%*1c%8f%*1c%*8f%*1c%8f%9f%*1c%*9f%*1c%9f%*1c%9f%*1c%11f%*1c%2d%*1c%3
>
>
>
d%*1c%5d%*1c%*4d%*1c%7d%*2c%16c%*2c%10c%*1c%*4d%*1c%*2d%*1c%3d%*1c%7f%*1c%*7
>
>
f%*1c%9f%*1c%9f%*[^\n]','headerlines',75,'whitespace','','delimiter','/n');
>
>
>
>
>
>
>
> fortran = '7s 9s 1x 8s 1x 8s 1x 8s 9s 1x 9s 1x 9s 1x 9s 1x 11s '
+\
>
>
>
> '1x 2s 1x 3s 1x 5s 1x 4s 1x 7s 2x 16s 2x 10s 1x 4s
1x 2s '
>
> +\
>
>
>
> '1x 3s 1x 7s 1x 7s 1x 9s 1x 9s'
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Elizabeth Satterfield
>
> Editor, Monthly Weather Review
>
>
>
> Chair, AMS Committee on Probability and Statistics
>
>
>
> Meteorologist, Code 7531
>
> Marine Meteorology Division
>
> U.S. Naval Research Laboratory
>
> T 831.656.4056 F 831.656.4026
>
> <http://www.nrl.navy.mil/> www.nrl.navy.mil
>
>
>
> __________________________________________________________________
>
> FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
>
> Any misuse or unauthorized disclosure may result in both civil and
criminal
>
> penalties.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
------------------------------------------------
Subject: reader for NRL "innovation files"
From: Satterfield, Dr. Elizabeth
Time: Thu Sep 24 16:17:47 2020
Hi John,
I have uploaded the files to the ftp server. We now output h5 files
instead of ascii. I put both formats and the python converter on the
ftp directory.
It would be great to have something like pb2nc, where we could choose
what data types to output to the nc file via a config. Also having
the innovation and residual information will be helpful for DA
diagnostics.
Thanks so much!
Liz
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, September 24, 2020 2:17 PM
To: Satterfield, Dr. Elizabeth <Elizabeth.Satterfield at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #96825] reader for NRL "innovation
files"
Hi Liz,
Sounds great. Please send along at least one sample file. If it’s
small enough you could send as an email attachment. Otherwise you can
post to our anonymous ftp site:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-
help-desk#ftp
With this info and that data, we can create a GitHub issue to make
sure we don’t lose track of this task.
It probably belongs in the METplus repo:
https://github.com/dtcenter/METplus/issues
Thanks,
John
On Wed, Sep 23, 2020 at 7:03 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
>
> Wed Sep 23 19:03:22 2020: Request 96825 was acted upon.
>
> Transaction: Ticket created by Elizabeth.Satterfield at nrlmry.navy.mil
>
> Queue: met_help
>
> Subject: reader for NRL "innovation files"
>
> Owner: Nobody
>
> Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
>
> Status: new
>
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96825
> >
>
>
>
>
>
> Hi John,
>
>
>
>
>
>
>
> We had discussed previously the prospect of creating a read tool for
> the NRL
>
> innovation files (similar to pb2nc). The innovation files are
> basically our
>
> version of prepBUFR.
>
>
>
>
>
>
>
> The innovation files are ascii files with a 75 line header. The
> column
>
> description is below. If you all could provide a reader to get
these
> files
>
> into MET that would be great! I'm happy to provide a sample data
set
> and
>
> any additional information that you may need.
>
>
>
>
>
>
>
> Thanks,
>
>
>
> Liz
>
>
>
>
>
>
>
>
>
>
>
> 1. n: Observation number for indexing obs.
>
>
>
>
>
>
>
> 2. ob: Observation value.
>
>
>
>
>
>
>
> 3. bk_ob: Background (short-term forecast) value, interpolated in
> space and
>
> time to the observation location/time.
>
>
>
>
>
>
>
> 4. t_bk_ob: Supplemental information required for assimilating the
>
> observation.
>
>
>
>
>
>
>
> 5. xiv_ob: Innovation, or difference between the observation and the
>
> background value (e.g., ob - bk_ob)
>
>
>
>
>
>
>
> 6. err_ob: Observation error standard deviation
>
>
>
>
>
>
>
> 7. etc_ob: Supplemental information required for assimilating the
>
> observation.
>
>
>
>
>
>
>
> 8. lat_ob: latitude of the observation.
>
>
>
>
>
>
>
> 9. lon_ob: longitude of the observation.
>
>
>
>
>
>
>
> 10. p_ob: pressure level of observation; except for radiances.
>
>
>
>
>
>
>
> 11. jvarty: Jvarty_ob in the code, this is the variable type code.
>
>
>
>
>
>
>
> 12. insty: Insty_ob in the code; instrument type code.
>
>
>
>
>
>
>
> 13. nvp: Number of observations in the "profile"
>
>
>
>
>
>
>
> 14. ichk: Quality indicator.
>
>
>
>
>
>
>
> 15. idt: Time difference in seconds from the center of the time
> window. For
>
> a 6-hr time window, the values range from -10800 to +10800.
>
>
>
>
>
>
>
> 16. c_pf_ob: Observation platform identifier (16 characters). For
> radiances,
>
> this includes sensor name (AMSU-A), channel number,
> ascending/descending
>
> flag and assimilate flag (A, M, R - for assimilate, monitor or
> reject). For
>
> radiosonde, it includes the block/station number.
>
>
>
>
>
>
>
> 17. c_db_ob: More platform identifiers (10 characters). For
radiances,
> this
>
> is includes the satellite name (NOAA15), and land/sea/ice flag. For
>
> radiosondes, it includes the retransmission status.
>
>
>
>
>
>
>
> 18. idp: NAVDAS computes the pressure change over the assimilation
> window
>
> and stores here.
>
>
>
>
>
>
>
> 19. lsi: Land/sea/ice flag
>
>
>
>
>
>
>
> 20. rej: Reject flag
>
>
>
>
>
>
>
> 21. bkerr: background error standard deviation, in observation
space.
>
>
>
>
>
>
>
> 22. cob: analysis solution, in observation space prior to being
> projected
>
> back into model space
>
>
>
>
>
>
>
> 23. resid: residual in observation space, or H(analysis -
background).
>
>
>
>
>
>
>
> A read statement would look like:
>
>
>
>
>
>
>
> C=textscan(fileID,
>
>
>
'%*7d%8f%*1c%8f%*1c%*8f%*1c%8f%9f%*1c%*9f%*1c%9f%*1c%9f%*1c%11f%*1c%2d
> %*1c%3
>
>
>
d%*1c%5d%*1c%*4d%*1c%7d%*2c%16c%*2c%10c%*1c%*4d%*1c%*2d%*1c%3d%*1c%7f%
> *1c%*7
>
>
f%*1c%9f%*1c%9f%*[^\n]','headerlines',75,'whitespace','','delimiter','
> /n');
>
>
>
>
>
>
>
> fortran = '7s 9s 1x 8s 1x 8s 1x 8s 9s 1x 9s 1x 9s 1x 9s 1x 11s '
+\
>
>
>
> '1x 2s 1x 3s 1x 5s 1x 4s 1x 7s 2x 16s 2x 10s 1x 4s
1x 2s '
>
> +\
>
>
>
> '1x 3s 1x 7s 1x 7s 1x 9s 1x 9s'
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Elizabeth Satterfield
>
> Editor, Monthly Weather Review
>
>
>
> Chair, AMS Committee on Probability and Statistics
>
>
>
> Meteorologist, Code 7531
>
> Marine Meteorology Division
>
> U.S. Naval Research Laboratory
>
> T 831.656.4056 F 831.656.4026
>
> <http://www.nrl.navy.mil/> www.nrl.navy.mil
>
>
>
> __________________________________________________________________
>
> FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
>
> Any misuse or unauthorized disclosure may result in both civil and
> criminal
>
> penalties.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
------------------------------------------------
Subject: reader for NRL "innovation files"
From: John Halley Gotway
Time: Fri Sep 25 09:57:08 2020
Liz,
Here's the GitHub issue I wrote up about this:
https://github.com/dtcenter/METplus/issues/635
I figure it should be a new METplus use case which uses python
embedding to
prepare the point observations and then verifies some model output
using
those observations... likely Point-Stat or Ensemble-Stat. Do you have
a
particular end-to-end use case in mind?
Also, regarding... "having the innovation and residual information
will be
helpful for DA diagnostics"... I'm trying to figure out exactly what
this
means. I clearly see the obs value in your description and we can
input
this into ascii2nc to prepare point observations for vx in Point-Stat
or
Ensemble-Stat. But I'm wondering if you also want to process this data
directly as fcst/obs matched pairs? If so, we could reformat this data
into
matched pair (MPR) format and use python embedding in Stat-Analysis to
derive partial sums and statistics from those pairs. Is that what
you're
thinking? If so, I should update the GitHub issue to include that
step.
Thanks,
John
On Thu, Sep 24, 2020 at 4:17 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96825 >
>
> Hi John,
>
> I have uploaded the files to the ftp server. We now output h5 files
> instead of ascii. I put both formats and the python converter on
the ftp
> directory.
>
> It would be great to have something like pb2nc, where we could
choose what
> data types to output to the nc file via a config. Also having the
> innovation and residual information will be helpful for DA
diagnostics.
>
> Thanks so much!
>
> Liz
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, September 24, 2020 2:17 PM
> To: Satterfield, Dr. Elizabeth
<Elizabeth.Satterfield at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #96825] reader for NRL "innovation
files"
>
> Hi Liz,
>
> Sounds great. Please send along at least one sample file. If it’s
small
> enough you could send as an email attachment. Otherwise you can post
to our
> anonymous ftp site:
>
>
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
help-desk#ftp
>
> With this info and that data, we can create a GitHub issue to make
sure we
> don’t lose track of this task.
>
> It probably belongs in the METplus repo:
>
> https://github.com/dtcenter/METplus/issues
>
> Thanks,
> John
>
>
> On Wed, Sep 23, 2020 at 7:03 PM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> >
> > Wed Sep 23 19:03:22 2020: Request 96825 was acted upon.
> >
> > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> >
> > Queue: met_help
> >
> > Subject: reader for NRL "innovation files"
> >
> > Owner: Nobody
> >
> > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> >
> > Status: new
> >
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96825
> > >
> >
> >
> >
> >
> >
> > Hi John,
> >
> >
> >
> >
> >
> >
> >
> > We had discussed previously the prospect of creating a read tool
for
> > the NRL
> >
> > innovation files (similar to pb2nc). The innovation files are
> > basically our
> >
> > version of prepBUFR.
> >
> >
> >
> >
> >
> >
> >
> > The innovation files are ascii files with a 75 line header. The
> > column
> >
> > description is below. If you all could provide a reader to get
these
> > files
> >
> > into MET that would be great! I'm happy to provide a sample data
set
> > and
> >
> > any additional information that you may need.
> >
> >
> >
> >
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Liz
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 1. n: Observation number for indexing obs.
> >
> >
> >
> >
> >
> >
> >
> > 2. ob: Observation value.
> >
> >
> >
> >
> >
> >
> >
> > 3. bk_ob: Background (short-term forecast) value, interpolated in
> > space and
> >
> > time to the observation location/time.
> >
> >
> >
> >
> >
> >
> >
> > 4. t_bk_ob: Supplemental information required for assimilating the
> >
> > observation.
> >
> >
> >
> >
> >
> >
> >
> > 5. xiv_ob: Innovation, or difference between the observation and
the
> >
> > background value (e.g., ob - bk_ob)
> >
> >
> >
> >
> >
> >
> >
> > 6. err_ob: Observation error standard deviation
> >
> >
> >
> >
> >
> >
> >
> > 7. etc_ob: Supplemental information required for assimilating the
> >
> > observation.
> >
> >
> >
> >
> >
> >
> >
> > 8. lat_ob: latitude of the observation.
> >
> >
> >
> >
> >
> >
> >
> > 9. lon_ob: longitude of the observation.
> >
> >
> >
> >
> >
> >
> >
> > 10. p_ob: pressure level of observation; except for radiances.
> >
> >
> >
> >
> >
> >
> >
> > 11. jvarty: Jvarty_ob in the code, this is the variable type code.
> >
> >
> >
> >
> >
> >
> >
> > 12. insty: Insty_ob in the code; instrument type code.
> >
> >
> >
> >
> >
> >
> >
> > 13. nvp: Number of observations in the "profile"
> >
> >
> >
> >
> >
> >
> >
> > 14. ichk: Quality indicator.
> >
> >
> >
> >
> >
> >
> >
> > 15. idt: Time difference in seconds from the center of the time
> > window. For
> >
> > a 6-hr time window, the values range from -10800 to +10800.
> >
> >
> >
> >
> >
> >
> >
> > 16. c_pf_ob: Observation platform identifier (16 characters). For
> > radiances,
> >
> > this includes sensor name (AMSU-A), channel number,
> > ascending/descending
> >
> > flag and assimilate flag (A, M, R - for assimilate, monitor or
> > reject). For
> >
> > radiosonde, it includes the block/station number.
> >
> >
> >
> >
> >
> >
> >
> > 17. c_db_ob: More platform identifiers (10 characters). For
radiances,
> > this
> >
> > is includes the satellite name (NOAA15), and land/sea/ice flag.
For
> >
> > radiosondes, it includes the retransmission status.
> >
> >
> >
> >
> >
> >
> >
> > 18. idp: NAVDAS computes the pressure change over the assimilation
> > window
> >
> > and stores here.
> >
> >
> >
> >
> >
> >
> >
> > 19. lsi: Land/sea/ice flag
> >
> >
> >
> >
> >
> >
> >
> > 20. rej: Reject flag
> >
> >
> >
> >
> >
> >
> >
> > 21. bkerr: background error standard deviation, in observation
space.
> >
> >
> >
> >
> >
> >
> >
> > 22. cob: analysis solution, in observation space prior to being
> > projected
> >
> > back into model space
> >
> >
> >
> >
> >
> >
> >
> > 23. resid: residual in observation space, or H(analysis -
background).
> >
> >
> >
> >
> >
> >
> >
> > A read statement would look like:
> >
> >
> >
> >
> >
> >
> >
> > C=textscan(fileID,
> >
> >
> >
'%*7d%8f%*1c%8f%*1c%*8f%*1c%8f%9f%*1c%*9f%*1c%9f%*1c%9f%*1c%11f%*1c%2d
> > %*1c%3
> >
> >
> >
d%*1c%5d%*1c%*4d%*1c%7d%*2c%16c%*2c%10c%*1c%*4d%*1c%*2d%*1c%3d%*1c%7f%
> > *1c%*7
> >
> >
f%*1c%9f%*1c%9f%*[^\n]','headerlines',75,'whitespace','','delimiter','
> > /n');
> >
> >
> >
> >
> >
> >
> >
> > fortran = '7s 9s 1x 8s 1x 8s 1x 8s 9s 1x 9s 1x 9s 1x 9s 1x 11s '
+\
> >
> >
> >
> > '1x 2s 1x 3s 1x 5s 1x 4s 1x 7s 2x 16s 2x 10s 1x
4s 1x
> 2s '
> >
> > +\
> >
> >
> >
> > '1x 3s 1x 7s 1x 7s 1x 9s 1x 9s'
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Elizabeth Satterfield
> >
> > Editor, Monthly Weather Review
> >
> >
> >
> > Chair, AMS Committee on Probability and Statistics
> >
> >
> >
> > Meteorologist, Code 7531
> >
> > Marine Meteorology Division
> >
> > U.S. Naval Research Laboratory
> >
> > T 831.656.4056 F 831.656.4026
> >
> > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> >
> >
> >
> > __________________________________________________________________
> >
> > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> >
> > Any misuse or unauthorized disclosure may result in both civil and
> > criminal
> >
> > penalties.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: reader for NRL "innovation files"
From: Satterfield, Dr. Elizabeth
Time: Fri Sep 25 10:09:41 2020
Thanks, John
Yes, the python embedding use case seems reasonable. We can simply
use the observations to verify against forecast fields. I am happy to
provide fields if needed.
Putting the obs values into ascii2nc will get us to the verification
task and is where we likely need to start. Having matched pairs would
also be useful.
There is additional information in the file that would be helpful to
retain (you could treat it as an additional observation type) , in
particular:
1. xiv: the innovation value
2. resid: residual in observation space
3. the prescribed errors oberr and bkerr
4. bk_ob the background in observation space
5. the QC flags
If that information is available along with the ob value, this enables
us to implement some DA diagnostics fairly easily.
Liz
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, September 25, 2020 8:57 AM
To: Satterfield, Dr. Elizabeth <Elizabeth.Satterfield at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #96825] reader for NRL "innovation
files"
Liz,
Here's the GitHub issue I wrote up about this:
https://github.com/dtcenter/METplus/issues/635
I figure it should be a new METplus use case which uses python
embedding to prepare the point observations and then verifies some
model output using those observations... likely Point-Stat or
Ensemble-Stat. Do you have a particular end-to-end use case in mind?
Also, regarding... "having the innovation and residual information
will be helpful for DA diagnostics"... I'm trying to figure out
exactly what this means. I clearly see the obs value in your
description and we can input this into ascii2nc to prepare point
observations for vx in Point-Stat or Ensemble-Stat. But I'm wondering
if you also want to process this data directly as fcst/obs matched
pairs? If so, we could reformat this data into matched pair (MPR)
format and use python embedding in Stat-Analysis to derive partial
sums and statistics from those pairs. Is that what you're thinking? If
so, I should update the GitHub issue to include that step.
Thanks,
John
On Thu, Sep 24, 2020 at 4:17 PM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96825 >
>
> Hi John,
>
> I have uploaded the files to the ftp server. We now output h5 files
> instead of ascii. I put both formats and the python converter on
the
> ftp directory.
>
> It would be great to have something like pb2nc, where we could
choose
> what data types to output to the nc file via a config. Also having
> the innovation and residual information will be helpful for DA
diagnostics.
>
> Thanks so much!
>
> Liz
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, September 24, 2020 2:17 PM
> To: Satterfield, Dr. Elizabeth
<Elizabeth.Satterfield at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #96825] reader for NRL "innovation
files"
>
> Hi Liz,
>
> Sounds great. Please send along at least one sample file. If it’s
> small enough you could send as an email attachment. Otherwise you
can
> post to our anonymous ftp site:
>
>
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
hel
> p-desk#ftp
>
> With this info and that data, we can create a GitHub issue to make
> sure we don’t lose track of this task.
>
> It probably belongs in the METplus repo:
>
> https://github.com/dtcenter/METplus/issues
>
> Thanks,
> John
>
>
> On Wed, Sep 23, 2020 at 7:03 PM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> >
> > Wed Sep 23 19:03:22 2020: Request 96825 was acted upon.
> >
> > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> >
> > Queue: met_help
> >
> > Subject: reader for NRL "innovation files"
> >
> > Owner: Nobody
> >
> > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> >
> > Status: new
> >
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96825
> > >
> >
> >
> >
> >
> >
> > Hi John,
> >
> >
> >
> >
> >
> >
> >
> > We had discussed previously the prospect of creating a read tool
for
> > the NRL
> >
> > innovation files (similar to pb2nc). The innovation files are
> > basically our
> >
> > version of prepBUFR.
> >
> >
> >
> >
> >
> >
> >
> > The innovation files are ascii files with a 75 line header. The
> > column
> >
> > description is below. If you all could provide a reader to get
> > these files
> >
> > into MET that would be great! I'm happy to provide a sample data
> > set and
> >
> > any additional information that you may need.
> >
> >
> >
> >
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Liz
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 1. n: Observation number for indexing obs.
> >
> >
> >
> >
> >
> >
> >
> > 2. ob: Observation value.
> >
> >
> >
> >
> >
> >
> >
> > 3. bk_ob: Background (short-term forecast) value, interpolated in
> > space and
> >
> > time to the observation location/time.
> >
> >
> >
> >
> >
> >
> >
> > 4. t_bk_ob: Supplemental information required for assimilating the
> >
> > observation.
> >
> >
> >
> >
> >
> >
> >
> > 5. xiv_ob: Innovation, or difference between the observation and
the
> >
> > background value (e.g., ob - bk_ob)
> >
> >
> >
> >
> >
> >
> >
> > 6. err_ob: Observation error standard deviation
> >
> >
> >
> >
> >
> >
> >
> > 7. etc_ob: Supplemental information required for assimilating the
> >
> > observation.
> >
> >
> >
> >
> >
> >
> >
> > 8. lat_ob: latitude of the observation.
> >
> >
> >
> >
> >
> >
> >
> > 9. lon_ob: longitude of the observation.
> >
> >
> >
> >
> >
> >
> >
> > 10. p_ob: pressure level of observation; except for radiances.
> >
> >
> >
> >
> >
> >
> >
> > 11. jvarty: Jvarty_ob in the code, this is the variable type code.
> >
> >
> >
> >
> >
> >
> >
> > 12. insty: Insty_ob in the code; instrument type code.
> >
> >
> >
> >
> >
> >
> >
> > 13. nvp: Number of observations in the "profile"
> >
> >
> >
> >
> >
> >
> >
> > 14. ichk: Quality indicator.
> >
> >
> >
> >
> >
> >
> >
> > 15. idt: Time difference in seconds from the center of the time
> > window. For
> >
> > a 6-hr time window, the values range from -10800 to +10800.
> >
> >
> >
> >
> >
> >
> >
> > 16. c_pf_ob: Observation platform identifier (16 characters). For
> > radiances,
> >
> > this includes sensor name (AMSU-A), channel number,
> > ascending/descending
> >
> > flag and assimilate flag (A, M, R - for assimilate, monitor or
> > reject). For
> >
> > radiosonde, it includes the block/station number.
> >
> >
> >
> >
> >
> >
> >
> > 17. c_db_ob: More platform identifiers (10 characters). For
> > radiances, this
> >
> > is includes the satellite name (NOAA15), and land/sea/ice flag.
For
> >
> > radiosondes, it includes the retransmission status.
> >
> >
> >
> >
> >
> >
> >
> > 18. idp: NAVDAS computes the pressure change over the assimilation
> > window
> >
> > and stores here.
> >
> >
> >
> >
> >
> >
> >
> > 19. lsi: Land/sea/ice flag
> >
> >
> >
> >
> >
> >
> >
> > 20. rej: Reject flag
> >
> >
> >
> >
> >
> >
> >
> > 21. bkerr: background error standard deviation, in observation
space.
> >
> >
> >
> >
> >
> >
> >
> > 22. cob: analysis solution, in observation space prior to being
> > projected
> >
> > back into model space
> >
> >
> >
> >
> >
> >
> >
> > 23. resid: residual in observation space, or H(analysis -
background).
> >
> >
> >
> >
> >
> >
> >
> > A read statement would look like:
> >
> >
> >
> >
> >
> >
> >
> > C=textscan(fileID,
> >
> >
> >
'%*7d%8f%*1c%8f%*1c%*8f%*1c%8f%9f%*1c%*9f%*1c%9f%*1c%9f%*1c%11f%*1c%
> > 2d
> > %*1c%3
> >
> >
> >
d%*1c%5d%*1c%*4d%*1c%7d%*2c%16c%*2c%10c%*1c%*4d%*1c%*2d%*1c%3d%*1c%7
> > f%
> > *1c%*7
> >
> >
f%*1c%9f%*1c%9f%*[^\n]','headerlines',75,'whitespace','','delimiter','
> > /n');
> >
> >
> >
> >
> >
> >
> >
> > fortran = '7s 9s 1x 8s 1x 8s 1x 8s 9s 1x 9s 1x 9s 1x 9s 1x 11s '
+\
> >
> >
> >
> > '1x 2s 1x 3s 1x 5s 1x 4s 1x 7s 2x 16s 2x 10s 1x
4s
> > 1x
> 2s '
> >
> > +\
> >
> >
> >
> > '1x 3s 1x 7s 1x 7s 1x 9s 1x 9s'
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Elizabeth Satterfield
> >
> > Editor, Monthly Weather Review
> >
> >
> >
> > Chair, AMS Committee on Probability and Statistics
> >
> >
> >
> > Meteorologist, Code 7531
> >
> > Marine Meteorology Division
> >
> > U.S. Naval Research Laboratory
> >
> > T 831.656.4056 F 831.656.4026
> >
> > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> >
> >
> >
> > __________________________________________________________________
> >
> > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> >
> > Any misuse or unauthorized disclosure may result in both civil and
> > criminal
> >
> > penalties.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: reader for NRL "innovation files"
From: John Halley Gotway
Time: Mon Sep 28 08:02:39 2020
Liz,
Thanks, I've added these details to the GitHub issue.
https://github.com/dtcenter/METplus/issues/635
One other question for you. If you have a GitHub account, I could add
you
to our DTCenter organization on GitHub. And I could add you to our
"METplus
Team" which would enable me to assign you on this development task.
We've
generally been assigning most issues to at least one engineer and one
scientist. And we keep track of our progress using comments on the
issue
and in our Git commit history. The engineer generally does the
development
work while the scientist does some testing... or at least advises on
the
development and testing. And anyone assigned to the issue receives
emails
whenever it is updated.
If you'd like to interact in this way, please send me your GitHub
username
to be added. And if not, that's fine too, and we'll just send any
questions
we have via email.
Thanks,
John
On Fri, Sep 25, 2020 at 10:09 AM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96825 >
>
> Thanks, John
>
> Yes, the python embedding use case seems reasonable. We can simply
use
> the observations to verify against forecast fields. I am happy to
provide
> fields if needed.
>
> Putting the obs values into ascii2nc will get us to the verification
task
> and is where we likely need to start. Having matched pairs would
also be
> useful.
>
> There is additional information in the file that would be helpful to
> retain (you could treat it as an additional observation type) , in
> particular:
>
> 1. xiv: the innovation value
> 2. resid: residual in observation space
> 3. the prescribed errors oberr and bkerr
> 4. bk_ob the background in observation space
> 5. the QC flags
>
> If that information is available along with the ob value, this
enables us
> to implement some DA diagnostics fairly easily.
>
> Liz
>
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, September 25, 2020 8:57 AM
> To: Satterfield, Dr. Elizabeth
<Elizabeth.Satterfield at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #96825] reader for NRL "innovation
files"
>
> Liz,
>
> Here's the GitHub issue I wrote up about this:
> https://github.com/dtcenter/METplus/issues/635
>
> I figure it should be a new METplus use case which uses python
embedding
> to prepare the point observations and then verifies some model
output using
> those observations... likely Point-Stat or Ensemble-Stat. Do you
have a
> particular end-to-end use case in mind?
>
> Also, regarding... "having the innovation and residual information
will be
> helpful for DA diagnostics"... I'm trying to figure out exactly what
this
> means. I clearly see the obs value in your description and we can
input
> this into ascii2nc to prepare point observations for vx in Point-
Stat or
> Ensemble-Stat. But I'm wondering if you also want to process this
data
> directly as fcst/obs matched pairs? If so, we could reformat this
data into
> matched pair (MPR) format and use python embedding in Stat-Analysis
to
> derive partial sums and statistics from those pairs. Is that what
you're
> thinking? If so, I should update the GitHub issue to include that
step.
>
> Thanks,
> John
>
> On Thu, Sep 24, 2020 at 4:17 PM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96825 >
> >
> > Hi John,
> >
> > I have uploaded the files to the ftp server. We now output h5
files
> > instead of ascii. I put both formats and the python converter on
the
> > ftp directory.
> >
> > It would be great to have something like pb2nc, where we could
choose
> > what data types to output to the nc file via a config. Also
having
> > the innovation and residual information will be helpful for DA
> diagnostics.
> >
> > Thanks so much!
> >
> > Liz
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, September 24, 2020 2:17 PM
> > To: Satterfield, Dr. Elizabeth
<Elizabeth.Satterfield at nrlmry.navy.mil>
> > Subject: Re: [rt.rap.ucar.edu #96825] reader for NRL "innovation
files"
> >
> > Hi Liz,
> >
> > Sounds great. Please send along at least one sample file. If it’s
> > small enough you could send as an email attachment. Otherwise you
can
> > post to our anonymous ftp site:
> >
> >
> > https://dtcenter.org/community-code/model-evaluation-tools-
met/met-hel
> > p-desk#ftp
> >
> > With this info and that data, we can create a GitHub issue to make
> > sure we don’t lose track of this task.
> >
> > It probably belongs in the METplus repo:
> >
> > https://github.com/dtcenter/METplus/issues
> >
> > Thanks,
> > John
> >
> >
> > On Wed, Sep 23, 2020 at 7:03 PM Satterfield, Dr. Elizabeth via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > >
> > > Wed Sep 23 19:03:22 2020: Request 96825 was acted upon.
> > >
> > > Transaction: Ticket created by
Elizabeth.Satterfield at nrlmry.navy.mil
> > >
> > > Queue: met_help
> > >
> > > Subject: reader for NRL "innovation files"
> > >
> > > Owner: Nobody
> > >
> > > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > >
> > > Status: new
> > >
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96825
> > > >
> > >
> > >
> > >
> > >
> > >
> > > Hi John,
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > We had discussed previously the prospect of creating a read tool
for
> > > the NRL
> > >
> > > innovation files (similar to pb2nc). The innovation files are
> > > basically our
> > >
> > > version of prepBUFR.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > The innovation files are ascii files with a 75 line header. The
> > > column
> > >
> > > description is below. If you all could provide a reader to get
> > > these files
> > >
> > > into MET that would be great! I'm happy to provide a sample
data
> > > set and
> > >
> > > any additional information that you may need.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Thanks,
> > >
> > >
> > >
> > > Liz
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 1. n: Observation number for indexing obs.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 2. ob: Observation value.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 3. bk_ob: Background (short-term forecast) value, interpolated
in
> > > space and
> > >
> > > time to the observation location/time.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 4. t_bk_ob: Supplemental information required for assimilating
the
> > >
> > > observation.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 5. xiv_ob: Innovation, or difference between the observation and
the
> > >
> > > background value (e.g., ob - bk_ob)
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 6. err_ob: Observation error standard deviation
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 7. etc_ob: Supplemental information required for assimilating
the
> > >
> > > observation.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 8. lat_ob: latitude of the observation.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 9. lon_ob: longitude of the observation.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 10. p_ob: pressure level of observation; except for radiances.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 11. jvarty: Jvarty_ob in the code, this is the variable type
code.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 12. insty: Insty_ob in the code; instrument type code.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 13. nvp: Number of observations in the "profile"
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 14. ichk: Quality indicator.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 15. idt: Time difference in seconds from the center of the time
> > > window. For
> > >
> > > a 6-hr time window, the values range from -10800 to +10800.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 16. c_pf_ob: Observation platform identifier (16 characters).
For
> > > radiances,
> > >
> > > this includes sensor name (AMSU-A), channel number,
> > > ascending/descending
> > >
> > > flag and assimilate flag (A, M, R - for assimilate, monitor or
> > > reject). For
> > >
> > > radiosonde, it includes the block/station number.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 17. c_db_ob: More platform identifiers (10 characters). For
> > > radiances, this
> > >
> > > is includes the satellite name (NOAA15), and land/sea/ice flag.
For
> > >
> > > radiosondes, it includes the retransmission status.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 18. idp: NAVDAS computes the pressure change over the
assimilation
> > > window
> > >
> > > and stores here.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 19. lsi: Land/sea/ice flag
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 20. rej: Reject flag
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 21. bkerr: background error standard deviation, in observation
space.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 22. cob: analysis solution, in observation space prior to being
> > > projected
> > >
> > > back into model space
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 23. resid: residual in observation space, or H(analysis -
background).
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > A read statement would look like:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > C=textscan(fileID,
> > >
> > >
> > >
'%*7d%8f%*1c%8f%*1c%*8f%*1c%8f%9f%*1c%*9f%*1c%9f%*1c%9f%*1c%11f%*1c%
> > > 2d
> > > %*1c%3
> > >
> > >
> > >
d%*1c%5d%*1c%*4d%*1c%7d%*2c%16c%*2c%10c%*1c%*4d%*1c%*2d%*1c%3d%*1c%7
> > > f%
> > > *1c%*7
> > >
> > >
f%*1c%9f%*1c%9f%*[^\n]','headerlines',75,'whitespace','','delimiter','
> > > /n');
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > fortran = '7s 9s 1x 8s 1x 8s 1x 8s 9s 1x 9s 1x 9s 1x 9s 1x 11s
' +\
> > >
> > >
> > >
> > > '1x 2s 1x 3s 1x 5s 1x 4s 1x 7s 2x 16s 2x 10s 1x
4s
> > > 1x
> > 2s '
> > >
> > > +\
> > >
> > >
> > >
> > > '1x 3s 1x 7s 1x 7s 1x 9s 1x 9s'
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Elizabeth Satterfield
> > >
> > > Editor, Monthly Weather Review
> > >
> > >
> > >
> > > Chair, AMS Committee on Probability and Statistics
> > >
> > >
> > >
> > > Meteorologist, Code 7531
> > >
> > > Marine Meteorology Division
> > >
> > > U.S. Naval Research Laboratory
> > >
> > > T 831.656.4056 F 831.656.4026
> > >
> > > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> > >
> > >
> > >
> > >
__________________________________________________________________
> > >
> > > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > >
> > > Any misuse or unauthorized disclosure may result in both civil
and
> > > criminal
> > >
> > > penalties.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: reader for NRL "innovation files"
From: Satterfield, Dr. Elizabeth
Time: Mon Sep 28 10:29:18 2020
Hi John,
We use bitbucket locally so I don't have a github account, but I'll
get one -- this seems like a good way to interact.
Liz
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Monday, September 28, 2020 7:03 AM
To: Satterfield, Dr. Elizabeth <Elizabeth.Satterfield at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #96825] reader for NRL "innovation
files"
Liz,
Thanks, I've added these details to the GitHub issue.
https://github.com/dtcenter/METplus/issues/635
One other question for you. If you have a GitHub account, I could add
you to our DTCenter organization on GitHub. And I could add you to our
"METplus Team" which would enable me to assign you on this development
task. We've generally been assigning most issues to at least one
engineer and one scientist. And we keep track of our progress using
comments on the issue and in our Git commit history. The engineer
generally does the development work while the scientist does some
testing... or at least advises on the development and testing. And
anyone assigned to the issue receives emails whenever it is updated.
If you'd like to interact in this way, please send me your GitHub
username to be added. And if not, that's fine too, and we'll just send
any questions we have via email.
Thanks,
John
On Fri, Sep 25, 2020 at 10:09 AM Satterfield, Dr. Elizabeth via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96825 >
>
> Thanks, John
>
> Yes, the python embedding use case seems reasonable. We can simply
> use the observations to verify against forecast fields. I am happy
to
> provide fields if needed.
>
> Putting the obs values into ascii2nc will get us to the verification
> task and is where we likely need to start. Having matched pairs
would
> also be useful.
>
> There is additional information in the file that would be helpful to
> retain (you could treat it as an additional observation type) , in
> particular:
>
> 1. xiv: the innovation value
> 2. resid: residual in observation space 3. the prescribed errors
oberr
> and bkerr 4. bk_ob the background in observation space 5. the QC
flags
>
> If that information is available along with the ob value, this
enables
> us to implement some DA diagnostics fairly easily.
>
> Liz
>
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, September 25, 2020 8:57 AM
> To: Satterfield, Dr. Elizabeth
<Elizabeth.Satterfield at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #96825] reader for NRL "innovation
files"
>
> Liz,
>
> Here's the GitHub issue I wrote up about this:
> https://github.com/dtcenter/METplus/issues/635
>
> I figure it should be a new METplus use case which uses python
> embedding to prepare the point observations and then verifies some
> model output using those observations... likely Point-Stat or
> Ensemble-Stat. Do you have a particular end-to-end use case in mind?
>
> Also, regarding... "having the innovation and residual information
> will be helpful for DA diagnostics"... I'm trying to figure out
> exactly what this means. I clearly see the obs value in your
> description and we can input this into ascii2nc to prepare point
> observations for vx in Point-Stat or Ensemble-Stat. But I'm
wondering
> if you also want to process this data directly as fcst/obs matched
> pairs? If so, we could reformat this data into matched pair (MPR)
> format and use python embedding in Stat-Analysis to derive partial
> sums and statistics from those pairs. Is that what you're thinking?
If so, I should update the GitHub issue to include that step.
>
> Thanks,
> John
>
> On Thu, Sep 24, 2020 at 4:17 PM Satterfield, Dr. Elizabeth via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96825 >
> >
> > Hi John,
> >
> > I have uploaded the files to the ftp server. We now output h5
files
> > instead of ascii. I put both formats and the python converter on
> > the ftp directory.
> >
> > It would be great to have something like pb2nc, where we could
> > choose what data types to output to the nc file via a config.
Also
> > having the innovation and residual information will be helpful for
> > DA
> diagnostics.
> >
> > Thanks so much!
> >
> > Liz
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, September 24, 2020 2:17 PM
> > To: Satterfield, Dr. Elizabeth
> > <Elizabeth.Satterfield at nrlmry.navy.mil>
> > Subject: Re: [rt.rap.ucar.edu #96825] reader for NRL "innovation
files"
> >
> > Hi Liz,
> >
> > Sounds great. Please send along at least one sample file. If it’s
> > small enough you could send as an email attachment. Otherwise you
> > can post to our anonymous ftp site:
> >
> >
> > https://dtcenter.org/community-code/model-evaluation-tools-
met/met-h
> > el
> > p-desk#ftp
> >
> > With this info and that data, we can create a GitHub issue to make
> > sure we don’t lose track of this task.
> >
> > It probably belongs in the METplus repo:
> >
> > https://github.com/dtcenter/METplus/issues
> >
> > Thanks,
> > John
> >
> >
> > On Wed, Sep 23, 2020 at 7:03 PM Satterfield, Dr. Elizabeth via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > >
> > > Wed Sep 23 19:03:22 2020: Request 96825 was acted upon.
> > >
> > > Transaction: Ticket created by
> > > Elizabeth.Satterfield at nrlmry.navy.mil
> > >
> > > Queue: met_help
> > >
> > > Subject: reader for NRL "innovation files"
> > >
> > > Owner: Nobody
> > >
> > > Requestors: Elizabeth.Satterfield at nrlmry.navy.mil
> > >
> > > Status: new
> > >
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96825
> > > >
> > >
> > >
> > >
> > >
> > >
> > > Hi John,
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > We had discussed previously the prospect of creating a read tool
> > > for the NRL
> > >
> > > innovation files (similar to pb2nc). The innovation files are
> > > basically our
> > >
> > > version of prepBUFR.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > The innovation files are ascii files with a 75 line header. The
> > > column
> > >
> > > description is below. If you all could provide a reader to get
> > > these files
> > >
> > > into MET that would be great! I'm happy to provide a sample
data
> > > set and
> > >
> > > any additional information that you may need.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Thanks,
> > >
> > >
> > >
> > > Liz
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 1. n: Observation number for indexing obs.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 2. ob: Observation value.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 3. bk_ob: Background (short-term forecast) value, interpolated
in
> > > space and
> > >
> > > time to the observation location/time.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 4. t_bk_ob: Supplemental information required for assimilating
the
> > >
> > > observation.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 5. xiv_ob: Innovation, or difference between the observation and
> > > the
> > >
> > > background value (e.g., ob - bk_ob)
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 6. err_ob: Observation error standard deviation
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 7. etc_ob: Supplemental information required for assimilating
the
> > >
> > > observation.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 8. lat_ob: latitude of the observation.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 9. lon_ob: longitude of the observation.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 10. p_ob: pressure level of observation; except for radiances.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 11. jvarty: Jvarty_ob in the code, this is the variable type
code.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 12. insty: Insty_ob in the code; instrument type code.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 13. nvp: Number of observations in the "profile"
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 14. ichk: Quality indicator.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 15. idt: Time difference in seconds from the center of the time
> > > window. For
> > >
> > > a 6-hr time window, the values range from -10800 to +10800.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 16. c_pf_ob: Observation platform identifier (16 characters).
For
> > > radiances,
> > >
> > > this includes sensor name (AMSU-A), channel number,
> > > ascending/descending
> > >
> > > flag and assimilate flag (A, M, R - for assimilate, monitor or
> > > reject). For
> > >
> > > radiosonde, it includes the block/station number.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 17. c_db_ob: More platform identifiers (10 characters). For
> > > radiances, this
> > >
> > > is includes the satellite name (NOAA15), and land/sea/ice flag.
> > > For
> > >
> > > radiosondes, it includes the retransmission status.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 18. idp: NAVDAS computes the pressure change over the
assimilation
> > > window
> > >
> > > and stores here.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 19. lsi: Land/sea/ice flag
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 20. rej: Reject flag
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 21. bkerr: background error standard deviation, in observation
space.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 22. cob: analysis solution, in observation space prior to being
> > > projected
> > >
> > > back into model space
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 23. resid: residual in observation space, or H(analysis -
background).
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > A read statement would look like:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > C=textscan(fileID,
> > >
> > >
> > >
'%*7d%8f%*1c%8f%*1c%*8f%*1c%8f%9f%*1c%*9f%*1c%9f%*1c%9f%*1c%11f%*1
> > > c%
> > > 2d
> > > %*1c%3
> > >
> > >
> > >
d%*1c%5d%*1c%*4d%*1c%7d%*2c%16c%*2c%10c%*1c%*4d%*1c%*2d%*1c%3d%*1c
> > > %7
> > > f%
> > > *1c%*7
> > >
> > >
f%*1c%9f%*1c%9f%*[^\n]','headerlines',75,'whitespace','','delimiter','
> > > /n');
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > fortran = '7s 9s 1x 8s 1x 8s 1x 8s 9s 1x 9s 1x 9s 1x 9s 1x 11s
' +\
> > >
> > >
> > >
> > > '1x 2s 1x 3s 1x 5s 1x 4s 1x 7s 2x 16s 2x 10s 1x
> > > 4s 1x
> > 2s '
> > >
> > > +\
> > >
> > >
> > >
> > > '1x 3s 1x 7s 1x 7s 1x 9s 1x 9s'
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Elizabeth Satterfield
> > >
> > > Editor, Monthly Weather Review
> > >
> > >
> > >
> > > Chair, AMS Committee on Probability and Statistics
> > >
> > >
> > >
> > > Meteorologist, Code 7531
> > >
> > > Marine Meteorology Division
> > >
> > > U.S. Naval Research Laboratory
> > >
> > > T 831.656.4056 F 831.656.4026
> > >
> > > <http://www.nrl.navy.mil/> www.nrl.navy.mil
> > >
> > >
> > >
> > >
__________________________________________________________________
> > >
> > > FOR OFFICIAL USE ONLY - PRIVACY SENSITIVE
> > >
> > > Any misuse or unauthorized disclosure may result in both civil
and
> > > criminal
> > >
> > > penalties.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list