[Met_help] [rt.rap.ucar.edu #85291] History for Daily mean ocean forecast fields in netcdf with MODE
John Halley Gotway via RT
met_help at ucar.edu
Tue Jul 9 12:03:29 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
We're now in a position where we actually want to start testing the MODE config but we're getting stuck on the basics of getting the files/forecasts/formats recognised, so we need some help.
First of all these fields are SST. We thought we'd start with something relatively easy!
One thing to note is that most fields are daily means. This is different from many atmospheric NWP outputs... more common in climate.
I thought it would easier to specify the analysis as the truth to begin with, at least these are on the same grid, but we do have satellite data which is on a different grid. They are not even on the same domain, with the observations domain is larger than the model. Please advise as to what to do about that? Should we preprocess outside. There is a regrid = {} argument in the config file but I haven't been able to work out how this works, or what it does.
My experience in using MODE has been with grib and grib headers for atmospheric models. Having read in the netcdf files in R I see that the variable we want is "votemper". As far as I can tell the "level" will have to be set to "deptht", though I am unsure as to the equivalences between grib and netcdf here.
Run using:
${bindir}/mode ${datapath}/model/surfaceprodm_op_am-dm.gridT_20180520._00.-12.nc ${datapath}/model/surfaceprodm_op_am-dm.gridT_20180520._00.108.nc ../config/MODEConfig_test.cfg -outdir ${outpath}/test
Now to the error message (when using the analysis):
eld609:/project/hive/scripts > run_mode.test
DEBUG 1: Default Config File: /project/ukmo/meteval/share/met/config/MODEConfig_default
DEBUG 1: Match Config File: ../config/MODEConfig_test.cfg
DEBUG 1: Merge Config File: ../config/MODEConfig_test.cfg
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using 0. --> i.e. it's not recognising this is a t+108h lead time.
WARNING:
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using 0.
WARNING:
DEBUG 1: Forecast File: /project/hive/input/model/surfaceprodm_op_am-dm.gridT_20180520._00.108.nc --> i.e. it's not recognising this is a t+108h lead time.
DEBUG 1: Observation File: /project/hive/input/model/surfaceprodm_op_am-dm.gridT_20180520._00.-12.nc
ERROR :
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &) -> needed 4 arguments for variable votemper, got 2
ERROR :
HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
#000: H5T.c line 1734 in H5Tclose(): not a datatype
major: Invalid arguments to routine
minor: Inappropriate type
NetCDF: HDF error
file: ncFile.cpp line:33
HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
#000: H5T.c line 1734 in H5Tclose(): not a datatype
major: Invalid arguments to routine
minor: Inappropriate type
NetCDF: HDF error
file: ncFile.cpp line:33
>From the R code I can see that votemper has 4 arguments, but I don't know whether this refers to the same thing in the error message. I also don't know what to specify in the config file as there are only two items to specify: name and level.
float votemper[x,y,deptht,time_counter]
units: degC
standard_name: potential temperature 25h mean
_FillValue: 9.96920996838687e+36
long_name: potential temperature 25h mean
online_operation: ave(X)
interval_operation: 300
interval_write: 86400
coordinates: time_counter deptht nav_lat nav_lon
Any help gratefully received... The files are too large to send so I have uploaded them to Dropbox. They can be accessed from here:
https://www.dropbox.com/l/scl/AAClFKazy1EPxoUsXBLALbBw8Pz-x-P5eiM
There is an analysis, forecast and satellite file.
Regards
Marion
--
Dr Marion Mittermaier Manager: Model Diagnostics and Novel Methods
Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
Tel: +44 1392 884830 Fax: +44 1392 885631
E:mail: marion.mittermaier at metoffice.gov.uk http://www.metoffice.gov.uk/research/people/marion-mittermaier
I am also the co-chair of the WMO WWRP/WGNE Joint Working Group for Forecast Verification Research<https://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html>
You can now follow our science on Twitter: @MetOffice_Sci<http://www.twitter.com/metoffice_sci>
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Daily mean ocean forecast fields in netcdf with MODE
From: marion.mittermaier at metoffice.gov.uk
Time: Fri May 25 04:17:00 2018
Hello,
Good news! I have found the section in the manual dealing with aspects
of netcdf that are just alien to me. I have managed to get mode to run
with the analysis field, using the level "(0,0,*,*)". See attached
postscript.
However, mode is struggling to diagnose the time, setting it to 1970.
I am wondering whether that is because the netcdf is not CF compliant.
I don't know how to check this... I've asked the folks creating the
files to let me know. The other reason may be that it is due to the
fact that the forecasts (and the analysis) are 25h means and the time
specification is a little strange, i.e. the 108h forecast is valid at
12 UTC but is an average between 00:00 and 00:00 (or it could be 23:30
to 23:30!)...
I have also not tried the regrid option to use with the satellite
field. I will try that when I am feeling brave!
Regards
Marion
P.S. the guide is brilliant but it's not always easy to find what you
need....
-----Original Message-----
From: met_help at ucar.edu via RT <met_help at ucar.edu>
Sent: 24 May 2018 15:16
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Subject: [rt.rap.ucar.edu #85291] AutoReply: Daily mean ocean forecast
fields in netcdf with MODE
Greetings,
This message has been automatically generated in response to the
creation of a trouble ticket regarding:
"Daily mean ocean forecast fields in netcdf with MODE", a summary of
which appears below.
There is no need to reply to this message right now. Your ticket has
been assigned an ID of [rt.rap.ucar.edu #85291].
Please include the string:
[rt.rap.ucar.edu #85291]
in the subject line of all future correspondence about this issue. To
do so, you may reply to this message.
For more information, please see:
MET Online Tutorial:
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
MET Users Guide:
https://www.dtcenter.org/met/users/docs/overview.php
MET FAQs:
https://www.dtcenter.org/met/users/support/faqs/index.php
MET-Help Email Archive:
http://mailman.ucar.edu/pipermail/met_help
Thank you,
met_help at ucar.edu
-------------------------------------------------------------------------
Hi,
We're now in a position where we actually want to start testing the
MODE config but we're getting stuck on the basics of getting the
files/forecasts/formats recognised, so we need some help.
First of all these fields are SST. We thought we'd start with
something relatively easy!
One thing to note is that most fields are daily means. This is
different from many atmospheric NWP outputs... more common in climate.
I thought it would easier to specify the analysis as the truth to
begin with, at least these are on the same grid, but we do have
satellite data which is on a different grid. They are not even on the
same domain, with the observations domain is larger than the model.
Please advise as to what to do about that? Should we preprocess
outside. There is a regrid = {} argument in the config file but I
haven't been able to work out how this works, or what it does.
My experience in using MODE has been with grib and grib headers for
atmospheric models. Having read in the netcdf files in R I see that
the variable we want is "votemper". As far as I can tell the "level"
will have to be set to "deptht", though I am unsure as to the
equivalences between grib and netcdf here.
Run using:
${bindir}/mode ${datapath}/model/surfaceprodm_op_am-
dm.gridT_20180520._00.-12.nc ${datapath}/model/surfaceprodm_op_am-
dm.gridT_20180520._00.108.nc ../config/MODEConfig_test.cfg -outdir
${outpath}/test
Now to the error message (when using the analysis):
eld609:/project/hive/scripts > run_mode.test DEBUG 1: Default Config
File: /project/ukmo/meteval/share/met/config/MODEConfig_default
DEBUG 1: Match Config File: ../config/MODEConfig_test.cfg DEBUG 1:
Merge Config File: ../config/MODEConfig_test.cfg
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using
0. --> i.e. it's not recognising this is a t+108h lead time.
WARNING:
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using
0.
WARNING:
DEBUG 1: Forecast File: /project/hive/input/model/surfaceprodm_op_am-
dm.gridT_20180520._00.108.nc --> i.e. it's not recognising this is a
t+108h lead time.
DEBUG 1: Observation File:
/project/hive/input/model/surfaceprodm_op_am-dm.gridT_20180520._00.-
12.nc
ERROR :
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &) ->
needed 4 arguments for variable votemper, got 2 ERROR :
HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
#000: H5T.c line 1734 in H5Tclose(): not a datatype
major: Invalid arguments to routine
minor: Inappropriate type
NetCDF: HDF error
file: ncFile.cpp line:33
HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
#000: H5T.c line 1734 in H5Tclose(): not a datatype
major: Invalid arguments to routine
minor: Inappropriate type
NetCDF: HDF error
file: ncFile.cpp line:33
>From the R code I can see that votemper has 4 arguments, but I don't
know whether this refers to the same thing in the error message. I
also don't know what to specify in the config file as there are only
two items to specify: name and level.
float votemper[x,y,deptht,time_counter]
units: degC
standard_name: potential temperature 25h mean
_FillValue: 9.96920996838687e+36
long_name: potential temperature 25h mean
online_operation: ave(X)
interval_operation: 300
interval_write: 86400
coordinates: time_counter deptht nav_lat nav_lon
Any help gratefully received... The files are too large to send so I
have uploaded them to Dropbox. They can be accessed from here:
https://www.dropbox.com/l/scl/AAClFKazy1EPxoUsXBLALbBw8Pz-x-P5eiM
There is an analysis, forecast and satellite file.
Regards
Marion
--
Dr Marion Mittermaier Manager: Model Diagnostics and Novel
Methods
Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
Tel: +44 1392 884830 Fax: +44 1392 885631
E:mail: marion.mittermaier at metoffice.gov.uk
http://www.metoffice.gov.uk/research/people/marion-mittermaier
I am also the co-chair of the WMO WWRP/WGNE Joint Working Group for
Forecast Verification
Research<https://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html>
You can now follow our science on Twitter:
@MetOffice_Sci<http://www.twitter.com/metoffice_sci>
------------------------------------------------
Subject: Daily mean ocean forecast fields in netcdf with MODE
From: John Halley Gotway
Time: Fri May 25 15:11:23 2018
Hi Marion,
This is John Halley Gotway. Glad you were able to make progress!
I took a quick look at this yesterday but didn't have time to delve in
too
deeply. I'm concerned about 2 things in particular:
- There is no map data included in your plot which indicates that
MET
isn't placing this data on the correct spot on the earth.
- As you've already found, we're not parsing the times correctly.
FYI, this doesn't help now... but one feature we're working on for the
next
release is enabling the user to point MET to a Python script to reads
the
data and define its metadata (projection and timing info, for
example). If
that feature was in place, one way of solving this problem would be
writing
a Python script to load up a data structure for MET to use. That way
we
could use a scripting solution to handle idiosyncrasies in data rather
than
having to change the input data files or the compiled MET code
directly.
Running in the debugger, I see the problem in extracting the grid
definition. MET get the corner and dimensions correct, but the
delta_lat
and delta_lon values are wrong (set to 0):
{name = 0x541964 <_ZL16latlon_proj_type> "LatLon", lat_ll =
-19.888889312744141, lon_ll = -40.066669464111328, delta_lat = 0,
delta_lon
= 0, Nlat = 375, Nlon = 297}
We are reading lat/lon data from the nav_lat and nav_lon variables
but
getting constants values from the 2d arrays.
Until we get the projection info correct, the regridding logic won't
work.
I have to head out for the day and am on travel next week. What's the
timeline for this work? Do you need a quick and less elegant
solution? Or
do you need a more elegant long-term solution that involves updates to
the
data format and/or the compiled MET code?
Thanks,
John
On Fri, May 25, 2018 at 4:17 AM, marion.mittermaier at metoffice.gov.uk
via RT
<met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=85291 >
>
> Hello,
>
> Good news! I have found the section in the manual dealing with
aspects of
> netcdf that are just alien to me. I have managed to get mode to run
with
> the analysis field, using the level "(0,0,*,*)". See attached
postscript.
>
> However, mode is struggling to diagnose the time, setting it to
1970. I am
> wondering whether that is because the netcdf is not CF compliant. I
don't
> know how to check this... I've asked the folks creating the files to
let me
> know. The other reason may be that it is due to the fact that the
forecasts
> (and the analysis) are 25h means and the time specification is a
little
> strange, i.e. the 108h forecast is valid at 12 UTC but is an average
> between 00:00 and 00:00 (or it could be 23:30 to 23:30!)...
>
> I have also not tried the regrid option to use with the satellite
field. I
> will try that when I am feeling brave!
>
> Regards
> Marion
>
> P.S. the guide is brilliant but it's not always easy to find what
you
> need....
>
> -----Original Message-----
> From: met_help at ucar.edu via RT <met_help at ucar.edu>
> Sent: 24 May 2018 15:16
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Subject: [rt.rap.ucar.edu #85291] AutoReply: Daily mean ocean
forecast
> fields in netcdf with MODE
>
> Greetings,
>
> This message has been automatically generated in response to the
creation
> of a trouble ticket regarding:
> "Daily mean ocean forecast fields in netcdf with MODE", a
summary
> of which appears below.
>
> There is no need to reply to this message right now. Your ticket
has been
> assigned an ID of [rt.rap.ucar.edu #85291].
>
> Please include the string:
>
> [rt.rap.ucar.edu #85291]
>
> in the subject line of all future correspondence about this issue.
To do
> so, you may reply to this message.
>
> For more information, please see:
>
> MET Online Tutorial:
>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>
> MET Users Guide:
> https://www.dtcenter.org/met/users/docs/overview.php
>
> MET FAQs:
> https://www.dtcenter.org/met/users/support/faqs/index.php
>
> MET-Help Email Archive:
> http://mailman.ucar.edu/pipermail/met_help
>
> Thank you,
> met_help at ucar.edu
>
>
-------------------------------------------------------------------------
> Hi,
>
> We're now in a position where we actually want to start testing the
MODE
> config but we're getting stuck on the basics of getting the
> files/forecasts/formats recognised, so we need some help.
>
> First of all these fields are SST. We thought we'd start with
something
> relatively easy!
>
> One thing to note is that most fields are daily means. This is
different
> from many atmospheric NWP outputs... more common in climate.
>
> I thought it would easier to specify the analysis as the truth to
begin
> with, at least these are on the same grid, but we do have satellite
data
> which is on a different grid. They are not even on the same domain,
with
> the observations domain is larger than the model. Please advise as
to what
> to do about that? Should we preprocess outside. There is a regrid =
{}
> argument in the config file but I haven't been able to work out how
this
> works, or what it does.
>
> My experience in using MODE has been with grib and grib headers for
> atmospheric models. Having read in the netcdf files in R I see that
the
> variable we want is "votemper". As far as I can tell the "level"
will have
> to be set to "deptht", though I am unsure as to the equivalences
between
> grib and netcdf here.
>
> Run using:
>
> ${bindir}/mode ${datapath}/model/surfaceprodm_op_am-dm.gridT_
> 20180520._00.-12.nc ${datapath}/model/surfaceprodm_op_am-dm.gridT_
> 20180520._00.108.nc ../config/MODEConfig_test.cfg -outdir
${outpath}/test
>
> Now to the error message (when using the analysis):
> eld609:/project/hive/scripts > run_mode.test DEBUG 1: Default Config
File:
> /project/ukmo/meteval/share/met/config/MODEConfig_default
> DEBUG 1: Match Config File: ../config/MODEConfig_test.cfg DEBUG 1:
Merge
> Config File: ../config/MODEConfig_test.cfg
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> --> i.e. it's not recognising this is a t+108h lead time.
> WARNING:
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> DEBUG 1: Forecast File: /project/hive/input/model/
> surfaceprodm_op_am-dm.gridT_20180520._00.108.nc --> i.e. it's not
> recognising this is a t+108h lead time.
> DEBUG 1: Observation File: /project/hive/input/model/
> surfaceprodm_op_am-dm.gridT_20180520._00.-12.nc
> ERROR :
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
->
> needed 4 arguments for variable votemper, got 2 ERROR :
> HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
> #000: H5T.c line 1734 in H5Tclose(): not a datatype
> major: Invalid arguments to routine
> minor: Inappropriate type
> NetCDF: HDF error
> file: ncFile.cpp line:33
> HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
> #000: H5T.c line 1734 in H5Tclose(): not a datatype
> major: Invalid arguments to routine
> minor: Inappropriate type
> NetCDF: HDF error
> file: ncFile.cpp line:33
>
> From the R code I can see that votemper has 4 arguments, but I don't
know
> whether this refers to the same thing in the error message. I also
don't
> know what to specify in the config file as there are only two items
to
> specify: name and level.
>
> float votemper[x,y,deptht,time_counter]
> units: degC
> standard_name: potential temperature 25h mean
> _FillValue: 9.96920996838687e+36
> long_name: potential temperature 25h mean
> online_operation: ave(X)
> interval_operation: 300
> interval_write: 86400
> coordinates: time_counter deptht nav_lat nav_lon
>
> Any help gratefully received... The files are too large to send so I
have
> uploaded them to Dropbox. They can be accessed from here:
> https://www.dropbox.com/l/scl/AAClFKazy1EPxoUsXBLALbBw8Pz-x-P5eiM
> There is an analysis, forecast and satellite file.
>
> Regards
> Marion
>
>
> --
> Dr Marion Mittermaier Manager: Model Diagnostics and Novel
Methods
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 1392 884830 Fax: +44 1392 885631
> E:mail: marion.mittermaier at metoffice.gov.uk
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
> I am also the co-chair of the WMO WWRP/WGNE Joint Working Group for
> Forecast Verification Research<https://www.wmo.int/
> pages/prog/arep/wwrp/new/Forecast_Verification.html>
>
> You can now follow our science on Twitter:
@MetOffice_Sci<http://www.
> twitter.com/metoffice_sci>
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #85291] Daily mean ocean forecast fields in netcdf with MODE
From: marion.mittermaier at metoffice.gov.uk
Time: Wed May 30 06:35:30 2018
Hi John,
You know I hadn't spotted the lack of coastlines, but now that you
mention it! Because ocean forecasts don't have output over land it
looked ok to me ;-)
I will consult with the rest of the project team regarding timings. I
guess it depends on how long you think a "quick and less elegant"
solution will take as compared to the long-term elegant one. I suspect
we would want a long-term and more elegant solution eventually, but we
won't be able to wait for that to materialise ..... but I'll get back
to you.
I have been given some new netcdf files which are CF compliant (the
ones I sent you are not), and I will test these to see if the time
decoding improves. It will at least eliminate that as part of the
reason why the time decoding isn't working correctly. If it changes
things I will upload them to the Dropbox folder for you to pick up.
Probably better working with the CF compliant files anyway.
Regards
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 25 May 2018 22:11
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Cc: Ryan, Andrew <andrew.ryan at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #85291] Daily mean ocean forecast fields
in netcdf with MODE
Hi Marion,
This is John Halley Gotway. Glad you were able to make progress!
I took a quick look at this yesterday but didn't have time to delve in
too deeply. I'm concerned about 2 things in particular:
- There is no map data included in your plot which indicates that
MET isn't placing this data on the correct spot on the earth.
- As you've already found, we're not parsing the times correctly.
FYI, this doesn't help now... but one feature we're working on for the
next release is enabling the user to point MET to a Python script to
reads the data and define its metadata (projection and timing info,
for example). If that feature was in place, one way of solving this
problem would be writing a Python script to load up a data structure
for MET to use. That way we could use a scripting solution to handle
idiosyncrasies in data rather than having to change the input data
files or the compiled MET code directly.
Running in the debugger, I see the problem in extracting the grid
definition. MET get the corner and dimensions correct, but the
delta_lat and delta_lon values are wrong (set to 0):
{name = 0x541964 <_ZL16latlon_proj_type> "LatLon", lat_ll =
-19.888889312744141, lon_ll = -40.066669464111328, delta_lat = 0,
delta_lon = 0, Nlat = 375, Nlon = 297}
We are reading lat/lon data from the nav_lat and nav_lon variables
but getting constants values from the 2d arrays.
Until we get the projection info correct, the regridding logic won't
work.
I have to head out for the day and am on travel next week. What's the
timeline for this work? Do you need a quick and less elegant
solution? Or do you need a more elegant long-term solution that
involves updates to the data format and/or the compiled MET code?
Thanks,
John
On Fri, May 25, 2018 at 4:17 AM, marion.mittermaier at metoffice.gov.uk
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=85291 >
>
> Hello,
>
> Good news! I have found the section in the manual dealing with
aspects
> of netcdf that are just alien to me. I have managed to get mode to
run
> with the analysis field, using the level "(0,0,*,*)". See attached
postscript.
>
> However, mode is struggling to diagnose the time, setting it to
1970.
> I am wondering whether that is because the netcdf is not CF
compliant.
> I don't know how to check this... I've asked the folks creating the
> files to let me know. The other reason may be that it is due to the
> fact that the forecasts (and the analysis) are 25h means and the
time
> specification is a little strange, i.e. the 108h forecast is valid
at
> 12 UTC but is an average between 00:00 and 00:00 (or it could be
23:30 to 23:30!)...
>
> I have also not tried the regrid option to use with the satellite
> field. I will try that when I am feeling brave!
>
> Regards
> Marion
>
> P.S. the guide is brilliant but it's not always easy to find what
you
> need....
>
> -----Original Message-----
> From: met_help at ucar.edu via RT <met_help at ucar.edu>
> Sent: 24 May 2018 15:16
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Subject: [rt.rap.ucar.edu #85291] AutoReply: Daily mean ocean
forecast
> fields in netcdf with MODE
>
> Greetings,
>
> This message has been automatically generated in response to the
> creation of a trouble ticket regarding:
> "Daily mean ocean forecast fields in netcdf with MODE", a
> summary of which appears below.
>
> There is no need to reply to this message right now. Your ticket
has
> been assigned an ID of [rt.rap.ucar.edu #85291].
>
> Please include the string:
>
> [rt.rap.ucar.edu #85291]
>
> in the subject line of all future correspondence about this issue.
To
> do so, you may reply to this message.
>
> For more information, please see:
>
> MET Online Tutorial:
>
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>
> MET Users Guide:
> https://www.dtcenter.org/met/users/docs/overview.php
>
> MET FAQs:
> https://www.dtcenter.org/met/users/support/faqs/index.php
>
> MET-Help Email Archive:
> http://mailman.ucar.edu/pipermail/met_help
>
> Thank you,
> met_help at ucar.edu
>
>
----------------------------------------------------------------------
> ---
> Hi,
>
> We're now in a position where we actually want to start testing the
> MODE config but we're getting stuck on the basics of getting the
> files/forecasts/formats recognised, so we need some help.
>
> First of all these fields are SST. We thought we'd start with
> something relatively easy!
>
> One thing to note is that most fields are daily means. This is
> different from many atmospheric NWP outputs... more common in
climate.
>
> I thought it would easier to specify the analysis as the truth to
> begin with, at least these are on the same grid, but we do have
> satellite data which is on a different grid. They are not even on
the
> same domain, with the observations domain is larger than the model.
> Please advise as to what to do about that? Should we preprocess
> outside. There is a regrid = {} argument in the config file but I
> haven't been able to work out how this works, or what it does.
>
> My experience in using MODE has been with grib and grib headers for
> atmospheric models. Having read in the netcdf files in R I see that
> the variable we want is "votemper". As far as I can tell the "level"
> will have to be set to "deptht", though I am unsure as to the
> equivalences between grib and netcdf here.
>
> Run using:
>
> ${bindir}/mode ${datapath}/model/surfaceprodm_op_am-dm.gridT_
> 20180520._00.-12.nc ${datapath}/model/surfaceprodm_op_am-dm.gridT_
> 20180520._00.108.nc ../config/MODEConfig_test.cfg -outdir
> ${outpath}/test
>
> Now to the error message (when using the analysis):
> eld609:/project/hive/scripts > run_mode.test DEBUG 1: Default Config
File:
> /project/ukmo/meteval/share/met/config/MODEConfig_default
> DEBUG 1: Match Config File: ../config/MODEConfig_test.cfg DEBUG 1:
> Merge Config File: ../config/MODEConfig_test.cfg
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> --> i.e. it's not recognising this is a t+108h lead time.
> WARNING:
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> DEBUG 1: Forecast File: /project/hive/input/model/
> surfaceprodm_op_am-dm.gridT_20180520._00.108.nc --> i.e. it's not
> recognising this is a t+108h lead time.
> DEBUG 1: Observation File: /project/hive/input/model/
> surfaceprodm_op_am-dm.gridT_20180520._00.-12.nc
> ERROR :
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
->
> needed 4 arguments for variable votemper, got 2 ERROR :
> HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
> #000: H5T.c line 1734 in H5Tclose(): not a datatype
> major: Invalid arguments to routine
> minor: Inappropriate type
> NetCDF: HDF error
> file: ncFile.cpp line:33
> HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
> #000: H5T.c line 1734 in H5Tclose(): not a datatype
> major: Invalid arguments to routine
> minor: Inappropriate type
> NetCDF: HDF error
> file: ncFile.cpp line:33
>
> From the R code I can see that votemper has 4 arguments, but I don't
> know whether this refers to the same thing in the error message. I
> also don't know what to specify in the config file as there are only
> two items to
> specify: name and level.
>
> float votemper[x,y,deptht,time_counter]
> units: degC
> standard_name: potential temperature 25h mean
> _FillValue: 9.96920996838687e+36
> long_name: potential temperature 25h mean
> online_operation: ave(X)
> interval_operation: 300
> interval_write: 86400
> coordinates: time_counter deptht nav_lat nav_lon
>
> Any help gratefully received... The files are too large to send so I
> have uploaded them to Dropbox. They can be accessed from here:
> https://www.dropbox.com/l/scl/AAClFKazy1EPxoUsXBLALbBw8Pz-x-P5eiM
> There is an analysis, forecast and satellite file.
>
> Regards
> Marion
>
>
> --
> Dr Marion Mittermaier Manager: Model Diagnostics and Novel
Methods
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 1392 884830 Fax: +44 1392 885631
> E:mail: marion.mittermaier at metoffice.gov.uk
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
> I am also the co-chair of the WMO WWRP/WGNE Joint Working Group for
> Forecast Verification Research<https://www.wmo.int/
> pages/prog/arep/wwrp/new/Forecast_Verification.html>
>
> You can now follow our science on Twitter:
@MetOffice_Sci<http://www.
> twitter.com/metoffice_sci>
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #85291] Daily mean ocean forecast fields in netcdf with MODE
From: marion.mittermaier at metoffice.gov.uk
Time: Mon Jun 25 03:41:37 2018
Hi John,
Hope all is well in Boulder.
Have you had any more thoughts on the code changes required?
At our end we think we have workarounds that can help us out in the
short term, if your "quick" solution still takes longer than we can
wait.
1. We can adjust the date/time manually using scripts, and moving the
output files to have the correct values (to avoid overwriting).
2. Regarding the regridding we can investigate regridding before
running MODE so that the fields are already on the same grid, as per
the example we sent through. The grid may still not be read correctly
but the output should be useable.
I have yet to test CF-compliant files. I have that on my to-do list
this week. If that yields any changes to item 1 above, I will let you
know.
Let me know your thoughts when you have a moment.
Many thanks,
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 25 May 2018 22:11
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Cc: Ryan, Andrew <andrew.ryan at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #85291] Daily mean ocean forecast fields
in netcdf with MODE
Hi Marion,
This is John Halley Gotway. Glad you were able to make progress!
I took a quick look at this yesterday but didn't have time to delve in
too deeply. I'm concerned about 2 things in particular:
- There is no map data included in your plot which indicates that
MET isn't placing this data on the correct spot on the earth.
- As you've already found, we're not parsing the times correctly.
FYI, this doesn't help now... but one feature we're working on for the
next release is enabling the user to point MET to a Python script to
reads the data and define its metadata (projection and timing info,
for example). If that feature was in place, one way of solving this
problem would be writing a Python script to load up a data structure
for MET to use. That way we could use a scripting solution to handle
idiosyncrasies in data rather than having to change the input data
files or the compiled MET code directly.
Running in the debugger, I see the problem in extracting the grid
definition. MET get the corner and dimensions correct, but the
delta_lat and delta_lon values are wrong (set to 0):
{name = 0x541964 <_ZL16latlon_proj_type> "LatLon", lat_ll =
-19.888889312744141, lon_ll = -40.066669464111328, delta_lat = 0,
delta_lon = 0, Nlat = 375, Nlon = 297}
We are reading lat/lon data from the nav_lat and nav_lon variables
but getting constants values from the 2d arrays.
Until we get the projection info correct, the regridding logic won't
work.
I have to head out for the day and am on travel next week. What's the
timeline for this work? Do you need a quick and less elegant
solution? Or do you need a more elegant long-term solution that
involves updates to the data format and/or the compiled MET code?
Thanks,
John
On Fri, May 25, 2018 at 4:17 AM, marion.mittermaier at metoffice.gov.uk
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=85291 >
>
> Hello,
>
> Good news! I have found the section in the manual dealing with
aspects
> of netcdf that are just alien to me. I have managed to get mode to
run
> with the analysis field, using the level "(0,0,*,*)". See attached
postscript.
>
> However, mode is struggling to diagnose the time, setting it to
1970.
> I am wondering whether that is because the netcdf is not CF
compliant.
> I don't know how to check this... I've asked the folks creating the
> files to let me know. The other reason may be that it is due to the
> fact that the forecasts (and the analysis) are 25h means and the
time
> specification is a little strange, i.e. the 108h forecast is valid
at
> 12 UTC but is an average between 00:00 and 00:00 (or it could be
23:30 to 23:30!)...
>
> I have also not tried the regrid option to use with the satellite
> field. I will try that when I am feeling brave!
>
> Regards
> Marion
>
> P.S. the guide is brilliant but it's not always easy to find what
you
> need....
>
> -----Original Message-----
> From: met_help at ucar.edu via RT <met_help at ucar.edu>
> Sent: 24 May 2018 15:16
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Subject: [rt.rap.ucar.edu #85291] AutoReply: Daily mean ocean
forecast
> fields in netcdf with MODE
>
> Greetings,
>
> This message has been automatically generated in response to the
> creation of a trouble ticket regarding:
> "Daily mean ocean forecast fields in netcdf with MODE", a
> summary of which appears below.
>
> There is no need to reply to this message right now. Your ticket
has
> been assigned an ID of [rt.rap.ucar.edu #85291].
>
> Please include the string:
>
> [rt.rap.ucar.edu #85291]
>
> in the subject line of all future correspondence about this issue.
To
> do so, you may reply to this message.
>
> For more information, please see:
>
> MET Online Tutorial:
>
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>
> MET Users Guide:
> https://www.dtcenter.org/met/users/docs/overview.php
>
> MET FAQs:
> https://www.dtcenter.org/met/users/support/faqs/index.php
>
> MET-Help Email Archive:
> http://mailman.ucar.edu/pipermail/met_help
>
> Thank you,
> met_help at ucar.edu
>
>
----------------------------------------------------------------------
> ---
> Hi,
>
> We're now in a position where we actually want to start testing the
> MODE config but we're getting stuck on the basics of getting the
> files/forecasts/formats recognised, so we need some help.
>
> First of all these fields are SST. We thought we'd start with
> something relatively easy!
>
> One thing to note is that most fields are daily means. This is
> different from many atmospheric NWP outputs... more common in
climate.
>
> I thought it would easier to specify the analysis as the truth to
> begin with, at least these are on the same grid, but we do have
> satellite data which is on a different grid. They are not even on
the
> same domain, with the observations domain is larger than the model.
> Please advise as to what to do about that? Should we preprocess
> outside. There is a regrid = {} argument in the config file but I
> haven't been able to work out how this works, or what it does.
>
> My experience in using MODE has been with grib and grib headers for
> atmospheric models. Having read in the netcdf files in R I see that
> the variable we want is "votemper". As far as I can tell the "level"
> will have to be set to "deptht", though I am unsure as to the
> equivalences between grib and netcdf here.
>
> Run using:
>
> ${bindir}/mode ${datapath}/model/surfaceprodm_op_am-dm.gridT_
> 20180520._00.-12.nc ${datapath}/model/surfaceprodm_op_am-dm.gridT_
> 20180520._00.108.nc ../config/MODEConfig_test.cfg -outdir
> ${outpath}/test
>
> Now to the error message (when using the analysis):
> eld609:/project/hive/scripts > run_mode.test DEBUG 1: Default Config
File:
> /project/ukmo/meteval/share/met/config/MODEConfig_default
> DEBUG 1: Match Config File: ../config/MODEConfig_test.cfg DEBUG 1:
> Merge Config File: ../config/MODEConfig_test.cfg
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> --> i.e. it's not recognising this is a t+108h lead time.
> WARNING:
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> DEBUG 1: Forecast File: /project/hive/input/model/
> surfaceprodm_op_am-dm.gridT_20180520._00.108.nc --> i.e. it's not
> recognising this is a t+108h lead time.
> DEBUG 1: Observation File: /project/hive/input/model/
> surfaceprodm_op_am-dm.gridT_20180520._00.-12.nc
> ERROR :
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
->
> needed 4 arguments for variable votemper, got 2 ERROR :
> HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
> #000: H5T.c line 1734 in H5Tclose(): not a datatype
> major: Invalid arguments to routine
> minor: Inappropriate type
> NetCDF: HDF error
> file: ncFile.cpp line:33
> HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
> #000: H5T.c line 1734 in H5Tclose(): not a datatype
> major: Invalid arguments to routine
> minor: Inappropriate type
> NetCDF: HDF error
> file: ncFile.cpp line:33
>
> From the R code I can see that votemper has 4 arguments, but I don't
> know whether this refers to the same thing in the error message. I
> also don't know what to specify in the config file as there are only
> two items to
> specify: name and level.
>
> float votemper[x,y,deptht,time_counter]
> units: degC
> standard_name: potential temperature 25h mean
> _FillValue: 9.96920996838687e+36
> long_name: potential temperature 25h mean
> online_operation: ave(X)
> interval_operation: 300
> interval_write: 86400
> coordinates: time_counter deptht nav_lat nav_lon
>
> Any help gratefully received... The files are too large to send so I
> have uploaded them to Dropbox. They can be accessed from here:
> https://www.dropbox.com/l/scl/AAClFKazy1EPxoUsXBLALbBw8Pz-x-P5eiM
> There is an analysis, forecast and satellite file.
>
> Regards
> Marion
>
>
> --
> Dr Marion Mittermaier Manager: Model Diagnostics and Novel
Methods
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 1392 884830 Fax: +44 1392 885631
> E:mail: marion.mittermaier at metoffice.gov.uk
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
> I am also the co-chair of the WMO WWRP/WGNE Joint Working Group for
> Forecast Verification Research<https://www.wmo.int/
> pages/prog/arep/wwrp/new/Forecast_Verification.html>
>
> You can now follow our science on Twitter:
@MetOffice_Sci<http://www.
> twitter.com/metoffice_sci>
>
>
>
>
------------------------------------------------
Subject: Daily mean ocean forecast fields in netcdf with MODE
From: marion.mittermaier at metoffice.gov.uk
Time: Thu Jun 28 01:18:28 2018
Hi John et al.,
I have rerun MODE with the CF-compliant netcdf files and the good news
is that it has fixed some of the issues. It does seem to correct the
time stamp issue but raises a warning, and this may still be because
the forecasts are daily means. See below.
As far as the coastlines are concerned, the CF compliance has also
seems to sort this issue. See attached. The grid decoding seems to
have worked. However, I haven't managed to get the regridding to work
with the satellite data because it says it "can't get data" (see the
second set of output below). As I've never used the regridding
facility before, I'm a bit stumped. The ob file is viewable using
ncview so it's not corrupt. I have uploaded the CF compliant files to
the Dropbox folder I used previously.
Any help gratefully received.
Thanks,
Marion
===============================================================================
EBUG 1: Default Config File:
/project/ukmo/meteval/share/met/config/MODEConfig_default
DEBUG 1: Match Config File: ../config/MODEConfig_test.cfg
DEBUG 1: Merge Config File: ../config/MODEConfig_test.cfg
DEBUG 1: Forecast File: /project/hive/input/model/MetO-NWS-PHYS-dm-
TEM_2018-06-13_Forecast_00.036.nc
DEBUG 1: Observation File: /project/hive/input/model/MetO-NWS-PHYS-dm-
TEM_2018-06-13_Forecast_00.-12.nc
WARNING:
WARNING: setup_fcst_obs_data() -> Forecast and observation valid times
do not match 20180614_120000 != 20180612_120000 for votemper(0,0,*,*)
versus votemper(0,0,*,*).
WARNING:
DEBUG 1: Forecast Field: votemper at 0,0,*,*
DEBUG 1: Observation Field: votemper at 0,0,*,*
DEBUG 2: Processing masking regions.
DEBUG 2: Identifying objects in the forecast and observation fields...
DEBUG 2: Computing contingency table statistics...
DEBUG 2: Identified: 38 forecast objects and 45 observation objects.
DEBUG 2: Performing merging (NONE) in the forecast field.
DEBUG 2: Performing merging (NONE) in the observation field.
DEBUG 2: Remaining: 38 forecast objects and 45 observation objects.
DEBUG 2: Performing matching (MERGE_BOTH) between the forecast and
observation fields.
DEBUG 1: Creating Fcst-Obs Object Statistics file:
/project/hive/output/mode/test/mode_000000L_20180614_120000V_000000A_obj.txt
DEBUG 1: Creating Contingency Table Statistics file:
/project/hive/output/mode/test/mode_000000L_20180614_120000V_000000A_cts.txt
DEBUG 1: Creating Object NetCDF file:
/project/hive/output/mode/test/mode_000000L_20180614_120000V_000000A_obj.nc
DEBUG 1: Loading forecast raw color table:
/project/ukmo/meteval/share/met/colortables/met_default.ctable
DEBUG 1: Loading observation raw color table:
/project/ukmo/meteval/share/met/colortables/met_default.ctable
DEBUG 1: Creating postscript file:
/project/hive/output/mode/test/mode_000000L_20180614_120000V_000000A.ps
Trying to use satellite product setting regrid to FCST. File is there
and can be viewed using ncview...
DEBUG 1: Default Config File:
/project/ukmo/meteval/share/met/config/MODEConfig_default
DEBUG 1: Match Config File: ../config/MODEConfig_test.cfg
DEBUG 1: Merge Config File: ../config/MODEConfig_test.cfg
DEBUG 1: Forecast File: /project/hive/input/model/MetO-NWS-PHYS-dm-
TEM_2018-06-13_Forecast_00.036.nc
DEBUG 1: Observation File:
/project/hive/input/observations/METEOFRANCE-EUR-
SST_L3MULTISENSOR_NRT-OBS_FULL_TIME_SERIE_2018-06-13.nc
ERROR :
ERROR : setup_fcst_obs_data() -> can't get data from file
"/project/hive/input/observations/METEOFRANCE-EUR-
SST_L3MULTISENSOR_NRT-OBS_FULL_TIME_SERIE_2018-06-13.nc"
ERROR :
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 25 May 2018 22:11
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Cc: Ryan, Andrew <andrew.ryan at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #85291] Daily mean ocean forecast fields
in netcdf with MODE
Hi Marion,
This is John Halley Gotway. Glad you were able to make progress!
I took a quick look at this yesterday but didn't have time to delve in
too deeply. I'm concerned about 2 things in particular:
- There is no map data included in your plot which indicates that
MET isn't placing this data on the correct spot on the earth.
- As you've already found, we're not parsing the times correctly.
FYI, this doesn't help now... but one feature we're working on for the
next release is enabling the user to point MET to a Python script to
reads the data and define its metadata (projection and timing info,
for example). If that feature was in place, one way of solving this
problem would be writing a Python script to load up a data structure
for MET to use. That way we could use a scripting solution to handle
idiosyncrasies in data rather than having to change the input data
files or the compiled MET code directly.
Running in the debugger, I see the problem in extracting the grid
definition. MET get the corner and dimensions correct, but the
delta_lat and delta_lon values are wrong (set to 0):
{name = 0x541964 <_ZL16latlon_proj_type> "LatLon", lat_ll =
-19.888889312744141, lon_ll = -40.066669464111328, delta_lat = 0,
delta_lon = 0, Nlat = 375, Nlon = 297}
We are reading lat/lon data from the nav_lat and nav_lon variables
but getting constants values from the 2d arrays.
Until we get the projection info correct, the regridding logic won't
work.
I have to head out for the day and am on travel next week. What's the
timeline for this work? Do you need a quick and less elegant
solution? Or do you need a more elegant long-term solution that
involves updates to the data format and/or the compiled MET code?
Thanks,
John
On Fri, May 25, 2018 at 4:17 AM, marion.mittermaier at metoffice.gov.uk
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=85291 >
>
> Hello,
>
> Good news! I have found the section in the manual dealing with
aspects
> of netcdf that are just alien to me. I have managed to get mode to
run
> with the analysis field, using the level "(0,0,*,*)". See attached
postscript.
>
> However, mode is struggling to diagnose the time, setting it to
1970.
> I am wondering whether that is because the netcdf is not CF
compliant.
> I don't know how to check this... I've asked the folks creating the
> files to let me know. The other reason may be that it is due to the
> fact that the forecasts (and the analysis) are 25h means and the
time
> specification is a little strange, i.e. the 108h forecast is valid
at
> 12 UTC but is an average between 00:00 and 00:00 (or it could be
23:30 to 23:30!)...
>
> I have also not tried the regrid option to use with the satellite
> field. I will try that when I am feeling brave!
>
> Regards
> Marion
>
> P.S. the guide is brilliant but it's not always easy to find what
you
> need....
>
> -----Original Message-----
> From: met_help at ucar.edu via RT <met_help at ucar.edu>
> Sent: 24 May 2018 15:16
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Subject: [rt.rap.ucar.edu #85291] AutoReply: Daily mean ocean
forecast
> fields in netcdf with MODE
>
> Greetings,
>
> This message has been automatically generated in response to the
> creation of a trouble ticket regarding:
> "Daily mean ocean forecast fields in netcdf with MODE", a
> summary of which appears below.
>
> There is no need to reply to this message right now. Your ticket
has
> been assigned an ID of [rt.rap.ucar.edu #85291].
>
> Please include the string:
>
> [rt.rap.ucar.edu #85291]
>
> in the subject line of all future correspondence about this issue.
To
> do so, you may reply to this message.
>
> For more information, please see:
>
> MET Online Tutorial:
>
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>
> MET Users Guide:
> https://www.dtcenter.org/met/users/docs/overview.php
>
> MET FAQs:
> https://www.dtcenter.org/met/users/support/faqs/index.php
>
> MET-Help Email Archive:
> http://mailman.ucar.edu/pipermail/met_help
>
> Thank you,
> met_help at ucar.edu
>
>
----------------------------------------------------------------------
> ---
> Hi,
>
> We're now in a position where we actually want to start testing the
> MODE config but we're getting stuck on the basics of getting the
> files/forecasts/formats recognised, so we need some help.
>
> First of all these fields are SST. We thought we'd start with
> something relatively easy!
>
> One thing to note is that most fields are daily means. This is
> different from many atmospheric NWP outputs... more common in
climate.
>
> I thought it would easier to specify the analysis as the truth to
> begin with, at least these are on the same grid, but we do have
> satellite data which is on a different grid. They are not even on
the
> same domain, with the observations domain is larger than the model.
> Please advise as to what to do about that? Should we preprocess
> outside. There is a regrid = {} argument in the config file but I
> haven't been able to work out how this works, or what it does.
>
> My experience in using MODE has been with grib and grib headers for
> atmospheric models. Having read in the netcdf files in R I see that
> the variable we want is "votemper". As far as I can tell the "level"
> will have to be set to "deptht", though I am unsure as to the
> equivalences between grib and netcdf here.
>
> Run using:
>
> ${bindir}/mode ${datapath}/model/surfaceprodm_op_am-dm.gridT_
> 20180520._00.-12.nc ${datapath}/model/surfaceprodm_op_am-dm.gridT_
> 20180520._00.108.nc ../config/MODEConfig_test.cfg -outdir
> ${outpath}/test
>
> Now to the error message (when using the analysis):
> eld609:/project/hive/scripts > run_mode.test DEBUG 1: Default Config
File:
> /project/ukmo/meteval/share/met/config/MODEConfig_default
> DEBUG 1: Match Config File: ../config/MODEConfig_test.cfg DEBUG 1:
> Merge Config File: ../config/MODEConfig_test.cfg
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> --> i.e. it's not recognising this is a t+108h lead time.
> WARNING:
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> DEBUG 1: Forecast File: /project/hive/input/model/
> surfaceprodm_op_am-dm.gridT_20180520._00.108.nc --> i.e. it's not
> recognising this is a t+108h lead time.
> DEBUG 1: Observation File: /project/hive/input/model/
> surfaceprodm_op_am-dm.gridT_20180520._00.-12.nc
> ERROR :
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
->
> needed 4 arguments for variable votemper, got 2 ERROR :
> HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
> #000: H5T.c line 1734 in H5Tclose(): not a datatype
> major: Invalid arguments to routine
> minor: Inappropriate type
> NetCDF: HDF error
> file: ncFile.cpp line:33
> HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
> #000: H5T.c line 1734 in H5Tclose(): not a datatype
> major: Invalid arguments to routine
> minor: Inappropriate type
> NetCDF: HDF error
> file: ncFile.cpp line:33
>
> From the R code I can see that votemper has 4 arguments, but I don't
> know whether this refers to the same thing in the error message. I
> also don't know what to specify in the config file as there are only
> two items to
> specify: name and level.
>
> float votemper[x,y,deptht,time_counter]
> units: degC
> standard_name: potential temperature 25h mean
> _FillValue: 9.96920996838687e+36
> long_name: potential temperature 25h mean
> online_operation: ave(X)
> interval_operation: 300
> interval_write: 86400
> coordinates: time_counter deptht nav_lat nav_lon
>
> Any help gratefully received... The files are too large to send so I
> have uploaded them to Dropbox. They can be accessed from here:
> https://www.dropbox.com/l/scl/AAClFKazy1EPxoUsXBLALbBw8Pz-x-P5eiM
> There is an analysis, forecast and satellite file.
>
> Regards
> Marion
>
>
> --
> Dr Marion Mittermaier Manager: Model Diagnostics and Novel
Methods
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 1392 884830 Fax: +44 1392 885631
> E:mail: marion.mittermaier at metoffice.gov.uk
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
> I am also the co-chair of the WMO WWRP/WGNE Joint Working Group for
> Forecast Verification Research<https://www.wmo.int/
> pages/prog/arep/wwrp/new/Forecast_Verification.html>
>
> You can now follow our science on Twitter:
@MetOffice_Sci<http://www.
> twitter.com/metoffice_sci>
>
>
>
>
------------------------------------------------
Subject: Daily mean ocean forecast fields in netcdf with MODE
From: John Halley Gotway
Time: Thu Jun 28 16:48:20 2018
Marion,
Sorry for the delay in responding. Thanks for sending the sample data
file. I *think* this is just an issue in the MODE configuration file
you're using. Looking in the log output that you sent for your
successful
run I see these lines:
DEBUG 1: Forecast Field: votemper at 0,0,*,*
DEBUG 1: Observation Field: votemper at 0,0,*,*
That means that MODE is reading a data variable named votemper from
both
the forecast and observation files. Using that same configuration
file for
comparing against the satellite obs produces the error message you're
seeing:
ERROR : setup_fcst_obs_data() -> can't get data from file
"METEOFRANCE-EUR-SST_L3MULTISENSOR_NRT-OBS_FULL_TIME_SERIE_2018-06-
13.nc"
The issue is that this data file doesn't contain a variable named
"votemper". Running "ncdump -h" on it reveals this data variable
name:
short sea_surface_temperature(time, lat, lon) ;
So I modified the MODE config file (see attached), re-ran, and it
worked.
**BUT** the object definition criteria resulted in 488 observation
objects,
which took a long time to process.
So I used plot_data_plane to plot it (see attached
sea_surface_temperature.png).
/usr/local/met-7.0/bin/plot_data_plane \
METEOFRANCE-EUR-SST_L3MULTISENSOR_NRT-OBS_FULL_TIME_SERIE_2018-06-
13.nc \
sea_surface_temperature.ps \
'name="sea_surface_temperature"; level="(0,*,*)";'
I believe that all those dark gray areas are missing data values. And
perhaps that's why MODE creates so many observation objects. You
could
convert those missing data values to a real data value by "censoring"
the
data:
/usr/local/met-7.0/bin/plot_data_plane \
METEOFRANCE-EUR-SST_L3MULTISENSOR_NRT-OBS_FULL_TIME_SERIE_2018-06-
13.nc \
sea_surface_temperature_CENSOR.ps \
'name="sea_surface_temperature"; level="(0,*,*)"; censor_thresh=eq-
9999;
censor_val=0;'
That'd convert those bad data values to a value of 0 (see attached
image).
Of course, you could choose a more appropriate value than 0.
And you can use that same "censor_thresh" and "censor_val" logic in
the
MODE config file (see attached). Still results in 124 observation
objects
though! See attached first page of the MODE output.
Hope that helps.
Thanks,
John
On Thu, Jun 28, 2018 at 1:19 AM marion.mittermaier at metoffice.gov.uk
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=85291 >
>
> Hi John et al.,
>
> I have rerun MODE with the CF-compliant netcdf files and the good
news is
> that it has fixed some of the issues. It does seem to correct the
time
> stamp issue but raises a warning, and this may still be because the
> forecasts are daily means. See below.
>
> As far as the coastlines are concerned, the CF compliance has also
seems
> to sort this issue. See attached. The grid decoding seems to have
worked.
> However, I haven't managed to get the regridding to work with the
satellite
> data because it says it "can't get data" (see the second set of
output
> below). As I've never used the regridding facility before, I'm a
bit
> stumped. The ob file is viewable using ncview so it's not corrupt. I
have
> uploaded the CF compliant files to the Dropbox folder I used
previously.
>
> Any help gratefully received.
>
> Thanks,
> Marion
>
>
>
===============================================================================
> EBUG 1: Default Config File:
> /project/ukmo/meteval/share/met/config/MODEConfig_default
> DEBUG 1: Match Config File: ../config/MODEConfig_test.cfg
> DEBUG 1: Merge Config File: ../config/MODEConfig_test.cfg
> DEBUG 1: Forecast File: /project/hive/input/model/
> MetO-NWS-PHYS-dm-TEM_2018-06-13_Forecast_00.036.nc
> DEBUG 1: Observation File:
> /project/hive/input/model/MetO-NWS-PHYS-dm-TEM_2018-06-
13_Forecast_00.-
> 12.nc
> WARNING:
> WARNING: setup_fcst_obs_data() -> Forecast and observation valid
times do
> not match 20180614_120000 != 20180612_120000 for votemper(0,0,*,*)
versus
> votemper(0,0,*,*).
> WARNING:
> DEBUG 1: Forecast Field: votemper at 0,0,*,*
> DEBUG 1: Observation Field: votemper at 0,0,*,*
> DEBUG 2: Processing masking regions.
> DEBUG 2: Identifying objects in the forecast and observation
fields...
> DEBUG 2: Computing contingency table statistics...
> DEBUG 2: Identified: 38 forecast objects and 45 observation objects.
> DEBUG 2: Performing merging (NONE) in the forecast field.
> DEBUG 2: Performing merging (NONE) in the observation field.
> DEBUG 2: Remaining: 38 forecast objects and 45 observation objects.
> DEBUG 2: Performing matching (MERGE_BOTH) between the forecast and
> observation fields.
> DEBUG 1: Creating Fcst-Obs Object Statistics file:
>
/project/hive/output/mode/test/mode_000000L_20180614_120000V_000000A_obj.txt
> DEBUG 1: Creating Contingency Table Statistics file:
>
/project/hive/output/mode/test/mode_000000L_20180614_120000V_000000A_cts.txt
> DEBUG 1: Creating Object NetCDF file:
>
/project/hive/output/mode/test/mode_000000L_20180614_120000V_000000A_obj.nc
> DEBUG 1: Loading forecast raw color table:
> /project/ukmo/meteval/share/met/colortables/met_default.ctable
> DEBUG 1: Loading observation raw color table:
> /project/ukmo/meteval/share/met/colortables/met_default.ctable
> DEBUG 1: Creating postscript file:
>
/project/hive/output/mode/test/mode_000000L_20180614_120000V_000000A.ps
>
> Trying to use satellite product setting regrid to FCST. File is
there and
> can be viewed using ncview...
>
> DEBUG 1: Default Config File:
> /project/ukmo/meteval/share/met/config/MODEConfig_default
> DEBUG 1: Match Config File: ../config/MODEConfig_test.cfg
> DEBUG 1: Merge Config File: ../config/MODEConfig_test.cfg
> DEBUG 1: Forecast File: /project/hive/input/model/
> MetO-NWS-PHYS-dm-TEM_2018-06-13_Forecast_00.036.nc
> DEBUG 1: Observation File:
> /project/hive/input/observations/METEOFRANCE-EUR-
SST_L3MULTISENSOR_NRT-OBS_FULL_TIME_SERIE_2018-06-13.nc
> ERROR :
> ERROR : setup_fcst_obs_data() -> can't get data from file
> "/project/hive/input/observations/METEOFRANCE-EUR-
SST_L3MULTISENSOR_NRT-OBS_FULL_TIME_SERIE_2018-06-13.nc"
> ERROR :
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 25 May 2018 22:11
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Cc: Ryan, Andrew <andrew.ryan at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #85291] Daily mean ocean forecast
fields in
> netcdf with MODE
>
> Hi Marion,
>
> This is John Halley Gotway. Glad you were able to make progress!
>
> I took a quick look at this yesterday but didn't have time to delve
in too
> deeply. I'm concerned about 2 things in particular:
> - There is no map data included in your plot which indicates that
MET
> isn't placing this data on the correct spot on the earth.
> - As you've already found, we're not parsing the times correctly.
>
> FYI, this doesn't help now... but one feature we're working on for
the
> next release is enabling the user to point MET to a Python script to
reads
> the data and define its metadata (projection and timing info, for
> example). If that feature was in place, one way of solving this
problem
> would be writing a Python script to load up a data structure for MET
to
> use. That way we could use a scripting solution to handle
idiosyncrasies
> in data rather than having to change the input data files or the
compiled
> MET code directly.
>
> Running in the debugger, I see the problem in extracting the grid
> definition. MET get the corner and dimensions correct, but the
delta_lat
> and delta_lon values are wrong (set to 0):
> {name = 0x541964 <_ZL16latlon_proj_type> "LatLon", lat_ll =
> -19.888889312744141, lon_ll = -40.066669464111328, delta_lat = 0,
delta_lon
> = 0, Nlat = 375, Nlon = 297}
> We are reading lat/lon data from the nav_lat and nav_lon variables
but
> getting constants values from the 2d arrays.
>
> Until we get the projection info correct, the regridding logic won't
work.
>
> I have to head out for the day and am on travel next week. What's
the
> timeline for this work? Do you need a quick and less elegant
solution? Or
> do you need a more elegant long-term solution that involves updates
to the
> data format and/or the compiled MET code?
>
> Thanks,
> John
>
>
>
>
>
> On Fri, May 25, 2018 at 4:17 AM, marion.mittermaier at metoffice.gov.uk
via
> RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=85291 >
> >
> > Hello,
> >
> > Good news! I have found the section in the manual dealing with
aspects
> > of netcdf that are just alien to me. I have managed to get mode to
run
> > with the analysis field, using the level "(0,0,*,*)". See attached
> postscript.
> >
> > However, mode is struggling to diagnose the time, setting it to
1970.
> > I am wondering whether that is because the netcdf is not CF
compliant.
> > I don't know how to check this... I've asked the folks creating
the
> > files to let me know. The other reason may be that it is due to
the
> > fact that the forecasts (and the analysis) are 25h means and the
time
> > specification is a little strange, i.e. the 108h forecast is valid
at
> > 12 UTC but is an average between 00:00 and 00:00 (or it could be
23:30
> to 23:30!)...
> >
> > I have also not tried the regrid option to use with the satellite
> > field. I will try that when I am feeling brave!
> >
> > Regards
> > Marion
> >
> > P.S. the guide is brilliant but it's not always easy to find what
you
> > need....
> >
> > -----Original Message-----
> > From: met_help at ucar.edu via RT <met_help at ucar.edu>
> > Sent: 24 May 2018 15:16
> > To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> > Subject: [rt.rap.ucar.edu #85291] AutoReply: Daily mean ocean
forecast
> > fields in netcdf with MODE
> >
> > Greetings,
> >
> > This message has been automatically generated in response to the
> > creation of a trouble ticket regarding:
> > "Daily mean ocean forecast fields in netcdf with MODE", a
> > summary of which appears below.
> >
> > There is no need to reply to this message right now. Your ticket
has
> > been assigned an ID of [rt.rap.ucar.edu #85291].
> >
> > Please include the string:
> >
> > [rt.rap.ucar.edu #85291]
> >
> > in the subject line of all future correspondence about this issue.
To
> > do so, you may reply to this message.
> >
> > For more information, please see:
> >
> > MET Online Tutorial:
> >
> >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> >
> > MET Users Guide:
> > https://www.dtcenter.org/met/users/docs/overview.php
> >
> > MET FAQs:
> > https://www.dtcenter.org/met/users/support/faqs/index.php
> >
> > MET-Help Email Archive:
> > http://mailman.ucar.edu/pipermail/met_help
> >
> > Thank you,
> > met_help at ucar.edu
> >
> >
----------------------------------------------------------------------
> > ---
> > Hi,
> >
> > We're now in a position where we actually want to start testing
the
> > MODE config but we're getting stuck on the basics of getting the
> > files/forecasts/formats recognised, so we need some help.
> >
> > First of all these fields are SST. We thought we'd start with
> > something relatively easy!
> >
> > One thing to note is that most fields are daily means. This is
> > different from many atmospheric NWP outputs... more common in
climate.
> >
> > I thought it would easier to specify the analysis as the truth to
> > begin with, at least these are on the same grid, but we do have
> > satellite data which is on a different grid. They are not even on
the
> > same domain, with the observations domain is larger than the
model.
> > Please advise as to what to do about that? Should we preprocess
> > outside. There is a regrid = {} argument in the config file but I
> > haven't been able to work out how this works, or what it does.
> >
> > My experience in using MODE has been with grib and grib headers
for
> > atmospheric models. Having read in the netcdf files in R I see
that
> > the variable we want is "votemper". As far as I can tell the
"level"
> > will have to be set to "deptht", though I am unsure as to the
> > equivalences between grib and netcdf here.
> >
> > Run using:
> >
> > ${bindir}/mode ${datapath}/model/surfaceprodm_op_am-dm.gridT_
> > 20180520._00.-12.nc ${datapath}/model/surfaceprodm_op_am-dm.gridT_
> > 20180520._00.108.nc ../config/MODEConfig_test.cfg -outdir
> > ${outpath}/test
> >
> > Now to the error message (when using the analysis):
> > eld609:/project/hive/scripts > run_mode.test DEBUG 1: Default
Config
> File:
> > /project/ukmo/meteval/share/met/config/MODEConfig_default
> > DEBUG 1: Match Config File: ../config/MODEConfig_test.cfg DEBUG 1:
> > Merge Config File: ../config/MODEConfig_test.cfg
> > WARNING:
> > WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> > --> i.e. it's not recognising this is a t+108h lead time.
> > WARNING:
> > WARNING:
> > WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> > WARNING:
> > DEBUG 1: Forecast File: /project/hive/input/model/
> > surfaceprodm_op_am-dm.gridT_20180520._00.108.nc --> i.e. it's not
> > recognising this is a t+108h lead time.
> > DEBUG 1: Observation File: /project/hive/input/model/
> > surfaceprodm_op_am-dm.gridT_20180520._00.-12.nc
> > ERROR :
> > ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&) ->
> > needed 4 arguments for variable votemper, got 2 ERROR :
> > HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
> > #000: H5T.c line 1734 in H5Tclose(): not a datatype
> > major: Invalid arguments to routine
> > minor: Inappropriate type
> > NetCDF: HDF error
> > file: ncFile.cpp line:33
> > HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
> > #000: H5T.c line 1734 in H5Tclose(): not a datatype
> > major: Invalid arguments to routine
> > minor: Inappropriate type
> > NetCDF: HDF error
> > file: ncFile.cpp line:33
> >
> > From the R code I can see that votemper has 4 arguments, but I
don't
> > know whether this refers to the same thing in the error message. I
> > also don't know what to specify in the config file as there are
only
> > two items to
> > specify: name and level.
> >
> > float votemper[x,y,deptht,time_counter]
> > units: degC
> > standard_name: potential temperature 25h mean
> > _FillValue: 9.96920996838687e+36
> > long_name: potential temperature 25h mean
> > online_operation: ave(X)
> > interval_operation: 300
> > interval_write: 86400
> > coordinates: time_counter deptht nav_lat nav_lon
> >
> > Any help gratefully received... The files are too large to send so
I
> > have uploaded them to Dropbox. They can be accessed from here:
> > https://www.dropbox.com/l/scl/AAClFKazy1EPxoUsXBLALbBw8Pz-x-P5eiM
> > There is an analysis, forecast and satellite file.
> >
> > Regards
> > Marion
> >
> >
> > --
> > Dr Marion Mittermaier Manager: Model Diagnostics and Novel
Methods
> >
> > Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> > Tel: +44 1392 884830 Fax: +44 1392 885631
> > E:mail: marion.mittermaier at metoffice.gov.uk
> > http://www.metoffice.gov.uk/research/people/marion-mittermaier
> >
> > I am also the co-chair of the WMO WWRP/WGNE Joint Working Group
for
> > Forecast Verification Research<https://www.wmo.int/
> > pages/prog/arep/wwrp/new/Forecast_Verification.html>
> >
> > You can now follow our science on Twitter:
@MetOffice_Sci<http://www.
> > twitter.com/metoffice_sci>
> >
> >
> >
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list