[Met_help] [rt.rap.ucar.edu #99831] History for Bad Init Time in NetCDF file
John Halley Gotway via RT
met_help at ucar.edu
Mon May 10 11:47:43 MDT 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
John, I am currently verifying global ensemble probability data against a model analysis (GALWEM) using Grid Stat. I am using NCEP climo data also. Running verification, in the NetCDF pairs file the climo data init_time_ut is negative since climo data is by definition from the past. So, now I am using the NetCDF gridstat pairs file in SeriesAnalysis and that is where the trouble occurs. Even though I am not using climo data in the SeriesAnalConfig file, I still get an error when processing the NetCDF pairs file:
ERROR : get_att_unixtime(const NcVarInfo &, const ConcatString &, unixtime &) -> attribute "init_time_ut" should be an integer or a string.
I have attached the two configuration files and sent the NetCDF file via DOD Safe. So a couple questions:
1) Am I doing something wrong causing the negative climo UT time in the NetCDF file produced by Gridstat?
2) Has this been corrected in later versions of MET (we are still on MET 8.1 - don't ask)?
3) Why should SeriesAnalysis care about this if climo is not selected in the configuration file?
Worst case, I will have to write a script to correct the ut time in NetCDF gridstat file. Is there an easy algorithm to calculate the correct UT time?
Thanks
Bob Craig
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Bad Init Time in NetCDF file
From: John Halley Gotway
Time: Mon May 10 11:13:53 2021
Hi Bob,
I see you're getting an error from MET's NetCDF library code about
parsing
timestamps. Thanks for sending the sample data so that I can replicate
this
behavior.
When I run plot_data_plane from version 8.1.2 (the latest bugfix
version of
8.1), I do see that error message:
*met-8.1.2/bin/plot_data_plane \*
*grid_stat_geps_all_960000L_20210415_000000V_pairs.nc \*
*CLIMO_MEAN_GUST_Z10_FULL.ps
'name="CLIMO_MEAN_GUST_Z10_FULL";level="(*,*)";'*
*ERROR : get_att_unixtime(const NcVarInfo &, const ConcatString &,
unixtime &) -> attribute "init_time_ut" should be an integer or a
string.*
Running version 9.1.3 (the latest bugfix version of 9.1), I DO NOT see
that
error message:
*met-9.1.3/bin/plot_data_plane \*
*grid_stat_geps_all_960000L_20210415_000000V_pairs.nc
CLIMO_MEAN_GUST_Z10_FULL.ps \
'name="CLIMO_MEAN_GUST_Z10_FULL";level="(*,*)";'*
Here's the resulting image:
[image: Screen Shot 2021-05-10 at 10.59.49 AM.png]
If needed, I could probably find the GitHub issue which fixed this
behavior. But if you can't recompile anything that won't help.
So how to proceed...
One option is disabling the gridded climo output from Grid-Stat by
turning
it off in the "nc_pairs_flag" in the Grid-Stat config file:
*nc_pairs_flag = {*
*...*
* climo = FALSE;*
*...*
*}*
That'd prevent the timestamps from being written, which would prevent
them
from causing a problem in Series-Analysis.
Honestly, I'm not sure why that's triggering a problem in Series-
Analysis.
Perhaps the NetCDF library code is parsing metadata from all the
variables
in the file when it's opened? When I plot a non-climo variable like
"OBS_GUST_Z10_SHEM" with 8.1.2 I do also see that error. So I assume
it's
inventorying the whole file.
As you mentioned, another option could be changing the timestamps in
that
NetCDF file.
But that's unfortunate and could get messy. In the past, I've done
that
using the ncatted and date commands via shell scripts. But am I right
in
assuming you'd use python for that instead?
Thanks,
John
On Mon, May 10, 2021 at 10:44 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Mon May 10 10:44:24 2021: Request 99831 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Bad Init Time in NetCDF file
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99831 >
>
>
> John, I am currently verifying global ensemble probability data
against a
> model analysis (GALWEM) using Grid Stat. I am using NCEP climo data
also.
> Running verification, in the NetCDF pairs file the climo data
init_time_ut
> is negative since climo data is by definition from the past. So,
now I am
> using the NetCDF gridstat pairs file in SeriesAnalysis and that is
where
> the trouble occurs. Even though I am not using climo data in the
> SeriesAnalConfig file, I still get an error when processing the
NetCDF
> pairs file:
>
> ERROR : get_att_unixtime(const NcVarInfo &, const ConcatString &,
> unixtime &) -> attribute "init_time_ut" should be an integer or a
string.
>
> I have attached the two configuration files and sent the NetCDF file
via
> DOD Safe. So a couple questions:
>
>
> 1) Am I doing something wrong causing the negative climo UT
time in
> the NetCDF file produced by Gridstat?
>
> 2) Has this been corrected in later versions of MET (we are
still on
> MET 8.1 - don't ask)?
>
> 3) Why should SeriesAnalysis care about this if climo is not
selected
> in the configuration file?
>
> Worst case, I will have to write a script to correct the ut time in
NetCDF
> gridstat file. Is there an easy algorithm to calculate the correct
UT
> time?
>
> Thanks
> Bob Craig
>
>
>
>
------------------------------------------------
Subject: Bad Init Time in NetCDF file
From: robert.craig.2 at us.af.mil
Time: Mon May 10 11:28:09 2021
Yes, python is what we use for scripting but if the code is fixed in
9.x we will wait until then. I will turn off climo in the NetCDF file
for now.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Monday, May 10, 2021 12:14 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
Cc: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXD
<christopher.melick at us.af.mil>; LACK, STEVEN A GS-13 USAF ACC 16
WS/WXD <steven.lack at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #99831] Bad Init Time
in NetCDF file
Hi Bob,
I see you're getting an error from MET's NetCDF library code about
parsing timestamps. Thanks for sending the sample data so that I can
replicate this behavior.
When I run plot_data_plane from version 8.1.2 (the latest bugfix
version of 8.1), I do see that error message:
*met-8.1.2/bin/plot_data_plane \*
*grid_stat_geps_all_960000L_20210415_000000V_pairs.nc \*
*CLIMO_MEAN_GUST_Z10_FULL.ps
'name="CLIMO_MEAN_GUST_Z10_FULL";level="(*,*)";'*
*ERROR : get_att_unixtime(const NcVarInfo &, const ConcatString &,
unixtime &) -> attribute "init_time_ut" should be an integer or a
string.*
Running version 9.1.3 (the latest bugfix version of 9.1), I DO NOT see
that error message:
*met-9.1.3/bin/plot_data_plane \*
*grid_stat_geps_all_960000L_20210415_000000V_pairs.nc
CLIMO_MEAN_GUST_Z10_FULL.ps \
'name="CLIMO_MEAN_GUST_Z10_FULL";level="(*,*)";'*
Here's the resulting image:
[image: Screen Shot 2021-05-10 at 10.59.49 AM.png]
If needed, I could probably find the GitHub issue which fixed this
behavior. But if you can't recompile anything that won't help.
So how to proceed...
One option is disabling the gridded climo output from Grid-Stat by
turning it off in the "nc_pairs_flag" in the Grid-Stat config file:
*nc_pairs_flag = {*
*...*
* climo = FALSE;*
*...*
*}*
That'd prevent the timestamps from being written, which would prevent
them from causing a problem in Series-Analysis.
Honestly, I'm not sure why that's triggering a problem in Series-
Analysis.
Perhaps the NetCDF library code is parsing metadata from all the
variables in the file when it's opened? When I plot a non-climo
variable like "OBS_GUST_Z10_SHEM" with 8.1.2 I do also see that error.
So I assume it's inventorying the whole file.
As you mentioned, another option could be changing the timestamps in
that NetCDF file.
But that's unfortunate and could get messy. In the past, I've done
that using the ncatted and date commands via shell scripts. But am I
right in assuming you'd use python for that instead?
Thanks,
John
On Mon, May 10, 2021 at 10:44 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Mon May 10 10:44:24 2021: Request 99831 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Bad Init Time in NetCDF file
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99831
> >
>
>
> John, I am currently verifying global ensemble probability data
> against a model analysis (GALWEM) using Grid Stat. I am using NCEP
climo data also.
> Running verification, in the NetCDF pairs file the climo data
> init_time_ut is negative since climo data is by definition from the
> past. So, now I am using the NetCDF gridstat pairs file in
> SeriesAnalysis and that is where the trouble occurs. Even though I
am
> not using climo data in the SeriesAnalConfig file, I still get an
> error when processing the NetCDF pairs file:
>
> ERROR : get_att_unixtime(const NcVarInfo &, const ConcatString &,
> unixtime &) -> attribute "init_time_ut" should be an integer or a
string.
>
> I have attached the two configuration files and sent the NetCDF file
via
> DOD Safe. So a couple questions:
>
>
> 1) Am I doing something wrong causing the negative climo UT
time in
> the NetCDF file produced by Gridstat?
>
> 2) Has this been corrected in later versions of MET (we are
still on
> MET 8.1 - don't ask)?
>
> 3) Why should SeriesAnalysis care about this if climo is not
selected
> in the configuration file?
>
> Worst case, I will have to write a script to correct the ut time in
> NetCDF gridstat file. Is there an easy algorithm to calculate the
> correct UT time?
>
> Thanks
> Bob Craig
>
>
>
>
------------------------------------------------
Subject: Bad Init Time in NetCDF file
From: John Halley Gotway
Time: Mon May 10 11:47:09 2021
Bob,
OK, and FYI, we are planning on doing the MET version 10.0.0 release
literally today.
I know things move slowly, but perhaps the AF will be able to skip
over the
9.1 release entirely.
I'll go ahead and resolve this ticket.
Thanks,
John
On Mon, May 10, 2021 at 11:28 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99831 >
>
> Yes, python is what we use for scripting but if the code is fixed in
9.x
> we will wait until then. I will turn off climo in the NetCDF file
for now.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Monday, May 10, 2021 12:14 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
> Cc: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXD <
> christopher.melick at us.af.mil>; LACK, STEVEN A GS-13 USAF ACC 16
WS/WXD <
> steven.lack at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #99831] Bad Init Time
in
> NetCDF file
>
> Hi Bob,
>
> I see you're getting an error from MET's NetCDF library code about
parsing
> timestamps. Thanks for sending the sample data so that I can
replicate this
> behavior.
>
> When I run plot_data_plane from version 8.1.2 (the latest bugfix
version
> of 8.1), I do see that error message:
>
> *met-8.1.2/bin/plot_data_plane \*
> *grid_stat_geps_all_960000L_20210415_000000V_pairs.nc \*
>
> *CLIMO_MEAN_GUST_Z10_FULL.ps
> 'name="CLIMO_MEAN_GUST_Z10_FULL";level="(*,*)";'*
>
> *ERROR : get_att_unixtime(const NcVarInfo &, const ConcatString &,
> unixtime &) -> attribute "init_time_ut" should be an integer or a
string.*
>
> Running version 9.1.3 (the latest bugfix version of 9.1), I DO NOT
see
> that error message:
>
> *met-9.1.3/bin/plot_data_plane \*
> *grid_stat_geps_all_960000L_20210415_000000V_pairs.nc
> CLIMO_MEAN_GUST_Z10_FULL.ps \
> 'name="CLIMO_MEAN_GUST_Z10_FULL";level="(*,*)";'*
>
> Here's the resulting image:
> [image: Screen Shot 2021-05-10 at 10.59.49 AM.png]
>
> If needed, I could probably find the GitHub issue which fixed this
> behavior. But if you can't recompile anything that won't help.
>
> So how to proceed...
>
> One option is disabling the gridded climo output from Grid-Stat by
turning
> it off in the "nc_pairs_flag" in the Grid-Stat config file:
> *nc_pairs_flag = {*
> *...*
>
> * climo = FALSE;*
> *...*
> *}*
>
> That'd prevent the timestamps from being written, which would
prevent them
> from causing a problem in Series-Analysis.
>
> Honestly, I'm not sure why that's triggering a problem in Series-
Analysis.
> Perhaps the NetCDF library code is parsing metadata from all the
variables
> in the file when it's opened? When I plot a non-climo variable like
> "OBS_GUST_Z10_SHEM" with 8.1.2 I do also see that error. So I assume
it's
> inventorying the whole file.
>
> As you mentioned, another option could be changing the timestamps in
that
> NetCDF file.
> But that's unfortunate and could get messy. In the past, I've done
that
> using the ncatted and date commands via shell scripts. But am I
right in
> assuming you'd use python for that instead?
>
> Thanks,
> John
>
> On Mon, May 10, 2021 at 10:44 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon May 10 10:44:24 2021: Request 99831 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: Bad Init Time in NetCDF file
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99831
> > >
> >
> >
> > John, I am currently verifying global ensemble probability data
> > against a model analysis (GALWEM) using Grid Stat. I am using
NCEP
> climo data also.
> > Running verification, in the NetCDF pairs file the climo data
> > init_time_ut is negative since climo data is by definition from
the
> > past. So, now I am using the NetCDF gridstat pairs file in
> > SeriesAnalysis and that is where the trouble occurs. Even though
I am
> > not using climo data in the SeriesAnalConfig file, I still get an
> > error when processing the NetCDF pairs file:
> >
> > ERROR : get_att_unixtime(const NcVarInfo &, const ConcatString &,
> > unixtime &) -> attribute "init_time_ut" should be an integer or a
string.
> >
> > I have attached the two configuration files and sent the NetCDF
file via
> > DOD Safe. So a couple questions:
> >
> >
> > 1) Am I doing something wrong causing the negative climo UT
time in
> > the NetCDF file produced by Gridstat?
> >
> > 2) Has this been corrected in later versions of MET (we are
still on
> > MET 8.1 - don't ask)?
> >
> > 3) Why should SeriesAnalysis care about this if climo is not
> selected
> > in the configuration file?
> >
> > Worst case, I will have to write a script to correct the ut time
in
> > NetCDF gridstat file. Is there an easy algorithm to calculate
the
> > correct UT time?
> >
> > Thanks
> > Bob Craig
> >
> >
> >
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list