[Met_help] [rt.rap.ucar.edu #80838] History for Weird netCDF variable names
John Halley Gotway via RT
met_help at ucar.edu
Thu Jun 15 09:59:55 MDT 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
I have been outputting the pairs netCDF files with gridstat and now I would
like to use them to produce a map of monthly average error maps. This
means I need to average over time for each lead time using a month's worth
of gridstat output.
My problem is that the netCDF files have weird names because I set the
dimension as "(0,*,*)" for example. So the variable I want to average is
called:
"DIFF_TMP2m_0\,\*\,\*_TMP_R1_CONUS". I am unable to use this name with nco
netcdf tools because of the slashes and commas. Do you know how I can
change this variable name?
Best,
Shannon
--
Shannon Rees
Meteorologist
Engility
Geophysical Fluid Dynamics Lab
Princeton University Forrestal Campus
201 Forrestal Rd Princeton, NJ
609-452-5384
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Weird netCDF variable names
From: John Halley Gotway
Time: Mon Jun 12 15:36:10 2017
Shannon,
I see what you mean! I tried using "ncrename" (which I believe is an
NCO
tool) on one such variable and the embedded commas made it very
unhappy.
Going forward, a simple solution would be having MET change those
commas to
underscores when writing the variable name. Do you agree that's the
way to
go?
But what to do about your existing data? I see 2 options:
(1) Use ncdump and ncgen to switch those commas to underscores:
ncdump in.nc | sed 's/\\,/_/g' > in.dump
ncgen -o out.nc in.dump
Then the NCO tools should be happy processing the "out.nc" files.
(2) Instead of using the NCO tools for this task, use the MET
Series-Analysis tool instead. This is exactly what it was designed to
do.
Pass the Grid-Stat matched pairs output files to Series-Analysis using
the
"-both" command line option. Then have it compute a gridded map of
mean
error (ME) from the continuous statistics (cnt) setting in the
output_stats
section:
cnt = [ "ME" ];
Additionally, you can compute gridded output for any of the other
statistics that MET computes. All you have to do is request them in
the
config file.
Hope that helps.
Thanks,
John
On Mon, Jun 12, 2017 at 12:36 PM, Shannon Rees - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Mon Jun 12 12:36:46 2017: Request 80838 was acted upon.
> Transaction: Ticket created by shannon.rees at noaa.gov
> Queue: met_help
> Subject: Weird netCDF variable names
> Owner: Nobody
> Requestors: shannon.rees at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80838 >
>
>
> Hello,
>
> I have been outputting the pairs netCDF files with gridstat and now
I would
> like to use them to produce a map of monthly average error maps.
This
> means I need to average over time for each lead time using a month's
worth
> of gridstat output.
>
> My problem is that the netCDF files have weird names because I set
the
> dimension as "(0,*,*)" for example. So the variable I want to
average is
> called:
> "DIFF_TMP2m_0\,\*\,\*_TMP_R1_CONUS". I am unable to use this name
with
> nco
> netcdf tools because of the slashes and commas. Do you know how I
can
> change this variable name?
>
> Best,
> Shannon
> --
> Shannon Rees
> Meteorologist
> Engility
> Geophysical Fluid Dynamics Lab
> Princeton University Forrestal Campus
> 201 Forrestal Rd Princeton, NJ
> 609-452-5384
>
>
------------------------------------------------
Subject: Weird netCDF variable names
From: Shannon Rees - NOAA Affiliate
Time: Tue Jun 13 11:44:52 2017
Hi John,
Thanks for reminding me of series-analysis! That is much preferable
to
writing my own script of nco commands.
I do agree underscores would be better than commas for the variable
names.
However, in my case I may prefer to not always include the level in
the
name, although I can see why it would be necessary sometimes. I am
using
RTMA 2-meter temperature grib data as my observations and there is a
strange problem where the record which contains the actual 2m TMP
field
varies from file to file (each file is one time). So sometimes I need
to
use the first record (level=R1), but sometimes it is the second
(level=R2). I don't see any rhyme or reason to it.
Using series-analysis to get a map of the mean statistics of all the
matched pairs seems to be difficult with and without the fix. If I
change
the variable name to use underscores, how do I edit the config file so
that
it will look for the correct variable name? For example, if I change
the
forecast variable name in the grid-stat output to
"FCST_TMP2m_0_\*_\*_CONUS" then in the config file I set:
fcst = {
field = [
{
name = "FCST_TMP2m_0_\*_\*_";
level = [ "CONUS" ];
}
];
}
obs = {
field = [
{
name = "OBS_TMP_R1_";
level = [ "CONUS" ];
}
];
}
I get this error from series-analysis:
"WARNING: get_series_entry() -> No match found for
FCST_TMP2m_0_*_*_CONUS in file: ../../output/series_analysis2/
grid_stat_060000L_20170401_pairs.nc"
WARNING:
ERROR :
ERROR : get_series_entry() -> Could not find data for
OBS_TMP_R1_CONUS in
file list:"
I realize "CONUS" is not the level name and that is probably an issue
here,
but series-analysis seems to name the variable by appending level to
the
end of the name. I also tried completely taking the level out of the
fcst
and obs name in the grid-stat output, and not specifying a level in
the
series-analysis config file, but it complains that it needs the level
info. Any idea what to do in this situation?
Thank you,
Shannon
On Mon, Jun 12, 2017 at 5:36 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Shannon,
>
> I see what you mean! I tried using "ncrename" (which I believe is
an NCO
> tool) on one such variable and the embedded commas made it very
unhappy.
>
> Going forward, a simple solution would be having MET change those
commas to
> underscores when writing the variable name. Do you agree that's the
way to
> go?
>
> But what to do about your existing data? I see 2 options:
>
> (1) Use ncdump and ncgen to switch those commas to underscores:
>
> ncdump in.nc | sed 's/\\,/_/g' > in.dump
> ncgen -o out.nc in.dump
>
> Then the NCO tools should be happy processing the "out.nc" files.
>
> (2) Instead of using the NCO tools for this task, use the MET
> Series-Analysis tool instead. This is exactly what it was designed
to do.
> Pass the Grid-Stat matched pairs output files to Series-Analysis
using the
> "-both" command line option. Then have it compute a gridded map of
mean
> error (ME) from the continuous statistics (cnt) setting in the
output_stats
> section:
> cnt = [ "ME" ];
>
> Additionally, you can compute gridded output for any of the other
> statistics that MET computes. All you have to do is request them in
the
> config file.
>
> Hope that helps.
>
> Thanks,
> John
>
>
>
>
>
> On Mon, Jun 12, 2017 at 12:36 PM, Shannon Rees - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Jun 12 12:36:46 2017: Request 80838 was acted upon.
> > Transaction: Ticket created by shannon.rees at noaa.gov
> > Queue: met_help
> > Subject: Weird netCDF variable names
> > Owner: Nobody
> > Requestors: shannon.rees at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80838 >
> >
> >
> > Hello,
> >
> > I have been outputting the pairs netCDF files with gridstat and
now I
> would
> > like to use them to produce a map of monthly average error maps.
This
> > means I need to average over time for each lead time using a
month's
> worth
> > of gridstat output.
> >
> > My problem is that the netCDF files have weird names because I set
the
> > dimension as "(0,*,*)" for example. So the variable I want to
average is
> > called:
> > "DIFF_TMP2m_0\,\*\,\*_TMP_R1_CONUS". I am unable to use this name
with
> > nco
> > netcdf tools because of the slashes and commas. Do you know how I
can
> > change this variable name?
> >
> > Best,
> > Shannon
> > --
> > Shannon Rees
> > Meteorologist
> > Engility
> > Geophysical Fluid Dynamics Lab
> > Princeton University Forrestal Campus
> > 201 Forrestal Rd Princeton, NJ
> > 609-452-5384
> >
> >
>
>
--
Shannon Rees
Meteorologist
Engility
Geophysical Fluid Dynamics Lab
Princeton University Forrestal Campus
201 Forrestal Rd Princeton, NJ
609-452-5384 <(609)%20452-5384>
------------------------------------------------
Subject: Weird netCDF variable names
From: John Halley Gotway
Time: Tue Jun 13 15:10:55 2017
Shannon,
I just tested this out using MET version 6.0 to make sure that
Series-Analysis works OK on this data.
(1) First I used ncdump to look at the variable names present in the
Grid-Stat matched pairs output file:
[johnhg at number5]% ncdump -h
grid_stat_GTG_latlon_060000L_20130827_180000V_pairs.nc | grep float
...
float FCST_edr_0\,0\,\*\,\*_FULL(lat, lon) ;
float OBS_edr_0\,0\,\*\,\*_FULL(lat, lon) ;
(2) Then I set up the Series-Analysis config file to reference these
fcst
and obs variables (I did *not* include the backslashes):
fcst = {
field = [
{
name = "FCST_edr_0,0,*,*_FULL";
level = [ "(*,*)" ];
}
];
}
obs = {
field = [
{
name = "OBS_edr_0,0,*,*_FULL";
level = [ "(*,*)" ];
}
];
}
I also increased the default block_size a lot!
block_size = 1024000;
(3) Then I ran Series-Analysis using the "-both" option:
met-6.0/bin/series_analysis \
-both grid_stat_GTG_latlon_060000L_20130827_180000V_pairs.nc \
-config SAConfig \
-out sa_out.nc
And that worked. Of course, you probably want a series longer than
length
1!
Please give that a try and let me know how it goes.
Thanks,
John
On Tue, Jun 13, 2017 at 11:44 AM, Shannon Rees - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80838 >
>
> Hi John,
>
> Thanks for reminding me of series-analysis! That is much preferable
to
> writing my own script of nco commands.
>
> I do agree underscores would be better than commas for the variable
names.
> However, in my case I may prefer to not always include the level in
the
> name, although I can see why it would be necessary sometimes. I am
using
> RTMA 2-meter temperature grib data as my observations and there is a
> strange problem where the record which contains the actual 2m TMP
field
> varies from file to file (each file is one time). So sometimes I
need to
> use the first record (level=R1), but sometimes it is the second
> (level=R2). I don't see any rhyme or reason to it.
>
> Using series-analysis to get a map of the mean statistics of all the
> matched pairs seems to be difficult with and without the fix. If I
change
> the variable name to use underscores, how do I edit the config file
so that
> it will look for the correct variable name? For example, if I
change the
> forecast variable name in the grid-stat output to
> "FCST_TMP2m_0_\*_\*_CONUS" then in the config file I set:
>
> fcst = {
> field = [
> {
> name = "FCST_TMP2m_0_\*_\*_";
> level = [ "CONUS" ];
> }
> ];
> }
> obs = {
> field = [
> {
> name = "OBS_TMP_R1_";
> level = [ "CONUS" ];
> }
> ];
> }
>
> I get this error from series-analysis:
> "WARNING: get_series_entry() -> No match found for
> FCST_TMP2m_0_*_*_CONUS in file: ../../output/series_analysis2/
> grid_stat_060000L_20170401_pairs.nc"
> WARNING:
>
> ERROR :
>
> ERROR : get_series_entry() -> Could not find data for
OBS_TMP_R1_CONUS in
> file list:"
>
> I realize "CONUS" is not the level name and that is probably an
issue here,
> but series-analysis seems to name the variable by appending level to
the
> end of the name. I also tried completely taking the level out of
the fcst
> and obs name in the grid-stat output, and not specifying a level in
the
> series-analysis config file, but it complains that it needs the
level
> info. Any idea what to do in this situation?
>
> Thank you,
> Shannon
>
>
> On Mon, Jun 12, 2017 at 5:36 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Shannon,
> >
> > I see what you mean! I tried using "ncrename" (which I believe is
an NCO
> > tool) on one such variable and the embedded commas made it very
unhappy.
> >
> > Going forward, a simple solution would be having MET change those
commas
> to
> > underscores when writing the variable name. Do you agree that's
the way
> to
> > go?
> >
> > But what to do about your existing data? I see 2 options:
> >
> > (1) Use ncdump and ncgen to switch those commas to underscores:
> >
> > ncdump in.nc | sed 's/\\,/_/g' > in.dump
> > ncgen -o out.nc in.dump
> >
> > Then the NCO tools should be happy processing the "out.nc" files.
> >
> > (2) Instead of using the NCO tools for this task, use the MET
> > Series-Analysis tool instead. This is exactly what it was
designed to
> do.
> > Pass the Grid-Stat matched pairs output files to Series-Analysis
using
> the
> > "-both" command line option. Then have it compute a gridded map
of mean
> > error (ME) from the continuous statistics (cnt) setting in the
> output_stats
> > section:
> > cnt = [ "ME" ];
> >
> > Additionally, you can compute gridded output for any of the other
> > statistics that MET computes. All you have to do is request them
in the
> > config file.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> >
> >
> >
> >
> > On Mon, Jun 12, 2017 at 12:36 PM, Shannon Rees - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Jun 12 12:36:46 2017: Request 80838 was acted upon.
> > > Transaction: Ticket created by shannon.rees at noaa.gov
> > > Queue: met_help
> > > Subject: Weird netCDF variable names
> > > Owner: Nobody
> > > Requestors: shannon.rees at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80838
> >
> > >
> > >
> > > Hello,
> > >
> > > I have been outputting the pairs netCDF files with gridstat and
now I
> > would
> > > like to use them to produce a map of monthly average error maps.
This
> > > means I need to average over time for each lead time using a
month's
> > worth
> > > of gridstat output.
> > >
> > > My problem is that the netCDF files have weird names because I
set the
> > > dimension as "(0,*,*)" for example. So the variable I want to
average
> is
> > > called:
> > > "DIFF_TMP2m_0\,\*\,\*_TMP_R1_CONUS". I am unable to use this
name
> with
> > > nco
> > > netcdf tools because of the slashes and commas. Do you know how
I can
> > > change this variable name?
> > >
> > > Best,
> > > Shannon
> > > --
> > > Shannon Rees
> > > Meteorologist
> > > Engility
> > > Geophysical Fluid Dynamics Lab
> > > Princeton University Forrestal Campus
> > > 201 Forrestal Rd Princeton, NJ
> > > 609-452-5384
> > >
> > >
> >
> >
>
>
> --
> Shannon Rees
> Meteorologist
> Engility
> Geophysical Fluid Dynamics Lab
> Princeton University Forrestal Campus
> 201 Forrestal Rd Princeton, NJ
> 609-452-5384 <(609)%20452-5384>
>
>
------------------------------------------------
Subject: Weird netCDF variable names
From: John Halley Gotway
Time: Tue Jun 13 15:36:05 2017
Shannon,
Sorry, I should have mentioned that I was just using a sample Grid-
Stat
NetCDF matched pairs output file. The variable names present in your
data
will not match the ones present in mine. But mine do have the
troublesome
commas in there.
If you're running Series-Analysis, there's no need to change the
commas to
underscores.
Thanks,
John
On Tue, Jun 13, 2017 at 3:10 PM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Shannon,
>
> I just tested this out using MET version 6.0 to make sure that
> Series-Analysis works OK on this data.
>
> (1) First I used ncdump to look at the variable names present in the
> Grid-Stat matched pairs output file:
>
> [johnhg at number5]% ncdump -h
grid_stat_GTG_latlon_060000L_20130827_180000V_pairs.nc
> | grep float
> ...
> float FCST_edr_0\,0\,\*\,\*_FULL(lat, lon) ;
> float OBS_edr_0\,0\,\*\,\*_FULL(lat, lon) ;
>
> (2) Then I set up the Series-Analysis config file to reference these
fcst
> and obs variables (I did *not* include the backslashes):
>
> fcst = {
> field = [
> {
> name = "FCST_edr_0,0,*,*_FULL";
> level = [ "(*,*)" ];
> }
> ];
> }
>
> obs = {
> field = [
> {
> name = "OBS_edr_0,0,*,*_FULL";
> level = [ "(*,*)" ];
> }
> ];
> }
>
> I also increased the default block_size a lot!
> block_size = 1024000;
>
> (3) Then I ran Series-Analysis using the "-both" option:
>
> met-6.0/bin/series_analysis \
> -both grid_stat_GTG_latlon_060000L_20130827_180000V_pairs.nc \
> -config SAConfig \
> -out sa_out.nc
>
> And that worked. Of course, you probably want a series longer than
length
> 1!
>
> Please give that a try and let me know how it goes.
>
> Thanks,
> John
>
>
>
>
> On Tue, Jun 13, 2017 at 11:44 AM, Shannon Rees - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80838 >
>>
>> Hi John,
>>
>> Thanks for reminding me of series-analysis! That is much
preferable to
>> writing my own script of nco commands.
>>
>> I do agree underscores would be better than commas for the variable
names.
>> However, in my case I may prefer to not always include the level in
the
>> name, although I can see why it would be necessary sometimes. I am
using
>> RTMA 2-meter temperature grib data as my observations and there is
a
>> strange problem where the record which contains the actual 2m TMP
field
>> varies from file to file (each file is one time). So sometimes I
need to
>> use the first record (level=R1), but sometimes it is the second
>> (level=R2). I don't see any rhyme or reason to it.
>>
>> Using series-analysis to get a map of the mean statistics of all
the
>> matched pairs seems to be difficult with and without the fix. If I
change
>> the variable name to use underscores, how do I edit the config file
so
>> that
>> it will look for the correct variable name? For example, if I
change the
>> forecast variable name in the grid-stat output to
>> "FCST_TMP2m_0_\*_\*_CONUS" then in the config file I set:
>>
>> fcst = {
>> field = [
>> {
>> name = "FCST_TMP2m_0_\*_\*_";
>> level = [ "CONUS" ];
>> }
>> ];
>> }
>> obs = {
>> field = [
>> {
>> name = "OBS_TMP_R1_";
>> level = [ "CONUS" ];
>> }
>> ];
>> }
>>
>> I get this error from series-analysis:
>> "WARNING: get_series_entry() -> No match found for
>> FCST_TMP2m_0_*_*_CONUS in file: ../../output/series_analysis2/
>> grid_stat_060000L_20170401_pairs.nc"
>> WARNING:
>>
>> ERROR :
>>
>> ERROR : get_series_entry() -> Could not find data for
OBS_TMP_R1_CONUS in
>> file list:"
>>
>> I realize "CONUS" is not the level name and that is probably an
issue
>> here,
>> but series-analysis seems to name the variable by appending level
to the
>> end of the name. I also tried completely taking the level out of
the fcst
>> and obs name in the grid-stat output, and not specifying a level in
the
>> series-analysis config file, but it complains that it needs the
level
>> info. Any idea what to do in this situation?
>>
>> Thank you,
>> Shannon
>>
>>
>> On Mon, Jun 12, 2017 at 5:36 PM, John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>> > Shannon,
>> >
>> > I see what you mean! I tried using "ncrename" (which I believe
is an
>> NCO
>> > tool) on one such variable and the embedded commas made it very
unhappy.
>> >
>> > Going forward, a simple solution would be having MET change those
>> commas to
>> > underscores when writing the variable name. Do you agree that's
the
>> way to
>> > go?
>> >
>> > But what to do about your existing data? I see 2 options:
>> >
>> > (1) Use ncdump and ncgen to switch those commas to underscores:
>> >
>> > ncdump in.nc | sed 's/\\,/_/g' > in.dump
>> > ncgen -o out.nc in.dump
>> >
>> > Then the NCO tools should be happy processing the "out.nc" files.
>> >
>> > (2) Instead of using the NCO tools for this task, use the MET
>> > Series-Analysis tool instead. This is exactly what it was
designed to
>> do.
>> > Pass the Grid-Stat matched pairs output files to Series-Analysis
using
>> the
>> > "-both" command line option. Then have it compute a gridded map
of mean
>> > error (ME) from the continuous statistics (cnt) setting in the
>> output_stats
>> > section:
>> > cnt = [ "ME" ];
>> >
>> > Additionally, you can compute gridded output for any of the other
>> > statistics that MET computes. All you have to do is request them
in the
>> > config file.
>> >
>> > Hope that helps.
>> >
>> > Thanks,
>> > John
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Jun 12, 2017 at 12:36 PM, Shannon Rees - NOAA Affiliate
via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Mon Jun 12 12:36:46 2017: Request 80838 was acted upon.
>> > > Transaction: Ticket created by shannon.rees at noaa.gov
>> > > Queue: met_help
>> > > Subject: Weird netCDF variable names
>> > > Owner: Nobody
>> > > Requestors: shannon.rees at noaa.gov
>> > > Status: new
>> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80838
>> >
>> > >
>> > >
>> > > Hello,
>> > >
>> > > I have been outputting the pairs netCDF files with gridstat and
now I
>> > would
>> > > like to use them to produce a map of monthly average error
maps. This
>> > > means I need to average over time for each lead time using a
month's
>> > worth
>> > > of gridstat output.
>> > >
>> > > My problem is that the netCDF files have weird names because I
set the
>> > > dimension as "(0,*,*)" for example. So the variable I want to
>> average is
>> > > called:
>> > > "DIFF_TMP2m_0\,\*\,\*_TMP_R1_CONUS". I am unable to use this
name
>> with
>> > > nco
>> > > netcdf tools because of the slashes and commas. Do you know
how I can
>> > > change this variable name?
>> > >
>> > > Best,
>> > > Shannon
>> > > --
>> > > Shannon Rees
>> > > Meteorologist
>> > > Engility
>> > > Geophysical Fluid Dynamics Lab
>> > > Princeton University Forrestal Campus
>> > > 201 Forrestal Rd Princeton, NJ
>> > > 609-452-5384
>> > >
>> > >
>> >
>> >
>>
>>
>> --
>> Shannon Rees
>> Meteorologist
>> Engility
>> Geophysical Fluid Dynamics Lab
>> Princeton University Forrestal Campus
>> 201 Forrestal Rd Princeton, NJ
>> 609-452-5384 <(609)%20452-5384>
>>
>>
>
------------------------------------------------
Subject: Weird netCDF variable names
From: Shannon Rees - NOAA Affiliate
Time: Wed Jun 14 16:07:45 2017
John,
I followed your instructions and it is working now.
Thanks!
Shannon
On Tue, Jun 13, 2017 at 5:10 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Shannon,
>
> I just tested this out using MET version 6.0 to make sure that
> Series-Analysis works OK on this data.
>
> (1) First I used ncdump to look at the variable names present in the
> Grid-Stat matched pairs output file:
>
> [johnhg at number5]% ncdump -h
> grid_stat_GTG_latlon_060000L_20130827_180000V_pairs.nc | grep float
> ...
> float FCST_edr_0\,0\,\*\,\*_FULL(lat, lon) ;
> float OBS_edr_0\,0\,\*\,\*_FULL(lat, lon) ;
>
> (2) Then I set up the Series-Analysis config file to reference these
fcst
> and obs variables (I did *not* include the backslashes):
>
> fcst = {
> field = [
> {
> name = "FCST_edr_0,0,*,*_FULL";
> level = [ "(*,*)" ];
> }
> ];
> }
>
> obs = {
> field = [
> {
> name = "OBS_edr_0,0,*,*_FULL";
> level = [ "(*,*)" ];
> }
> ];
> }
>
> I also increased the default block_size a lot!
> block_size = 1024000;
>
> (3) Then I ran Series-Analysis using the "-both" option:
>
> met-6.0/bin/series_analysis \
> -both grid_stat_GTG_latlon_060000L_20130827_180000V_pairs.nc \
> -config SAConfig \
> -out sa_out.nc
>
> And that worked. Of course, you probably want a series longer than
length
> 1!
>
> Please give that a try and let me know how it goes.
>
> Thanks,
> John
>
>
>
>
> On Tue, Jun 13, 2017 at 11:44 AM, Shannon Rees - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80838 >
> >
> > Hi John,
> >
> > Thanks for reminding me of series-analysis! That is much
preferable to
> > writing my own script of nco commands.
> >
> > I do agree underscores would be better than commas for the
variable
> names.
> > However, in my case I may prefer to not always include the level
in the
> > name, although I can see why it would be necessary sometimes. I
am using
> > RTMA 2-meter temperature grib data as my observations and there is
a
> > strange problem where the record which contains the actual 2m TMP
field
> > varies from file to file (each file is one time). So sometimes I
need to
> > use the first record (level=R1), but sometimes it is the second
> > (level=R2). I don't see any rhyme or reason to it.
> >
> > Using series-analysis to get a map of the mean statistics of all
the
> > matched pairs seems to be difficult with and without the fix. If
I
> change
> > the variable name to use underscores, how do I edit the config
file so
> that
> > it will look for the correct variable name? For example, if I
change the
> > forecast variable name in the grid-stat output to
> > "FCST_TMP2m_0_\*_\*_CONUS" then in the config file I set:
> >
> > fcst = {
> > field = [
> > {
> > name = "FCST_TMP2m_0_\*_\*_";
> > level = [ "CONUS" ];
> > }
> > ];
> > }
> > obs = {
> > field = [
> > {
> > name = "OBS_TMP_R1_";
> > level = [ "CONUS" ];
> > }
> > ];
> > }
> >
> > I get this error from series-analysis:
> > "WARNING: get_series_entry() -> No match found for
> > FCST_TMP2m_0_*_*_CONUS in file: ../../output/series_analysis2/
> > grid_stat_060000L_20170401_pairs.nc"
> > WARNING:
> >
> > ERROR :
> >
> > ERROR : get_series_entry() -> Could not find data for
OBS_TMP_R1_CONUS
> in
> > file list:"
> >
> > I realize "CONUS" is not the level name and that is probably an
issue
> here,
> > but series-analysis seems to name the variable by appending level
to the
> > end of the name. I also tried completely taking the level out of
the
> fcst
> > and obs name in the grid-stat output, and not specifying a level
in the
> > series-analysis config file, but it complains that it needs the
level
> > info. Any idea what to do in this situation?
> >
> > Thank you,
> > Shannon
> >
> >
> > On Mon, Jun 12, 2017 at 5:36 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Shannon,
> > >
> > > I see what you mean! I tried using "ncrename" (which I believe
is an
> NCO
> > > tool) on one such variable and the embedded commas made it very
> unhappy.
> > >
> > > Going forward, a simple solution would be having MET change
those
> commas
> > to
> > > underscores when writing the variable name. Do you agree that's
the
> way
> > to
> > > go?
> > >
> > > But what to do about your existing data? I see 2 options:
> > >
> > > (1) Use ncdump and ncgen to switch those commas to underscores:
> > >
> > > ncdump in.nc | sed 's/\\,/_/g' > in.dump
> > > ncgen -o out.nc in.dump
> > >
> > > Then the NCO tools should be happy processing the "out.nc"
files.
> > >
> > > (2) Instead of using the NCO tools for this task, use the MET
> > > Series-Analysis tool instead. This is exactly what it was
designed to
> > do.
> > > Pass the Grid-Stat matched pairs output files to Series-Analysis
using
> > the
> > > "-both" command line option. Then have it compute a gridded map
of
> mean
> > > error (ME) from the continuous statistics (cnt) setting in the
> > output_stats
> > > section:
> > > cnt = [ "ME" ];
> > >
> > > Additionally, you can compute gridded output for any of the
other
> > > statistics that MET computes. All you have to do is request
them in
> the
> > > config file.
> > >
> > > Hope that helps.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Jun 12, 2017 at 12:36 PM, Shannon Rees - NOAA Affiliate
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Jun 12 12:36:46 2017: Request 80838 was acted upon.
> > > > Transaction: Ticket created by shannon.rees at noaa.gov
> > > > Queue: met_help
> > > > Subject: Weird netCDF variable names
> > > > Owner: Nobody
> > > > Requestors: shannon.rees at noaa.gov
> > > > Status: new
> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=80838
> > >
> > > >
> > > >
> > > > Hello,
> > > >
> > > > I have been outputting the pairs netCDF files with gridstat
and now I
> > > would
> > > > like to use them to produce a map of monthly average error
maps.
> This
> > > > means I need to average over time for each lead time using a
month's
> > > worth
> > > > of gridstat output.
> > > >
> > > > My problem is that the netCDF files have weird names because I
set
> the
> > > > dimension as "(0,*,*)" for example. So the variable I want to
> average
> > is
> > > > called:
> > > > "DIFF_TMP2m_0\,\*\,\*_TMP_R1_CONUS". I am unable to use this
name
> > with
> > > > nco
> > > > netcdf tools because of the slashes and commas. Do you know
how I
> can
> > > > change this variable name?
> > > >
> > > > Best,
> > > > Shannon
> > > > --
> > > > Shannon Rees
> > > > Meteorologist
> > > > Engility
> > > > Geophysical Fluid Dynamics Lab
> > > > Princeton University Forrestal Campus
> > > > 201 Forrestal Rd Princeton, NJ
> > > > 609-452-5384
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Shannon Rees
> > Meteorologist
> > Engility
> > Geophysical Fluid Dynamics Lab
> > Princeton University Forrestal Campus
> > 201 Forrestal Rd Princeton, NJ
> > 609-452-5384 <(609)%20452-5384>
> >
> >
>
>
--
Shannon Rees
Meteorologist
Engility
Geophysical Fluid Dynamics Lab
Princeton University Forrestal Campus
201 Forrestal Rd Princeton, NJ
609-452-5384
------------------------------------------------
More information about the Met_help
mailing list