[Met_help] [rt.rap.ucar.edu #95358] History for model vs satellite AOD: Missing values
Julie Prestopnik via RT
met_help at ucar.edu
Tue Jul 14 10:05:47 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I am trying to evaluate model forecast Aerosol optical depth against
gridded Satellite observation. Observation data is average of AOD between
11z and 13z of a particular day (19th May) and contains missing values.
Verifying model forecast is 12z of that day. I am able to read the
observation data using plot_data_plane but when I am using series_analysis
to check stats it is showing 0 matched pairs. So, the questions to you :
1. How to modify config file in this case to read model vs obs which
contains missing data. Also increase matched pairs.
2. Also,observation netcdf file is generated by partnering
satellite branch to help evaluate our model result. Do you have any
suggestions on how to make the gridded netcdf file that is not a
MET-version dependent and what attributes are needed to make it
compatible with future MET release ?
Files are in WCOSS- Mars at :
Obs file
: /gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/newfiles
Forecast file:
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/
VIIRS_L3Gridded/geaer.t00z.pgrb2a.0p25.f012
Sample config :
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_stat_testVIIRS_FV3.sh
Thanks
Partha
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: model vs satellite AOD: Missing values
From: Julie Prestopnik
Time: Tue May 26 15:28:31 2020
Hi Partha.
John is not working for the next couple of days. I asked Howard to
take a
look to see if he can help. Hopefully, he will be able to take a look
and
reply. If he is unable to help, I will have John follow up when he is
working again. Thank you for your patience. We will get back to you
as
soon as we can.
Julie
On Tue, May 26, 2020 at 8:30 AM Partha Bhattacharjee - NOAA Affiliate
via
RT <met_help at ucar.edu> wrote:
>
> Tue May 26 08:29:35 2020: Request 95358 was acted upon.
> Transaction: Ticket created by partha.bhattacharjee at noaa.gov
> Queue: met_help
> Subject: model vs satellite AOD: Missing values
> Owner: Nobody
> Requestors: partha.bhattacharjee at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95358 >
>
>
> Hi,
> I am trying to evaluate model forecast Aerosol optical depth against
> gridded Satellite observation. Observation data is average of AOD
between
> 11z and 13z of a particular day (19th May) and contains missing
values.
> Verifying model forecast is 12z of that day. I am able to read the
> observation data using plot_data_plane but when I am using
series_analysis
> to check stats it is showing 0 matched pairs. So, the questions to
you :
>
> 1. How to modify config file in this case to read model vs obs
which
> contains missing data. Also increase matched pairs.
>
> 2. Also,observation netcdf file is generated by partnering
> satellite branch to help evaluate our model result. Do you have any
> suggestions on how to make the gridded netcdf file that is not a
> MET-version dependent and what attributes are needed to make it
> compatible with future MET release ?
>
> Files are in WCOSS- Mars at :
> Obs file
> :
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/newfiles
> Forecast file:
> /gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/
> VIIRS_L3Gridded/geaer.t00z.pgrb2a.0p25.f012
>
> Sample config :
>
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_stat_testVIIRS_FV3.sh
>
> Thanks
> Partha
>
>
--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: model vs satellite AOD: Missing values
From: Howard Soh
Time: Tue May 26 17:46:38 2020
There is no matching pairs because the time information is missing and
the default time ("19700101_000000") was assigned to the AOD data.
Option 1:
Add "init_time" and "valid_time" variable attributes to each data
variable (VIIRS-L3-AOD_GEFS_20200519_1200.nc).
For example,
AOD_HM_Quality:init_time = "20200519_120000" ;
AOD_HM_Quality:valid_time = "20200519_120000" ;
Option 2: Make CF-complaint NetCDF with time variable and projection
variable (standard name and units)
[MET version dependency]
The point2grid was added recently and the format of gridded point
observation NetCDF file was not changed. There is no MET version
dependent content nor format.
Cheers,
Howard
FYI:
I don't have access to WCOSS. Please upload the modified file to the
FTP server for more supports.
[How to send files for MET-help]
https://dtcenter.org/met/users/support/met_help.php
Scroll down to "How To Send Us Data" section.
On Tue May 26 08:29:35 2020, partha.bhattacharjee at noaa.gov wrote:
> Hi,
> I am trying to evaluate model forecast Aerosol optical depth against
> gridded Satellite observation. Observation data is average of AOD
> between
> 11z and 13z of a particular day (19th May) and contains missing
> values.
> Verifying model forecast is 12z of that day. I am able to read the
> observation data using plot_data_plane but when I am using
> series_analysis
> to check stats it is showing 0 matched pairs. So, the questions to
you
> :
>
> 1. How to modify config file in this case to read model vs obs
which
> contains missing data. Also increase matched pairs.
>
> 2. Also,observation netcdf file is generated by partnering
> satellite branch to help evaluate our model result. Do you have any
> suggestions on how to make the gridded netcdf file that is not a
> MET-version dependent and what attributes are needed to make it
> compatible with future MET release ?
>
> Files are in WCOSS- Mars at :
> Obs file
> :
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/newfiles
> Forecast file:
> /gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/
> VIIRS_L3Gridded/geaer.t00z.pgrb2a.0p25.f012
>
> Sample config :
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_stat_testVIIRS_FV3.sh
>
> Thanks
> Partha
------------------------------------------------
Subject: model vs satellite AOD: Missing values
From: Partha Bhattacharjee - NOAA Affiliate
Time: Tue May 26 18:11:29 2020
Thanks Howard for looking into it. I sent your response to the
satellite
team who are generating this data and I will test again (and reach MET
help) with their updated data.
On Tue, May 26, 2020 at 7:46 PM Howard Soh via RT <met_help at ucar.edu>
wrote:
> There is no matching pairs because the time information is missing
and the
> default time ("19700101_000000") was assigned to the AOD data.
>
> Option 1:
> Add "init_time" and "valid_time" variable attributes to each data
variable
> (VIIRS-L3-AOD_GEFS_20200519_1200.nc).
> For example,
> AOD_HM_Quality:init_time = "20200519_120000" ;
> AOD_HM_Quality:valid_time = "20200519_120000" ;
>
> Option 2: Make CF-complaint NetCDF with time variable and projection
> variable (standard name and units)
>
>
> [MET version dependency]
> The point2grid was added recently and the format of gridded point
> observation NetCDF file was not changed. There is no MET version
dependent
> content nor format.
>
> Cheers,
> Howard
>
> FYI:
>
> I don't have access to WCOSS. Please upload the modified file to the
FTP
> server for more supports.
>
> [How to send files for MET-help]
>
> https://dtcenter.org/met/users/support/met_help.php
> Scroll down to "How To Send Us Data" section.
>
>
> On Tue May 26 08:29:35 2020, partha.bhattacharjee at noaa.gov wrote:
> > Hi,
> > I am trying to evaluate model forecast Aerosol optical depth
against
> > gridded Satellite observation. Observation data is average of AOD
> > between
> > 11z and 13z of a particular day (19th May) and contains missing
> > values.
> > Verifying model forecast is 12z of that day. I am able to read the
> > observation data using plot_data_plane but when I am using
> > series_analysis
> > to check stats it is showing 0 matched pairs. So, the questions to
you
> > :
> >
> > 1. How to modify config file in this case to read model vs obs
which
> > contains missing data. Also increase matched pairs.
> >
> > 2. Also,observation netcdf file is generated by partnering
> > satellite branch to help evaluate our model result. Do you have
any
> > suggestions on how to make the gridded netcdf file that is not a
> > MET-version dependent and what attributes are needed to make it
> > compatible with future MET release ?
> >
> > Files are in WCOSS- Mars at :
> > Obs file
> > :
> >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/newfiles
> > Forecast file:
> > /gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/
> > VIIRS_L3Gridded/geaer.t00z.pgrb2a.0p25.f012
> >
> > Sample config :
> >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_stat_testVIIRS_FV3.sh
> >
> > Thanks
> > Partha
>
>
>
>
--
Partha S. Bhattacharjee, Ph.D.
I.M. Systems Group
Environmental Modeling Centre
NCWCP
5830 University Research Court.
College Park, MD 20740
------------------------------------------------
Subject: model vs satellite AOD: Missing values
From: Howard Soh
Time: Wed May 27 12:06:03 2020
The required variable attribute is "valid_time_ut". So the minimum
requirement is "valid_time_ut" attribute.
Please add "valid_time_ut". After adding "valid_time_ut" attribute
(command below), 74615 matched pairs were found.
DEBUG 2: Finished processing statistics for 74615 of 1036800 grid
points.
DEBUG 2: Skipped 962185 of 1036800 points with no valid data.
DEBUG 2: Skipped 0 of 1036800 points that did not meet the valid data
threshold.
Command to add variable attributes:
ncatted -a init_time,AOD_HM_Quality,a,c,"20200519_120000" VIIRS-L3-
AOD_GEFS_20200519_1200.nc
ncatted -a valid_time,AOD_HM_Quality,a,c,"20200519_120000" VIIRS-
L3-AOD_GEFS_20200519_1200.nc
ncatted -a init_time_ut,AOD_HM_Quality,a,l,1589889600 VIIRS-L3-
AOD_GEFS_20200519_1200.nc
ncatted -a valid_time_ut,AOD_HM_Quality,a,l,1589889600 VIIRS-L3-
AOD_GEFS_20200519_1200.nc
Cheers,
Howard
On Tue May 26 18:11:29 2020, partha.bhattacharjee at noaa.gov wrote:
> Thanks Howard for looking into it. I sent your response to the
> satellite
> team who are generating this data and I will test again (and reach
MET
> help) with their updated data.
>
> On Tue, May 26, 2020 at 7:46 PM Howard Soh via RT
<met_help at ucar.edu>
> wrote:
>
> > There is no matching pairs because the time information is missing
> > and the
> > default time ("19700101_000000") was assigned to the AOD data.
> >
> > Option 1:
> > Add "init_time" and "valid_time" variable attributes to each data
> > variable
> > (VIIRS-L3-AOD_GEFS_20200519_1200.nc).
> > For example,
> > AOD_HM_Quality:init_time = "20200519_120000" ;
> > AOD_HM_Quality:valid_time = "20200519_120000" ;
> >
> > Option 2: Make CF-complaint NetCDF with time variable and
projection
> > variable (standard name and units)
> >
> >
> > [MET version dependency]
> > The point2grid was added recently and the format of gridded point
> > observation NetCDF file was not changed. There is no MET version
> > dependent
> > content nor format.
> >
> > Cheers,
> > Howard
> >
> > FYI:
> >
> > I don't have access to WCOSS. Please upload the modified file to
the
> > FTP
> > server for more supports.
> >
> > [How to send files for MET-help]
> >
> > https://dtcenter.org/met/users/support/met_help.php
> > Scroll down to "How To Send Us Data" section.
> >
> >
> > On Tue May 26 08:29:35 2020, partha.bhattacharjee at noaa.gov wrote:
> > > Hi,
> > > I am trying to evaluate model forecast Aerosol optical depth
> > > against
> > > gridded Satellite observation. Observation data is average of
AOD
> > > between
> > > 11z and 13z of a particular day (19th May) and contains missing
> > > values.
> > > Verifying model forecast is 12z of that day. I am able to read
the
> > > observation data using plot_data_plane but when I am using
> > > series_analysis
> > > to check stats it is showing 0 matched pairs. So, the questions
to
> > > you
> > > :
> > >
> > > 1. How to modify config file in this case to read model vs obs
> > > which
> > > contains missing data. Also increase matched pairs.
> > >
> > > 2. Also,observation netcdf file is generated by partnering
> > > satellite branch to help evaluate our model result. Do you have
any
> > > suggestions on how to make the gridded netcdf file that is not a
> > > MET-version dependent and what attributes are needed to make it
> > > compatible with future MET release ?
> > >
> > > Files are in WCOSS- Mars at :
> > > Obs file
> > > :
> > >
> >
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/newfiles
> > > Forecast file:
> > >
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/
> > > VIIRS_L3Gridded/geaer.t00z.pgrb2a.0p25.f012
> > >
> > > Sample config :
> > >
> >
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_stat_testVIIRS_FV3.sh
> > >
> > > Thanks
> > > Partha
> >
> >
> >
> >
------------------------------------------------
Subject: model vs satellite AOD: Missing values
From: Partha Bhattacharjee - NOAA Affiliate
Time: Thu May 28 10:14:13 2020
Howard,
Thanks. valid_time_ut is MET specific function. What is the value for
“valid_time_ut”, i.e., what is the definition? Is there a
straightforward
definition of this "valid_time_ut", for example, seconds from 1970-01-
01
00:00:00UTC?
Thanks
Partha
On Wed, May 27, 2020 at 2:06 PM Howard Soh via RT <met_help at ucar.edu>
wrote:
> The required variable attribute is "valid_time_ut". So the minimum
> requirement is "valid_time_ut" attribute.
>
> Please add "valid_time_ut". After adding "valid_time_ut" attribute
> (command below), 74615 matched pairs were found.
>
> DEBUG 2: Finished processing statistics for 74615 of 1036800 grid
points.
> DEBUG 2: Skipped 962185 of 1036800 points with no valid data.
> DEBUG 2: Skipped 0 of 1036800 points that did not meet the valid
data
> threshold.
>
> Command to add variable attributes:
>
> ncatted -a init_time,AOD_HM_Quality,a,c,"20200519_120000"
> VIIRS-L3-AOD_GEFS_20200519_1200.nc
> ncatted -a valid_time,AOD_HM_Quality,a,c,"20200519_120000"
> VIIRS-L3-AOD_GEFS_20200519_1200.nc
> ncatted -a init_time_ut,AOD_HM_Quality,a,l,1589889600
> VIIRS-L3-AOD_GEFS_20200519_1200.nc
> ncatted -a valid_time_ut,AOD_HM_Quality,a,l,1589889600
> VIIRS-L3-AOD_GEFS_20200519_1200.nc
>
>
> Cheers,
> Howard
>
> On Tue May 26 18:11:29 2020, partha.bhattacharjee at noaa.gov wrote:
> > Thanks Howard for looking into it. I sent your response to the
> > satellite
> > team who are generating this data and I will test again (and reach
MET
> > help) with their updated data.
> >
> > On Tue, May 26, 2020 at 7:46 PM Howard Soh via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > There is no matching pairs because the time information is
missing
> > > and the
> > > default time ("19700101_000000") was assigned to the AOD data.
> > >
> > > Option 1:
> > > Add "init_time" and "valid_time" variable attributes to each
data
> > > variable
> > > (VIIRS-L3-AOD_GEFS_20200519_1200.nc).
> > > For example,
> > > AOD_HM_Quality:init_time = "20200519_120000" ;
> > > AOD_HM_Quality:valid_time = "20200519_120000" ;
> > >
> > > Option 2: Make CF-complaint NetCDF with time variable and
projection
> > > variable (standard name and units)
> > >
> > >
> > > [MET version dependency]
> > > The point2grid was added recently and the format of gridded
point
> > > observation NetCDF file was not changed. There is no MET version
> > > dependent
> > > content nor format.
> > >
> > > Cheers,
> > > Howard
> > >
> > > FYI:
> > >
> > > I don't have access to WCOSS. Please upload the modified file to
the
> > > FTP
> > > server for more supports.
> > >
> > > [How to send files for MET-help]
> > >
> > > https://dtcenter.org/met/users/support/met_help.php
> > > Scroll down to "How To Send Us Data" section.
> > >
> > >
> > > On Tue May 26 08:29:35 2020, partha.bhattacharjee at noaa.gov
wrote:
> > > > Hi,
> > > > I am trying to evaluate model forecast Aerosol optical depth
> > > > against
> > > > gridded Satellite observation. Observation data is average of
AOD
> > > > between
> > > > 11z and 13z of a particular day (19th May) and contains
missing
> > > > values.
> > > > Verifying model forecast is 12z of that day. I am able to read
the
> > > > observation data using plot_data_plane but when I am using
> > > > series_analysis
> > > > to check stats it is showing 0 matched pairs. So, the
questions to
> > > > you
> > > > :
> > > >
> > > > 1. How to modify config file in this case to read model vs obs
> > > > which
> > > > contains missing data. Also increase matched pairs.
> > > >
> > > > 2. Also,observation netcdf file is generated by partnering
> > > > satellite branch to help evaluate our model result. Do you
have any
> > > > suggestions on how to make the gridded netcdf file that is not
a
> > > > MET-version dependent and what attributes are needed to make
it
> > > > compatible with future MET release ?
> > > >
> > > > Files are in WCOSS- Mars at :
> > > > Obs file
> > > > :
> > > >
> > >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/newfiles
> > > > Forecast file:
> > > >
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/
> > > > VIIRS_L3Gridded/geaer.t00z.pgrb2a.0p25.f012
> > > >
> > > > Sample config :
> > > >
> > >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_stat_testVIIRS_FV3.sh
> > > >
> > > > Thanks
> > > > Partha
> > >
> > >
> > >
> > >
>
>
>
>
--
Partha S. Bhattacharjee, Ph.D.
I.M. Systems Group
Environmental Modeling Centre
NCWCP
5830 University Research Court.
College Park, MD 20740
------------------------------------------------
Subject: model vs satellite AOD: Missing values
From: John Halley Gotway
Time: Thu May 28 11:15:46 2020
Hi Partha (and Howard),
I was out of the office this week and am just taking a look at this
issue
now. I ran series_analysis using the data Julie P pulled down from
WCOSS,
like this:
*/usr/local/met-8.0/bin/series_analysis \-fcst
geaer.t00z.pgrb2a.0p25.f012
\-obs VIIRS-L3-AOD_GEFS_20200519_1200.nc \-out
output_METseries_202005test/series_fh12_HQ202005.nc \-config
SeriesAnlConfig_May21_fh12 \-paired*
And I see the behavior that Partha describes:
*DEBUG 2: Skipped 1036800 of 1036800 points with no valid data.*
And as Howard notes, the VIIRS NetCDF file lacks the "valid_time_ut"
NetCDF
variable attribute. So I ran the ncatted command Howard sent:
*ncatted -a valid_time_ut,AOD_HM_Quality,a,l,1589889600
VIIRS-L3-AOD_GEFS_20200519_1200.nc <http://1200.nc/>*
And then when I rerun the Series-Analysis command listed above, it
runs
fine. I ran plot_data_plane to plot the output variable named
"series_cnt_ME" and have attached a PNG of the resulting image.
*plot_data_plane output_METseries_202005test/series_fh12_HQ202005.nc
series_cnt_ME.ps 'name="series_cnt_ME"; level="(*,*)";'*
Partha, the "ut" at the "valid_time_ut" variable name stands for
"unixtime"
which is the number of seconds since 1970-01-01:
https://en.wikipedia.org/wiki/Unix_time
When writing shell scripts, I often use the "date" command to convert
back/forth between unixtime and human-readable dates.
For example, to convert from 20200519_120000 to unixtime, run:
*> date -ud ''2020-05-19' UTC '12:00:00'' +%s*
*1589889600*
And to convert that unixtime back to YMDH format, run:
*> date -ud '1970-01-01 UTC '1589889600' seconds' +%Y%m%d%H2020051912*
Thanks,
John
On Thu, May 28, 2020 at 10:14 AM Partha Bhattacharjee - NOAA Affiliate
via
RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95358 >
>
> Howard,
> Thanks. valid_time_ut is MET specific function. What is the value
for
> “valid_time_ut”, i.e., what is the definition? Is there a
straightforward
> definition of this "valid_time_ut", for example, seconds from 1970-
01-01
> 00:00:00UTC?
>
> Thanks
> Partha
>
>
> On Wed, May 27, 2020 at 2:06 PM Howard Soh via RT
<met_help at ucar.edu>
> wrote:
>
> > The required variable attribute is "valid_time_ut". So the minimum
> > requirement is "valid_time_ut" attribute.
> >
> > Please add "valid_time_ut". After adding "valid_time_ut" attribute
> > (command below), 74615 matched pairs were found.
> >
> > DEBUG 2: Finished processing statistics for 74615 of 1036800 grid
points.
> > DEBUG 2: Skipped 962185 of 1036800 points with no valid data.
> > DEBUG 2: Skipped 0 of 1036800 points that did not meet the valid
data
> > threshold.
> >
> > Command to add variable attributes:
> >
> > ncatted -a init_time,AOD_HM_Quality,a,c,"20200519_120000"
> > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > ncatted -a valid_time,AOD_HM_Quality,a,c,"20200519_120000"
> > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > ncatted -a init_time_ut,AOD_HM_Quality,a,l,1589889600
> > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > ncatted -a valid_time_ut,AOD_HM_Quality,a,l,1589889600
> > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> >
> >
> > Cheers,
> > Howard
> >
> > On Tue May 26 18:11:29 2020, partha.bhattacharjee at noaa.gov wrote:
> > > Thanks Howard for looking into it. I sent your response to the
> > > satellite
> > > team who are generating this data and I will test again (and
reach MET
> > > help) with their updated data.
> > >
> > > On Tue, May 26, 2020 at 7:46 PM Howard Soh via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > > There is no matching pairs because the time information is
missing
> > > > and the
> > > > default time ("19700101_000000") was assigned to the AOD data.
> > > >
> > > > Option 1:
> > > > Add "init_time" and "valid_time" variable attributes to each
data
> > > > variable
> > > > (VIIRS-L3-AOD_GEFS_20200519_1200.nc).
> > > > For example,
> > > > AOD_HM_Quality:init_time = "20200519_120000" ;
> > > > AOD_HM_Quality:valid_time = "20200519_120000" ;
> > > >
> > > > Option 2: Make CF-complaint NetCDF with time variable and
projection
> > > > variable (standard name and units)
> > > >
> > > >
> > > > [MET version dependency]
> > > > The point2grid was added recently and the format of gridded
point
> > > > observation NetCDF file was not changed. There is no MET
version
> > > > dependent
> > > > content nor format.
> > > >
> > > > Cheers,
> > > > Howard
> > > >
> > > > FYI:
> > > >
> > > > I don't have access to WCOSS. Please upload the modified file
to the
> > > > FTP
> > > > server for more supports.
> > > >
> > > > [How to send files for MET-help]
> > > >
> > > > https://dtcenter.org/met/users/support/met_help.php
> > > > Scroll down to "How To Send Us Data" section.
> > > >
> > > >
> > > > On Tue May 26 08:29:35 2020, partha.bhattacharjee at noaa.gov
wrote:
> > > > > Hi,
> > > > > I am trying to evaluate model forecast Aerosol optical depth
> > > > > against
> > > > > gridded Satellite observation. Observation data is average
of AOD
> > > > > between
> > > > > 11z and 13z of a particular day (19th May) and contains
missing
> > > > > values.
> > > > > Verifying model forecast is 12z of that day. I am able to
read the
> > > > > observation data using plot_data_plane but when I am using
> > > > > series_analysis
> > > > > to check stats it is showing 0 matched pairs. So, the
questions to
> > > > > you
> > > > > :
> > > > >
> > > > > 1. How to modify config file in this case to read model vs
obs
> > > > > which
> > > > > contains missing data. Also increase matched pairs.
> > > > >
> > > > > 2. Also,observation netcdf file is generated by partnering
> > > > > satellite branch to help evaluate our model result. Do you
have any
> > > > > suggestions on how to make the gridded netcdf file that is
not a
> > > > > MET-version dependent and what attributes are needed to make
it
> > > > > compatible with future MET release ?
> > > > >
> > > > > Files are in WCOSS- Mars at :
> > > > > Obs file
> > > > > :
> > > > >
> > > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/newfiles
> > > > > Forecast file:
> > > > >
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/
> > > > > VIIRS_L3Gridded/geaer.t00z.pgrb2a.0p25.f012
> > > > >
> > > > > Sample config :
> > > > >
> > > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_stat_testVIIRS_FV3.sh
> > > > >
> > > > > Thanks
> > > > > Partha
> > > >
> > > >
> > > >
> > > >
> >
> >
> >
> >
>
> --
> Partha S. Bhattacharjee, Ph.D.
> I.M. Systems Group
> Environmental Modeling Centre
> NCWCP
> 5830 University Research Court.
> College Park, MD 20740
>
>
------------------------------------------------
Subject: model vs satellite AOD: Missing values
From: Partha Bhattacharjee - NOAA Affiliate
Time: Thu May 28 12:33:19 2020
Thanks John. What is the difference between "init_time_ut" and
"valid_time_ut"? If we use the nominal time (for example 2020-05-19
12:00:00UTC for the this example data) to calculate the unix time
should it
be ok ?
On Thu, May 28, 2020 at 1:16 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Hi Partha (and Howard),
>
> I was out of the office this week and am just taking a look at this
issue
> now. I ran series_analysis using the data Julie P pulled down from
WCOSS,
> like this:
>
>
>
>
>
>
> */usr/local/met-8.0/bin/series_analysis \-fcst
geaer.t00z.pgrb2a.0p25.f012
> \-obs VIIRS-L3-AOD_GEFS_20200519_1200.nc \-out
> output_METseries_202005test/series_fh12_HQ202005.nc \-config
> SeriesAnlConfig_May21_fh12 \-paired*
>
> And I see the behavior that Partha describes:
> *DEBUG 2: Skipped 1036800 of 1036800 points with no valid data.*
>
> And as Howard notes, the VIIRS NetCDF file lacks the "valid_time_ut"
NetCDF
> variable attribute. So I ran the ncatted command Howard sent:
>
> *ncatted -a valid_time_ut,AOD_HM_Quality,a,l,1589889600
> VIIRS-L3-AOD_GEFS_20200519_1200.nc <http://1200.nc/>*
>
> And then when I rerun the Series-Analysis command listed above, it
runs
> fine. I ran plot_data_plane to plot the output variable named
> "series_cnt_ME" and have attached a PNG of the resulting image.
>
> *plot_data_plane output_METseries_202005test/series_fh12_HQ202005.nc
> series_cnt_ME.ps 'name="series_cnt_ME"; level="(*,*)";'*
>
> Partha, the "ut" at the "valid_time_ut" variable name stands for
"unixtime"
> which is the number of seconds since 1970-01-01:
> https://en.wikipedia.org/wiki/Unix_time
>
> When writing shell scripts, I often use the "date" command to
convert
> back/forth between unixtime and human-readable dates.
> For example, to convert from 20200519_120000 to unixtime, run:
>
>
> *> date -ud ''2020-05-19' UTC '12:00:00'' +%s*
> *1589889600*
>
> And to convert that unixtime back to YMDH format, run:
>
>
> *> date -ud '1970-01-01 UTC '1589889600' seconds'
+%Y%m%d%H2020051912*
>
> Thanks,
> John
>
> On Thu, May 28, 2020 at 10:14 AM Partha Bhattacharjee - NOAA
Affiliate via
> RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95358 >
> >
> > Howard,
> > Thanks. valid_time_ut is MET specific function. What is the value
for
> > “valid_time_ut”, i.e., what is the definition? Is there a
> straightforward
> > definition of this "valid_time_ut", for example, seconds from
1970-01-01
> > 00:00:00UTC?
> >
> > Thanks
> > Partha
> >
> >
> > On Wed, May 27, 2020 at 2:06 PM Howard Soh via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > The required variable attribute is "valid_time_ut". So the
minimum
> > > requirement is "valid_time_ut" attribute.
> > >
> > > Please add "valid_time_ut". After adding "valid_time_ut"
attribute
> > > (command below), 74615 matched pairs were found.
> > >
> > > DEBUG 2: Finished processing statistics for 74615 of 1036800
grid
> points.
> > > DEBUG 2: Skipped 962185 of 1036800 points with no valid data.
> > > DEBUG 2: Skipped 0 of 1036800 points that did not meet the valid
data
> > > threshold.
> > >
> > > Command to add variable attributes:
> > >
> > > ncatted -a init_time,AOD_HM_Quality,a,c,"20200519_120000"
> > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > ncatted -a valid_time,AOD_HM_Quality,a,c,"20200519_120000"
> > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > ncatted -a init_time_ut,AOD_HM_Quality,a,l,1589889600
> > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > ncatted -a valid_time_ut,AOD_HM_Quality,a,l,1589889600
> > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > >
> > >
> > > Cheers,
> > > Howard
> > >
> > > On Tue May 26 18:11:29 2020, partha.bhattacharjee at noaa.gov
wrote:
> > > > Thanks Howard for looking into it. I sent your response to the
> > > > satellite
> > > > team who are generating this data and I will test again (and
reach
> MET
> > > > help) with their updated data.
> > > >
> > > > On Tue, May 26, 2020 at 7:46 PM Howard Soh via RT
<met_help at ucar.edu
> >
> > > > wrote:
> > > >
> > > > > There is no matching pairs because the time information is
missing
> > > > > and the
> > > > > default time ("19700101_000000") was assigned to the AOD
data.
> > > > >
> > > > > Option 1:
> > > > > Add "init_time" and "valid_time" variable attributes to each
data
> > > > > variable
> > > > > (VIIRS-L3-AOD_GEFS_20200519_1200.nc).
> > > > > For example,
> > > > > AOD_HM_Quality:init_time = "20200519_120000" ;
> > > > > AOD_HM_Quality:valid_time = "20200519_120000" ;
> > > > >
> > > > > Option 2: Make CF-complaint NetCDF with time variable and
> projection
> > > > > variable (standard name and units)
> > > > >
> > > > >
> > > > > [MET version dependency]
> > > > > The point2grid was added recently and the format of gridded
point
> > > > > observation NetCDF file was not changed. There is no MET
version
> > > > > dependent
> > > > > content nor format.
> > > > >
> > > > > Cheers,
> > > > > Howard
> > > > >
> > > > > FYI:
> > > > >
> > > > > I don't have access to WCOSS. Please upload the modified
file to
> the
> > > > > FTP
> > > > > server for more supports.
> > > > >
> > > > > [How to send files for MET-help]
> > > > >
> > > > > https://dtcenter.org/met/users/support/met_help.php
> > > > > Scroll down to "How To Send Us Data" section.
> > > > >
> > > > >
> > > > > On Tue May 26 08:29:35 2020, partha.bhattacharjee at noaa.gov
wrote:
> > > > > > Hi,
> > > > > > I am trying to evaluate model forecast Aerosol optical
depth
> > > > > > against
> > > > > > gridded Satellite observation. Observation data is average
of AOD
> > > > > > between
> > > > > > 11z and 13z of a particular day (19th May) and contains
missing
> > > > > > values.
> > > > > > Verifying model forecast is 12z of that day. I am able to
read
> the
> > > > > > observation data using plot_data_plane but when I am using
> > > > > > series_analysis
> > > > > > to check stats it is showing 0 matched pairs. So, the
questions
> to
> > > > > > you
> > > > > > :
> > > > > >
> > > > > > 1. How to modify config file in this case to read model vs
obs
> > > > > > which
> > > > > > contains missing data. Also increase matched pairs.
> > > > > >
> > > > > > 2. Also,observation netcdf file is generated by partnering
> > > > > > satellite branch to help evaluate our model result. Do you
have
> any
> > > > > > suggestions on how to make the gridded netcdf file that is
not a
> > > > > > MET-version dependent and what attributes are needed to
make it
> > > > > > compatible with future MET release ?
> > > > > >
> > > > > > Files are in WCOSS- Mars at :
> > > > > > Obs file
> > > > > > :
> > > > > >
> > > > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/newfiles
> > > > > > Forecast file:
> > > > > >
> /gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/
> > > > > > VIIRS_L3Gridded/geaer.t00z.pgrb2a.0p25.f012
> > > > > >
> > > > > > Sample config :
> > > > > >
> > > > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_stat_testVIIRS_FV3.sh
> > > > > >
> > > > > > Thanks
> > > > > > Partha
> > > > >
> > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
> >
> > --
> > Partha S. Bhattacharjee, Ph.D.
> > I.M. Systems Group
> > Environmental Modeling Centre
> > NCWCP
> > 5830 University Research Court.
> > College Park, MD 20740
> >
> >
>
>
--
Partha S. Bhattacharjee, Ph.D.
I.M. Systems Group
Environmental Modeling Centre
NCWCP
5830 University Research Court.
College Park, MD 20740
------------------------------------------------
Subject: model vs satellite AOD: Missing values
From: John Halley Gotway
Time: Thu May 28 12:35:34 2020
The "init_time" is the model initialization time. Since this does not
apply
to your satellite observation data, it does not need to be included in
the
NetCDF variable attributes.
John
On Thu, May 28, 2020 at 12:33 PM Partha Bhattacharjee - NOAA Affiliate
via
RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95358 >
>
> Thanks John. What is the difference between "init_time_ut" and
> "valid_time_ut"? If we use the nominal time (for example 2020-05-19
> 12:00:00UTC for the this example data) to calculate the unix time
should it
> be ok ?
>
> On Thu, May 28, 2020 at 1:16 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Hi Partha (and Howard),
> >
> > I was out of the office this week and am just taking a look at
this issue
> > now. I ran series_analysis using the data Julie P pulled down from
WCOSS,
> > like this:
> >
> >
> >
> >
> >
> >
> > */usr/local/met-8.0/bin/series_analysis \-fcst
> geaer.t00z.pgrb2a.0p25.f012
> > \-obs VIIRS-L3-AOD_GEFS_20200519_1200.nc \-out
> > output_METseries_202005test/series_fh12_HQ202005.nc \-config
> > SeriesAnlConfig_May21_fh12 \-paired*
> >
> > And I see the behavior that Partha describes:
> > *DEBUG 2: Skipped 1036800 of 1036800 points with no valid data.*
> >
> > And as Howard notes, the VIIRS NetCDF file lacks the
"valid_time_ut"
> NetCDF
> > variable attribute. So I ran the ncatted command Howard sent:
> >
> > *ncatted -a valid_time_ut,AOD_HM_Quality,a,l,1589889600
> > VIIRS-L3-AOD_GEFS_20200519_1200.nc <http://1200.nc/>*
> >
> > And then when I rerun the Series-Analysis command listed above, it
runs
> > fine. I ran plot_data_plane to plot the output variable named
> > "series_cnt_ME" and have attached a PNG of the resulting image.
> >
> > *plot_data_plane
output_METseries_202005test/series_fh12_HQ202005.nc
> > series_cnt_ME.ps 'name="series_cnt_ME"; level="(*,*)";'*
> >
> > Partha, the "ut" at the "valid_time_ut" variable name stands for
> "unixtime"
> > which is the number of seconds since 1970-01-01:
> > https://en.wikipedia.org/wiki/Unix_time
> >
> > When writing shell scripts, I often use the "date" command to
convert
> > back/forth between unixtime and human-readable dates.
> > For example, to convert from 20200519_120000 to unixtime, run:
> >
> >
> > *> date -ud ''2020-05-19' UTC '12:00:00'' +%s*
> > *1589889600*
> >
> > And to convert that unixtime back to YMDH format, run:
> >
> >
> > *> date -ud '1970-01-01 UTC '1589889600' seconds'
+%Y%m%d%H2020051912*
> >
> > Thanks,
> > John
> >
> > On Thu, May 28, 2020 at 10:14 AM Partha Bhattacharjee - NOAA
Affiliate
> via
> > RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95358 >
> > >
> > > Howard,
> > > Thanks. valid_time_ut is MET specific function. What is the
value for
> > > “valid_time_ut”, i.e., what is the definition? Is there a
> > straightforward
> > > definition of this "valid_time_ut", for example, seconds from
> 1970-01-01
> > > 00:00:00UTC?
> > >
> > > Thanks
> > > Partha
> > >
> > >
> > > On Wed, May 27, 2020 at 2:06 PM Howard Soh via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > > The required variable attribute is "valid_time_ut". So the
minimum
> > > > requirement is "valid_time_ut" attribute.
> > > >
> > > > Please add "valid_time_ut". After adding "valid_time_ut"
attribute
> > > > (command below), 74615 matched pairs were found.
> > > >
> > > > DEBUG 2: Finished processing statistics for 74615 of 1036800
grid
> > points.
> > > > DEBUG 2: Skipped 962185 of 1036800 points with no valid data.
> > > > DEBUG 2: Skipped 0 of 1036800 points that did not meet the
valid data
> > > > threshold.
> > > >
> > > > Command to add variable attributes:
> > > >
> > > > ncatted -a init_time,AOD_HM_Quality,a,c,"20200519_120000"
> > > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > > ncatted -a valid_time,AOD_HM_Quality,a,c,"20200519_120000"
> > > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > > ncatted -a init_time_ut,AOD_HM_Quality,a,l,1589889600
> > > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > > ncatted -a valid_time_ut,AOD_HM_Quality,a,l,1589889600
> > > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > >
> > > >
> > > > Cheers,
> > > > Howard
> > > >
> > > > On Tue May 26 18:11:29 2020, partha.bhattacharjee at noaa.gov
wrote:
> > > > > Thanks Howard for looking into it. I sent your response to
the
> > > > > satellite
> > > > > team who are generating this data and I will test again (and
reach
> > MET
> > > > > help) with their updated data.
> > > > >
> > > > > On Tue, May 26, 2020 at 7:46 PM Howard Soh via RT <
> met_help at ucar.edu
> > >
> > > > > wrote:
> > > > >
> > > > > > There is no matching pairs because the time information is
> missing
> > > > > > and the
> > > > > > default time ("19700101_000000") was assigned to the AOD
data.
> > > > > >
> > > > > > Option 1:
> > > > > > Add "init_time" and "valid_time" variable attributes to
each data
> > > > > > variable
> > > > > > (VIIRS-L3-AOD_GEFS_20200519_1200.nc).
> > > > > > For example,
> > > > > > AOD_HM_Quality:init_time = "20200519_120000" ;
> > > > > > AOD_HM_Quality:valid_time = "20200519_120000" ;
> > > > > >
> > > > > > Option 2: Make CF-complaint NetCDF with time variable and
> > projection
> > > > > > variable (standard name and units)
> > > > > >
> > > > > >
> > > > > > [MET version dependency]
> > > > > > The point2grid was added recently and the format of
gridded point
> > > > > > observation NetCDF file was not changed. There is no MET
version
> > > > > > dependent
> > > > > > content nor format.
> > > > > >
> > > > > > Cheers,
> > > > > > Howard
> > > > > >
> > > > > > FYI:
> > > > > >
> > > > > > I don't have access to WCOSS. Please upload the modified
file to
> > the
> > > > > > FTP
> > > > > > server for more supports.
> > > > > >
> > > > > > [How to send files for MET-help]
> > > > > >
> > > > > > https://dtcenter.org/met/users/support/met_help.php
> > > > > > Scroll down to "How To Send Us Data" section.
> > > > > >
> > > > > >
> > > > > > On Tue May 26 08:29:35 2020, partha.bhattacharjee at noaa.gov
> wrote:
> > > > > > > Hi,
> > > > > > > I am trying to evaluate model forecast Aerosol optical
depth
> > > > > > > against
> > > > > > > gridded Satellite observation. Observation data is
average of
> AOD
> > > > > > > between
> > > > > > > 11z and 13z of a particular day (19th May) and contains
missing
> > > > > > > values.
> > > > > > > Verifying model forecast is 12z of that day. I am able
to read
> > the
> > > > > > > observation data using plot_data_plane but when I am
using
> > > > > > > series_analysis
> > > > > > > to check stats it is showing 0 matched pairs. So, the
questions
> > to
> > > > > > > you
> > > > > > > :
> > > > > > >
> > > > > > > 1. How to modify config file in this case to read model
vs obs
> > > > > > > which
> > > > > > > contains missing data. Also increase matched pairs.
> > > > > > >
> > > > > > > 2. Also,observation netcdf file is generated by
partnering
> > > > > > > satellite branch to help evaluate our model result. Do
you have
> > any
> > > > > > > suggestions on how to make the gridded netcdf file that
is not
> a
> > > > > > > MET-version dependent and what attributes are needed to
make it
> > > > > > > compatible with future MET release ?
> > > > > > >
> > > > > > > Files are in WCOSS- Mars at :
> > > > > > > Obs file
> > > > > > > :
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/newfiles
> > > > > > > Forecast file:
> > > > > > >
> > /gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/
> > > > > > > VIIRS_L3Gridded/geaer.t00z.pgrb2a.0p25.f012
> > > > > > >
> > > > > > > Sample config :
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_stat_testVIIRS_FV3.sh
> > > > > > >
> > > > > > > Thanks
> > > > > > > Partha
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > > Partha S. Bhattacharjee, Ph.D.
> > > I.M. Systems Group
> > > Environmental Modeling Centre
> > > NCWCP
> > > 5830 University Research Court.
> > > College Park, MD 20740
> > >
> > >
> >
> >
>
> --
> Partha S. Bhattacharjee, Ph.D.
> I.M. Systems Group
> Environmental Modeling Centre
> NCWCP
> 5830 University Research Court.
> College Park, MD 20740
>
>
------------------------------------------------
Subject: model vs satellite AOD: Missing values
From: Partha Bhattacharjee - NOAA Affiliate
Time: Fri Jun 05 23:06:42 2020
Thanks John. After modification of the observation file which now
includes
"valid_time_ut" I am able to get the same number of matched pairs as
Howard
and you have found earlier.
Now testing 1 month of data at 3 hour intervals and I found no
matching
pairs in series_stat analysis (although I checked a few individual day
and
forecast hours where matching pairs do exist between obs and model
forecast).
You can check here
:
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_stat_202005VIIRS_FV3.sh
and in the same directory output file ("monthtest.txt") is generated
when
running that script. I do not see any error message in output file,
but it
does not show any matching pairs at the end. Any help on this would be
appreciated
Thanks
Partha
On Thu, May 28, 2020 at 2:35 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> The "init_time" is the model initialization time. Since this does
not apply
> to your satellite observation data, it does not need to be included
in the
> NetCDF variable attributes.
>
> John
>
> On Thu, May 28, 2020 at 12:33 PM Partha Bhattacharjee - NOAA
Affiliate via
> RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95358 >
> >
> > Thanks John. What is the difference between "init_time_ut" and
> > "valid_time_ut"? If we use the nominal time (for example 2020-05-
19
> > 12:00:00UTC for the this example data) to calculate the unix time
should
> it
> > be ok ?
> >
> > On Thu, May 28, 2020 at 1:16 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Hi Partha (and Howard),
> > >
> > > I was out of the office this week and am just taking a look at
this
> issue
> > > now. I ran series_analysis using the data Julie P pulled down
from
> WCOSS,
> > > like this:
> > >
> > >
> > >
> > >
> > >
> > >
> > > */usr/local/met-8.0/bin/series_analysis \-fcst
> > geaer.t00z.pgrb2a.0p25.f012
> > > \-obs VIIRS-L3-AOD_GEFS_20200519_1200.nc \-out
> > > output_METseries_202005test/series_fh12_HQ202005.nc \-config
> > > SeriesAnlConfig_May21_fh12 \-paired*
> > >
> > > And I see the behavior that Partha describes:
> > > *DEBUG 2: Skipped 1036800 of 1036800 points with no valid data.*
> > >
> > > And as Howard notes, the VIIRS NetCDF file lacks the
"valid_time_ut"
> > NetCDF
> > > variable attribute. So I ran the ncatted command Howard sent:
> > >
> > > *ncatted -a valid_time_ut,AOD_HM_Quality,a,l,1589889600
> > > VIIRS-L3-AOD_GEFS_20200519_1200.nc <http://1200.nc/>*
> > >
> > > And then when I rerun the Series-Analysis command listed above,
it runs
> > > fine. I ran plot_data_plane to plot the output variable named
> > > "series_cnt_ME" and have attached a PNG of the resulting image.
> > >
> > > *plot_data_plane
output_METseries_202005test/series_fh12_HQ202005.nc
> > > series_cnt_ME.ps 'name="series_cnt_ME"; level="(*,*)";'*
> > >
> > > Partha, the "ut" at the "valid_time_ut" variable name stands for
> > "unixtime"
> > > which is the number of seconds since 1970-01-01:
> > > https://en.wikipedia.org/wiki/Unix_time
> > >
> > > When writing shell scripts, I often use the "date" command to
convert
> > > back/forth between unixtime and human-readable dates.
> > > For example, to convert from 20200519_120000 to unixtime, run:
> > >
> > >
> > > *> date -ud ''2020-05-19' UTC '12:00:00'' +%s*
> > > *1589889600*
> > >
> > > And to convert that unixtime back to YMDH format, run:
> > >
> > >
> > > *> date -ud '1970-01-01 UTC '1589889600' seconds'
+%Y%m%d%H2020051912*
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, May 28, 2020 at 10:14 AM Partha Bhattacharjee - NOAA
Affiliate
> > via
> > > RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95358
>
> > > >
> > > > Howard,
> > > > Thanks. valid_time_ut is MET specific function. What is the
value for
> > > > “valid_time_ut”, i.e., what is the definition? Is there a
> > > straightforward
> > > > definition of this "valid_time_ut", for example, seconds from
> > 1970-01-01
> > > > 00:00:00UTC?
> > > >
> > > > Thanks
> > > > Partha
> > > >
> > > >
> > > > On Wed, May 27, 2020 at 2:06 PM Howard Soh via RT
<met_help at ucar.edu
> >
> > > > wrote:
> > > >
> > > > > The required variable attribute is "valid_time_ut". So the
minimum
> > > > > requirement is "valid_time_ut" attribute.
> > > > >
> > > > > Please add "valid_time_ut". After adding "valid_time_ut"
attribute
> > > > > (command below), 74615 matched pairs were found.
> > > > >
> > > > > DEBUG 2: Finished processing statistics for 74615 of 1036800
grid
> > > points.
> > > > > DEBUG 2: Skipped 962185 of 1036800 points with no valid
data.
> > > > > DEBUG 2: Skipped 0 of 1036800 points that did not meet the
valid
> data
> > > > > threshold.
> > > > >
> > > > > Command to add variable attributes:
> > > > >
> > > > > ncatted -a init_time,AOD_HM_Quality,a,c,"20200519_120000"
> > > > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > > > ncatted -a
valid_time,AOD_HM_Quality,a,c,"20200519_120000"
> > > > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > > > ncatted -a init_time_ut,AOD_HM_Quality,a,l,1589889600
> > > > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > > > ncatted -a valid_time_ut,AOD_HM_Quality,a,l,1589889600
> > > > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > > >
> > > > >
> > > > > Cheers,
> > > > > Howard
> > > > >
> > > > > On Tue May 26 18:11:29 2020, partha.bhattacharjee at noaa.gov
wrote:
> > > > > > Thanks Howard for looking into it. I sent your response to
the
> > > > > > satellite
> > > > > > team who are generating this data and I will test again
(and
> reach
> > > MET
> > > > > > help) with their updated data.
> > > > > >
> > > > > > On Tue, May 26, 2020 at 7:46 PM Howard Soh via RT <
> > met_help at ucar.edu
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > There is no matching pairs because the time information
is
> > missing
> > > > > > > and the
> > > > > > > default time ("19700101_000000") was assigned to the AOD
data.
> > > > > > >
> > > > > > > Option 1:
> > > > > > > Add "init_time" and "valid_time" variable attributes to
each
> data
> > > > > > > variable
> > > > > > > (VIIRS-L3-AOD_GEFS_20200519_1200.nc).
> > > > > > > For example,
> > > > > > > AOD_HM_Quality:init_time = "20200519_120000" ;
> > > > > > > AOD_HM_Quality:valid_time = "20200519_120000" ;
> > > > > > >
> > > > > > > Option 2: Make CF-complaint NetCDF with time variable
and
> > > projection
> > > > > > > variable (standard name and units)
> > > > > > >
> > > > > > >
> > > > > > > [MET version dependency]
> > > > > > > The point2grid was added recently and the format of
gridded
> point
> > > > > > > observation NetCDF file was not changed. There is no MET
> version
> > > > > > > dependent
> > > > > > > content nor format.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Howard
> > > > > > >
> > > > > > > FYI:
> > > > > > >
> > > > > > > I don't have access to WCOSS. Please upload the modified
file
> to
> > > the
> > > > > > > FTP
> > > > > > > server for more supports.
> > > > > > >
> > > > > > > [How to send files for MET-help]
> > > > > > >
> > > > > > > https://dtcenter.org/met/users/support/met_help.php
> > > > > > > Scroll down to "How To Send Us Data" section.
> > > > > > >
> > > > > > >
> > > > > > > On Tue May 26 08:29:35 2020,
partha.bhattacharjee at noaa.gov
> > wrote:
> > > > > > > > Hi,
> > > > > > > > I am trying to evaluate model forecast Aerosol optical
depth
> > > > > > > > against
> > > > > > > > gridded Satellite observation. Observation data is
average of
> > AOD
> > > > > > > > between
> > > > > > > > 11z and 13z of a particular day (19th May) and
contains
> missing
> > > > > > > > values.
> > > > > > > > Verifying model forecast is 12z of that day. I am able
to
> read
> > > the
> > > > > > > > observation data using plot_data_plane but when I am
using
> > > > > > > > series_analysis
> > > > > > > > to check stats it is showing 0 matched pairs. So, the
> questions
> > > to
> > > > > > > > you
> > > > > > > > :
> > > > > > > >
> > > > > > > > 1. How to modify config file in this case to read
model vs
> obs
> > > > > > > > which
> > > > > > > > contains missing data. Also increase matched pairs.
> > > > > > > >
> > > > > > > > 2. Also,observation netcdf file is generated by
partnering
> > > > > > > > satellite branch to help evaluate our model result. Do
you
> have
> > > any
> > > > > > > > suggestions on how to make the gridded netcdf file
that is
> not
> > a
> > > > > > > > MET-version dependent and what attributes are needed
to make
> it
> > > > > > > > compatible with future MET release ?
> > > > > > > >
> > > > > > > > Files are in WCOSS- Mars at :
> > > > > > > > Obs file
> > > > > > > > :
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/newfiles
> > > > > > > > Forecast file:
> > > > > > > >
> > >
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/
> > > > > > > > VIIRS_L3Gridded/geaer.t00z.pgrb2a.0p25.f012
> > > > > > > >
> > > > > > > > Sample config :
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_stat_testVIIRS_FV3.sh
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Partha
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > Partha S. Bhattacharjee, Ph.D.
> > > > I.M. Systems Group
> > > > Environmental Modeling Centre
> > > > NCWCP
> > > > 5830 University Research Court.
> > > > College Park, MD 20740
> > > >
> > > >
> > >
> > >
> >
> > --
> > Partha S. Bhattacharjee, Ph.D.
> > I.M. Systems Group
> > Environmental Modeling Centre
> > NCWCP
> > 5830 University Research Court.
> > College Park, MD 20740
> >
> >
>
>
--
Partha S. Bhattacharjee, Ph.D.
I.M. Systems Group
Environmental Modeling Centre
NCWCP
5830 University Research Court.
College Park, MD 20740
------------------------------------------------
Subject: model vs satellite AOD: Missing values
From: John Halley Gotway
Time: Mon Jun 08 14:48:00 2020
Partha,
OK, I logged on to wcoss and see...
Your run script: /gpfs/dell2/emc/verification/n
oscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_
stat_202005VIIRS_FV3.sh
Your output file:
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/output_METseries_202005/series_fh021_HMQTAOD_20200519.nc
I used ncdump to look at the output file, but it doesn't contain any
statistical output:
*cd
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded*
*module load ips/18.0.5.274 NetCDF/4.5.0*
*ncdump -h output_METseries_202005/series_fh021_HMQTAOD_20200519.nc*
There should be output variables for each statistic you requested in
the
config file. So let's look at your config file:
*vi SeriesAnlConfig_May21HMQ_fh021*
To be clear, you do NOT need to specify 240 repeated levels in the
config
file. Instead, you just define a single field (name/level) for the
forecast and observation. When you pass it 240 input files, it'll read
that
same field from each of the input files. So that'll simplify your
config
file.
Please see the attached version of your config file that I edited. I
listed
only a single field for the forecast and obs data. And I also set the
"vld_thresh" to a value of 0.01. Since you had that commented out,
Series-Analysis was using the default value of 1.0... meaning that
output
stats were only computed at grid points where there were valid data
values
for ALL input times. With your data, that's never the case. So
hopefully,
using "vld_thresh = 0.01" will result in some output statistics.
Please let me know how it goes.
Thanks,
John
On Fri, Jun 5, 2020 at 11:06 PM Partha Bhattacharjee - NOAA Affiliate
via
RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95358 >
>
> Thanks John. After modification of the observation file which now
includes
> "valid_time_ut" I am able to get the same number of matched pairs as
Howard
> and you have found earlier.
> Now testing 1 month of data at 3 hour intervals and I found no
matching
> pairs in series_stat analysis (although I checked a few individual
day and
> forecast hours where matching pairs do exist between obs and model
> forecast).
> You can check here
> :
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_stat_202005VIIRS_FV3.sh
> and in the same directory output file ("monthtest.txt") is generated
when
> running that script. I do not see any error message in output file,
but it
> does not show any matching pairs at the end. Any help on this would
be
> appreciated
>
> Thanks
> Partha
>
>
>
>
>
>
> On Thu, May 28, 2020 at 2:35 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > The "init_time" is the model initialization time. Since this does
not
> apply
> > to your satellite observation data, it does not need to be
included in
> the
> > NetCDF variable attributes.
> >
> > John
> >
> > On Thu, May 28, 2020 at 12:33 PM Partha Bhattacharjee - NOAA
Affiliate
> via
> > RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95358 >
> > >
> > > Thanks John. What is the difference between "init_time_ut" and
> > > "valid_time_ut"? If we use the nominal time (for example 2020-
05-19
> > > 12:00:00UTC for the this example data) to calculate the unix
time
> should
> > it
> > > be ok ?
> > >
> > > On Thu, May 28, 2020 at 1:16 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Partha (and Howard),
> > > >
> > > > I was out of the office this week and am just taking a look at
this
> > issue
> > > > now. I ran series_analysis using the data Julie P pulled down
from
> > WCOSS,
> > > > like this:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > */usr/local/met-8.0/bin/series_analysis \-fcst
> > > geaer.t00z.pgrb2a.0p25.f012
> > > > \-obs VIIRS-L3-AOD_GEFS_20200519_1200.nc \-out
> > > > output_METseries_202005test/series_fh12_HQ202005.nc \-config
> > > > SeriesAnlConfig_May21_fh12 \-paired*
> > > >
> > > > And I see the behavior that Partha describes:
> > > > *DEBUG 2: Skipped 1036800 of 1036800 points with no valid
data.*
> > > >
> > > > And as Howard notes, the VIIRS NetCDF file lacks the
"valid_time_ut"
> > > NetCDF
> > > > variable attribute. So I ran the ncatted command Howard sent:
> > > >
> > > > *ncatted -a valid_time_ut,AOD_HM_Quality,a,l,1589889600
> > > > VIIRS-L3-AOD_GEFS_20200519_1200.nc <http://1200.nc/>*
> > > >
> > > > And then when I rerun the Series-Analysis command listed
above, it
> runs
> > > > fine. I ran plot_data_plane to plot the output variable named
> > > > "series_cnt_ME" and have attached a PNG of the resulting
image.
> > > >
> > > > *plot_data_plane
output_METseries_202005test/series_fh12_HQ202005.nc
> > > > series_cnt_ME.ps 'name="series_cnt_ME"; level="(*,*)";'*
> > > >
> > > > Partha, the "ut" at the "valid_time_ut" variable name stands
for
> > > "unixtime"
> > > > which is the number of seconds since 1970-01-01:
> > > > https://en.wikipedia.org/wiki/Unix_time
> > > >
> > > > When writing shell scripts, I often use the "date" command to
convert
> > > > back/forth between unixtime and human-readable dates.
> > > > For example, to convert from 20200519_120000 to unixtime, run:
> > > >
> > > >
> > > > *> date -ud ''2020-05-19' UTC '12:00:00'' +%s*
> > > > *1589889600*
> > > >
> > > > And to convert that unixtime back to YMDH format, run:
> > > >
> > > >
> > > > *> date -ud '1970-01-01 UTC '1589889600' seconds'
> +%Y%m%d%H2020051912*
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, May 28, 2020 at 10:14 AM Partha Bhattacharjee - NOAA
> Affiliate
> > > via
> > > > RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95358 >
> > > > >
> > > > > Howard,
> > > > > Thanks. valid_time_ut is MET specific function. What is the
value
> for
> > > > > “valid_time_ut”, i.e., what is the definition? Is there a
> > > > straightforward
> > > > > definition of this "valid_time_ut", for example, seconds
from
> > > 1970-01-01
> > > > > 00:00:00UTC?
> > > > >
> > > > > Thanks
> > > > > Partha
> > > > >
> > > > >
> > > > > On Wed, May 27, 2020 at 2:06 PM Howard Soh via RT <
> met_help at ucar.edu
> > >
> > > > > wrote:
> > > > >
> > > > > > The required variable attribute is "valid_time_ut". So the
> minimum
> > > > > > requirement is "valid_time_ut" attribute.
> > > > > >
> > > > > > Please add "valid_time_ut". After adding "valid_time_ut"
> attribute
> > > > > > (command below), 74615 matched pairs were found.
> > > > > >
> > > > > > DEBUG 2: Finished processing statistics for 74615 of
1036800 grid
> > > > points.
> > > > > > DEBUG 2: Skipped 962185 of 1036800 points with no valid
data.
> > > > > > DEBUG 2: Skipped 0 of 1036800 points that did not meet the
valid
> > data
> > > > > > threshold.
> > > > > >
> > > > > > Command to add variable attributes:
> > > > > >
> > > > > > ncatted -a
init_time,AOD_HM_Quality,a,c,"20200519_120000"
> > > > > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > > > > ncatted -a
valid_time,AOD_HM_Quality,a,c,"20200519_120000"
> > > > > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > > > > ncatted -a init_time_ut,AOD_HM_Quality,a,l,1589889600
> > > > > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > > > > ncatted -a valid_time_ut,AOD_HM_Quality,a,l,1589889600
> > > > > > VIIRS-L3-AOD_GEFS_20200519_1200.nc
> > > > > >
> > > > > >
> > > > > > Cheers,
> > > > > > Howard
> > > > > >
> > > > > > On Tue May 26 18:11:29 2020, partha.bhattacharjee at noaa.gov
> wrote:
> > > > > > > Thanks Howard for looking into it. I sent your response
to the
> > > > > > > satellite
> > > > > > > team who are generating this data and I will test again
(and
> > reach
> > > > MET
> > > > > > > help) with their updated data.
> > > > > > >
> > > > > > > On Tue, May 26, 2020 at 7:46 PM Howard Soh via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > There is no matching pairs because the time
information is
> > > missing
> > > > > > > > and the
> > > > > > > > default time ("19700101_000000") was assigned to the
AOD
> data.
> > > > > > > >
> > > > > > > > Option 1:
> > > > > > > > Add "init_time" and "valid_time" variable attributes
to each
> > data
> > > > > > > > variable
> > > > > > > > (VIIRS-L3-AOD_GEFS_20200519_1200.nc).
> > > > > > > > For example,
> > > > > > > > AOD_HM_Quality:init_time = "20200519_120000" ;
> > > > > > > > AOD_HM_Quality:valid_time = "20200519_120000" ;
> > > > > > > >
> > > > > > > > Option 2: Make CF-complaint NetCDF with time variable
and
> > > > projection
> > > > > > > > variable (standard name and units)
> > > > > > > >
> > > > > > > >
> > > > > > > > [MET version dependency]
> > > > > > > > The point2grid was added recently and the format of
gridded
> > point
> > > > > > > > observation NetCDF file was not changed. There is no
MET
> > version
> > > > > > > > dependent
> > > > > > > > content nor format.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Howard
> > > > > > > >
> > > > > > > > FYI:
> > > > > > > >
> > > > > > > > I don't have access to WCOSS. Please upload the
modified file
> > to
> > > > the
> > > > > > > > FTP
> > > > > > > > server for more supports.
> > > > > > > >
> > > > > > > > [How to send files for MET-help]
> > > > > > > >
> > > > > > > > https://dtcenter.org/met/users/support/met_help.php
> > > > > > > > Scroll down to "How To Send Us Data" section.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue May 26 08:29:35 2020,
partha.bhattacharjee at noaa.gov
> > > wrote:
> > > > > > > > > Hi,
> > > > > > > > > I am trying to evaluate model forecast Aerosol
optical
> depth
> > > > > > > > > against
> > > > > > > > > gridded Satellite observation. Observation data is
average
> of
> > > AOD
> > > > > > > > > between
> > > > > > > > > 11z and 13z of a particular day (19th May) and
contains
> > missing
> > > > > > > > > values.
> > > > > > > > > Verifying model forecast is 12z of that day. I am
able to
> > read
> > > > the
> > > > > > > > > observation data using plot_data_plane but when I am
using
> > > > > > > > > series_analysis
> > > > > > > > > to check stats it is showing 0 matched pairs. So,
the
> > questions
> > > > to
> > > > > > > > > you
> > > > > > > > > :
> > > > > > > > >
> > > > > > > > > 1. How to modify config file in this case to read
model vs
> > obs
> > > > > > > > > which
> > > > > > > > > contains missing data. Also increase matched pairs.
> > > > > > > > >
> > > > > > > > > 2. Also,observation netcdf file is generated by
partnering
> > > > > > > > > satellite branch to help evaluate our model result.
Do you
> > have
> > > > any
> > > > > > > > > suggestions on how to make the gridded netcdf file
that is
> > not
> > > a
> > > > > > > > > MET-version dependent and what attributes are needed
to
> make
> > it
> > > > > > > > > compatible with future MET release ?
> > > > > > > > >
> > > > > > > > > Files are in WCOSS- Mars at :
> > > > > > > > > Obs file
> > > > > > > > > :
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/newfiles
> > > > > > > > > Forecast file:
> > > > > > > > >
> > > >
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/
> > > > > > > > > VIIRS_L3Gridded/geaer.t00z.pgrb2a.0p25.f012
> > > > > > > > >
> > > > > > > > > Sample config :
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Partha.Bhattacharjee/Aerosol/VIIRS_L3Gridded/series_stat_testVIIRS_FV3.sh
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Partha
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Partha S. Bhattacharjee, Ph.D.
> > > > > I.M. Systems Group
> > > > > Environmental Modeling Centre
> > > > > NCWCP
> > > > > 5830 University Research Court.
> > > > > College Park, MD 20740
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > Partha S. Bhattacharjee, Ph.D.
> > > I.M. Systems Group
> > > Environmental Modeling Centre
> > > NCWCP
> > > 5830 University Research Court.
> > > College Park, MD 20740
> > >
> > >
> >
> >
>
> --
> Partha S. Bhattacharjee, Ph.D.
> I.M. Systems Group
> Environmental Modeling Centre
> NCWCP
> 5830 University Research Court.
> College Park, MD 20740
>
>
------------------------------------------------
More information about the Met_help
mailing list