[Met_help] [rt.rap.ucar.edu #95411] History for hang on regrid_data_plane
Howard Soh via RT
met_help at ucar.edu
Tue Jun 16 13:47:58 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
To Whom It May Concern:
I have an LatLon 0.03x0.03 satellite observed AOD at 20200519 20Z (global
but most with fill_value). I want to use regrid_data_plane to regrid it to
CMAQ Lambert Conformal Grid. But it seems to hang during the Interpolation
step.
Can you find out what went wrong? Sometimes it is more than an hour
waiting before I kill it.
runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-Chun.Huang/plot/viirs/
with command > bash run.viirs_aqm_aod.sh viirs aqm 20200519 20200519 >
regrid.log 2>&1 &
You can check the "regrid.log" in the script directory for input,
model_grid, and output file location on venus. I will keep it going
from May 29 19:15Z.
One thing I do not understand is that the output file is already generated
while the job is still
running. /gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-AOD_AQM_aqm_20200519_20.nc
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: hang on regrid_data_plane
From: Howard Soh
Time: Wed Jun 03 15:41:02 2020
I don't have access to venus.
Would you upload the files (config, script, log, and data files) to
MET ftp server?
Which MET version are you using?
The GOES-AOD data was supported by regrid_data_plane (V8.x). It's
moved to point2grid (V9.0). The regrid_data_plane still processes
GOES-AOD data in different way (interpolated based on the neighbor
data by using width and shape, not by the physical location). The
output is different from point2grid. VIIRS AOD is not supported by
MET, yet. There might be a workaround.
Cheers,
Howard
On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov wrote:
> To Whom It May Concern:
>
> I have an LatLon 0.03x0.03 satellite observed AOD at 20200519 20Z
> (global
> but most with fill_value). I want to use regrid_data_plane to
regrid
> it to
> CMAQ Lambert Conformal Grid. But it seems to hang during the
> Interpolation
> step.
>
> Can you find out what went wrong? Sometimes it is more than an hour
> waiting before I kill it.
>
> runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-
> Chun.Huang/plot/viirs/
>
> with command > bash run.viirs_aqm_aod.sh viirs aqm 20200519 20200519
>
> regrid.log 2>&1 &
>
> You can check the "regrid.log" in the script directory for input,
> model_grid, and output file location on venus. I will keep it going
> from May 29 19:15Z.
>
> One thing I do not understand is that the output file is already
> generated
> while the job is still
> running. /gpfs/dell2/emc/modeling/noscrub/Ho-
> Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
> AOD_AQM_aqm_20200519_20.nc
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
------------------------------------------------
Subject: hang on regrid_data_plane
From: John Halley Gotway
Time: Wed Jun 03 16:45:55 2020
Howard,
I'll go grab this data and copy it over to dakota. Ho-Chun, sorry for
the
delay in getting started with this. I'll go grab it now.
John
On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
>
> I don't have access to venus.
> Would you upload the files (config, script, log, and data files) to
MET
> ftp server?
>
> Which MET version are you using?
>
> The GOES-AOD data was supported by regrid_data_plane (V8.x). It's
moved to
> point2grid (V9.0). The regrid_data_plane still processes GOES-AOD
data in
> different way (interpolated based on the neighbor data by using
width and
> shape, not by the physical location). The output is different from
> point2grid. VIIRS AOD is not supported by MET, yet. There might be a
> workaround.
>
> Cheers,
> Howard
>
> On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov wrote:
> > To Whom It May Concern:
> >
> > I have an LatLon 0.03x0.03 satellite observed AOD at 20200519 20Z
> > (global
> > but most with fill_value). I want to use regrid_data_plane to
regrid
> > it to
> > CMAQ Lambert Conformal Grid. But it seems to hang during the
> > Interpolation
> > step.
> >
> > Can you find out what went wrong? Sometimes it is more than an
hour
> > waiting before I kill it.
> >
> > runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-
> > Chun.Huang/plot/viirs/
> >
> > with command > bash run.viirs_aqm_aod.sh viirs aqm 20200519
20200519 >
> > regrid.log 2>&1 &
> >
> > You can check the "regrid.log" in the script directory for input,
> > model_grid, and output file location on venus. I will keep it
going
> > from May 29 19:15Z.
> >
> > One thing I do not understand is that the output file is already
> > generated
> > while the job is still
> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
> > AOD_AQM_aqm_20200519_20.nc
> >
> > Ho-Chun Huang
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct., Rm. 2792
> >
> > College Park, MD 20740
> >
> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >
> > 301-683-3958
>
>
>
>
------------------------------------------------
Subject: hang on regrid_data_plane
From: John Halley Gotway
Time: Wed Jun 03 17:05:37 2020
Howard,
OK, please find input data for testing here:
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
When you untar this file, you'll find the contents of Ho-Chun's
directory:
venus:/gpfs/dell2/emc/modeling/save/Ho-Chun.Huang/plot/viirs
In addition, I copied the input/output directories from
/gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/VIIRS_AOD into that
viirs
directory. Also, I included the file named "aqm.aot.148.grid" which
defines
the target grid.
If you're able to take a look at this data and see how it runs, that'd
be
great! Ho-Chun found that it took over 12 hours!? to run on WCOSS.
Thanks,
John
On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Howard,
>
> I'll go grab this data and copy it over to dakota. Ho-Chun, sorry
for the
> delay in getting started with this. I'll go grab it now.
>
> John
>
> On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT <met_help at ucar.edu>
> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
>>
>> I don't have access to venus.
>> Would you upload the files (config, script, log, and data files) to
MET
>> ftp server?
>>
>> Which MET version are you using?
>>
>> The GOES-AOD data was supported by regrid_data_plane (V8.x). It's
moved
>> to point2grid (V9.0). The regrid_data_plane still processes GOES-
AOD data
>> in different way (interpolated based on the neighbor data by using
width
>> and shape, not by the physical location). The output is different
from
>> point2grid. VIIRS AOD is not supported by MET, yet. There might be
a
>> workaround.
>>
>> Cheers,
>> Howard
>>
>> On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov wrote:
>> > To Whom It May Concern:
>> >
>> > I have an LatLon 0.03x0.03 satellite observed AOD at 20200519 20Z
>> > (global
>> > but most with fill_value). I want to use regrid_data_plane to
regrid
>> > it to
>> > CMAQ Lambert Conformal Grid. But it seems to hang during the
>> > Interpolation
>> > step.
>> >
>> > Can you find out what went wrong? Sometimes it is more than an
hour
>> > waiting before I kill it.
>> >
>> > runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-
>> > Chun.Huang/plot/viirs/
>> >
>> > with command > bash run.viirs_aqm_aod.sh viirs aqm 20200519
20200519 >
>> > regrid.log 2>&1 &
>> >
>> > You can check the "regrid.log" in the script directory for input,
>> > model_grid, and output file location on venus. I will keep it
going
>> > from May 29 19:15Z.
>> >
>> > One thing I do not understand is that the output file is already
>> > generated
>> > while the job is still
>> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
>> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
>> > AOD_AQM_aqm_20200519_20.nc
>> >
>> > Ho-Chun Huang
>> >
>> > IMSG at NOAA/NWS/NCEP/EMC
>> >
>> > 5830 University Research Ct., Rm. 2792
>> >
>> > College Park, MD 20740
>> >
>> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>> >
>> > 301-683-3958
>>
>>
>>
>>
------------------------------------------------
Subject: hang on regrid_data_plane
From: Ho-Chun Huang - NOAA Affiliate
Time: Thu Jun 04 06:53:39 2020
Hi, John and Howard:
(1) You should be able to see my MET version in the run script : 9.0
(2) Background information, we used to discuss between NCAR MET,
NESDIS,
and EMC about using VIIRS pixel AOD data per granule and mapping pixel
observed AOD into a selected model grid. It has been determined that
the
data volume is too large for MET to handle VIIRS granule pixel data
ingestion. EMC has asked NESDIS aerosol groups to provide *L3
"gridded"
nefCDF* files for global and regional AQM model
evaluation/verification.
I am testing the regional one VIIRS-L3-AOD_*AQM*_20200519_200000.nc,
while Partha Bhattacharjee is testing VIIRS-L3-
AOD_*GEFS*_20200519_120000.nc
for global GEFS-aerosol verification [reference helpdesk ticket
95358].
(3) *I am not trying to using point2grid*, I am trying to use MET
regridding utilities to map a *gridded* netCDF to a target model
*gridded*
netCDF. Unless there is another better tool in MET packages that I
am not
aware of, I assume regrid_data_plane is for general mapping purpose.
(4) For each hourly period, orbital satellites overpassed only a small
area
around the globe. I have asked NESDIS aerosol to use fill-values for
non-observed grids and with compress level=1 hoping regard-data_plane
can
skip those "-9999.f" grid quickly. I may be wrong. In reality, I may
only
use files from 18Z to 23Z for verification on CONUS, Alaska, and
Hawaii.
The reason to keep it global coverage is to simplify NESDIS production
processes and be flexible for future applications. Please study my
problem,
if changes on the data production side [NESDIS aerosol group] can
speed-up
the regridding processes, please let me know.
Thank you for your help.
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Howard,
>
> OK, please find input data for testing here:
>
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
>
> When you untar this file, you'll find the contents of Ho-Chun's
directory:
> venus:/gpfs/dell2/emc/modeling/save/Ho-Chun.Huang/plot/viirs
>
> In addition, I copied the input/output directories from
> /gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/VIIRS_AOD into that
viirs
> directory. Also, I included the file named "aqm.aot.148.grid" which
defines
> the target grid.
>
> If you're able to take a look at this data and see how it runs,
that'd be
> great! Ho-Chun found that it took over 12 hours!? to run on WCOSS.
>
> Thanks,
> John
>
> On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Howard,
> >
> > I'll go grab this data and copy it over to dakota. Ho-Chun, sorry
for the
> > delay in getting started with this. I'll go grab it now.
> >
> > John
> >
> > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT
<met_help at ucar.edu>
> > wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
> >>
> >> I don't have access to venus.
> >> Would you upload the files (config, script, log, and data files)
to MET
> >> ftp server?
> >>
> >> Which MET version are you using?
> >>
> >> The GOES-AOD data was supported by regrid_data_plane (V8.x). It's
moved
> >> to point2grid (V9.0). The regrid_data_plane still processes GOES-
AOD
> data
> >> in different way (interpolated based on the neighbor data by
using width
> >> and shape, not by the physical location). The output is different
from
> >> point2grid. VIIRS AOD is not supported by MET, yet. There might
be a
> >> workaround.
> >>
> >> Cheers,
> >> Howard
> >>
> >> On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov wrote:
> >> > To Whom It May Concern:
> >> >
> >> > I have an LatLon 0.03x0.03 satellite observed AOD at 20200519
20Z
> >> > (global
> >> > but most with fill_value). I want to use regrid_data_plane to
regrid
> >> > it to
> >> > CMAQ Lambert Conformal Grid. But it seems to hang during the
> >> > Interpolation
> >> > step.
> >> >
> >> > Can you find out what went wrong? Sometimes it is more than an
hour
> >> > waiting before I kill it.
> >> >
> >> > runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-
> >> > Chun.Huang/plot/viirs/
> >> >
> >> > with command > bash run.viirs_aqm_aod.sh viirs aqm 20200519
20200519
> >
> >> > regrid.log 2>&1 &
> >> >
> >> > You can check the "regrid.log" in the script directory for
input,
> >> > model_grid, and output file location on venus. I will keep it
going
> >> > from May 29 19:15Z.
> >> >
> >> > One thing I do not understand is that the output file is
already
> >> > generated
> >> > while the job is still
> >> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
> >> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
> >> > AOD_AQM_aqm_20200519_20.nc
> >> >
> >> > Ho-Chun Huang
> >> >
> >> > IMSG at NOAA/NWS/NCEP/EMC
> >> >
> >> > 5830 University Research Ct., Rm. 2792
> >> >
> >> > College Park, MD 20740
> >> >
> >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >> >
> >> > 301-683-3958
> >>
> >>
> >>
> >>
>
>
------------------------------------------------
Subject: hang on regrid_data_plane
From: Howard Soh
Time: Thu Jun 04 08:48:11 2020
I will investigate why it takes too long.
I ran regrid_data_plane for just one variable (AOD_H_Quality,
dimension is 6000 by 12000) and it took about 4 hours at dakota.
DEBUG 1: Writing output file: VIIRS-L3-
AOD_AQM_aqm_20200519_20.regrided.nc
real 239m2.552s
user 206m42.332s
sys 32m18.028s
The VIIRS-L3-AOD_AQM_20200519_200000.nc does not have the time
variable "t". So point2grid does not handle this case with workaround.
Cheers,
Howard
On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov wrote:
> Hi, John and Howard:
>
> (1) You should be able to see my MET version in the run script : 9.0
>
> (2) Background information, we used to discuss between NCAR MET,
> NESDIS,
> and EMC about using VIIRS pixel AOD data per granule and mapping
pixel
> observed AOD into a selected model grid. It has been determined
that
> the
> data volume is too large for MET to handle VIIRS granule pixel data
> ingestion. EMC has asked NESDIS aerosol groups to provide *L3
> "gridded"
> nefCDF* files for global and regional AQM model
> evaluation/verification.
> I am testing the regional one VIIRS-L3-AOD_*AQM*_20200519_200000.nc,
> while Partha Bhattacharjee is testing VIIRS-L3-
> AOD_*GEFS*_20200519_120000.nc
> for global GEFS-aerosol verification [reference helpdesk ticket
> 95358].
>
> (3) *I am not trying to using point2grid*, I am trying to use MET
> regridding utilities to map a *gridded* netCDF to a target model
> *gridded*
> netCDF. Unless there is another better tool in MET packages that I
> am not
> aware of, I assume regrid_data_plane is for general mapping purpose.
>
> (4) For each hourly period, orbital satellites overpassed only a
small
> area
> around the globe. I have asked NESDIS aerosol to use fill-values for
> non-observed grids and with compress level=1 hoping regard-
data_plane
> can
> skip those "-9999.f" grid quickly. I may be wrong. In reality, I
may
> only
> use files from 18Z to 23Z for verification on CONUS, Alaska, and
> Hawaii.
> The reason to keep it global coverage is to simplify NESDIS
production
> processes and be flexible for future applications. Please study my
> problem,
> if changes on the data production side [NESDIS aerosol group] can
> speed-up
> the regridding processes, please let me know.
>
> Thank you for your help.
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway via RT
> <met_help at ucar.edu>
> wrote:
>
> > Howard,
> >
> > OK, please find input data for testing here:
> >
> >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> >
> > When you untar this file, you'll find the contents of Ho-Chun's
> > directory:
> > venus:/gpfs/dell2/emc/modeling/save/Ho-Chun.Huang/plot/viirs
> >
> > In addition, I copied the input/output directories from
> > /gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/VIIRS_AOD into that
> > viirs
> > directory. Also, I included the file named "aqm.aot.148.grid"
which
> > defines
> > the target grid.
> >
> > If you're able to take a look at this data and see how it runs,
> > that'd be
> > great! Ho-Chun found that it took over 12 hours!? to run on WCOSS.
> >
> > Thanks,
> > John
> >
> > On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Howard,
> > >
> > > I'll go grab this data and copy it over to dakota. Ho-Chun,
sorry
> > > for the
> > > delay in getting started with this. I'll go grab it now.
> > >
> > > John
> > >
> > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
> > >>
> > >> I don't have access to venus.
> > >> Would you upload the files (config, script, log, and data
files)
> > >> to MET
> > >> ftp server?
> > >>
> > >> Which MET version are you using?
> > >>
> > >> The GOES-AOD data was supported by regrid_data_plane (V8.x).
It's
> > >> moved
> > >> to point2grid (V9.0). The regrid_data_plane still processes
GOES-
> > >> AOD
> > data
> > >> in different way (interpolated based on the neighbor data by
using
> > >> width
> > >> and shape, not by the physical location). The output is
different
> > >> from
> > >> point2grid. VIIRS AOD is not supported by MET, yet. There might
be
> > >> a
> > >> workaround.
> > >>
> > >> Cheers,
> > >> Howard
> > >>
> > >> On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov wrote:
> > >> > To Whom It May Concern:
> > >> >
> > >> > I have an LatLon 0.03x0.03 satellite observed AOD at 20200519
> > >> > 20Z
> > >> > (global
> > >> > but most with fill_value). I want to use regrid_data_plane
to
> > >> > regrid
> > >> > it to
> > >> > CMAQ Lambert Conformal Grid. But it seems to hang during the
> > >> > Interpolation
> > >> > step.
> > >> >
> > >> > Can you find out what went wrong? Sometimes it is more than
an
> > >> > hour
> > >> > waiting before I kill it.
> > >> >
> > >> > runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-
> > >> > Chun.Huang/plot/viirs/
> > >> >
> > >> > with command > bash run.viirs_aqm_aod.sh viirs aqm 20200519
> > >> > 20200519
> > >
> > >> > regrid.log 2>&1 &
> > >> >
> > >> > You can check the "regrid.log" in the script directory for
> > >> > input,
> > >> > model_grid, and output file location on venus. I will keep
it
> > >> > going
> > >> > from May 29 19:15Z.
> > >> >
> > >> > One thing I do not understand is that the output file is
already
> > >> > generated
> > >> > while the job is still
> > >> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
> > >> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
> > >> > AOD_AQM_aqm_20200519_20.nc
> > >> >
> > >> > Ho-Chun Huang
> > >> >
> > >> > IMSG at NOAA/NWS/NCEP/EMC
> > >> >
> > >> > 5830 University Research Ct., Rm. 2792
> > >> >
> > >> > College Park, MD 20740
> > >> >
> > >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > >> >
> > >> > 301-683-3958
> > >>
> > >>
> > >>
> > >>
> >
> >
------------------------------------------------
Subject: hang on regrid_data_plane
From: Ho-Chun Huang - NOAA Affiliate
Time: Thu Jun 04 09:05:32 2020
Hi, Howard:
Please let me know if any global attributes or variables addition on
the
observed file will help to speedup the regridiing process. I will
make a
request to the NESDIS aerosol group.
Thank you for your effort.
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Thu, Jun 4, 2020 at 10:48 AM Howard Soh via RT <met_help at ucar.edu>
wrote:
> I will investigate why it takes too long.
>
> I ran regrid_data_plane for just one variable (AOD_H_Quality,
dimension is
> 6000 by 12000) and it took about 4 hours at dakota.
>
> DEBUG 1: Writing output file: VIIRS-L3-
AOD_AQM_aqm_20200519_20.regrided.nc
>
> real 239m2.552s
> user 206m42.332s
> sys 32m18.028s
>
> The VIIRS-L3-AOD_AQM_20200519_200000.nc does not have the time
variable
> "t". So point2grid does not handle this case with workaround.
>
> Cheers,
> Howard
>
> On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov wrote:
> > Hi, John and Howard:
> >
> > (1) You should be able to see my MET version in the run script :
9.0
> >
> > (2) Background information, we used to discuss between NCAR MET,
> > NESDIS,
> > and EMC about using VIIRS pixel AOD data per granule and mapping
pixel
> > observed AOD into a selected model grid. It has been determined
that
> > the
> > data volume is too large for MET to handle VIIRS granule pixel
data
> > ingestion. EMC has asked NESDIS aerosol groups to provide *L3
> > "gridded"
> > nefCDF* files for global and regional AQM model
> > evaluation/verification.
> > I am testing the regional one VIIRS-L3-
AOD_*AQM*_20200519_200000.nc,
> > while Partha Bhattacharjee is testing VIIRS-L3-
> > AOD_*GEFS*_20200519_120000.nc
> > for global GEFS-aerosol verification [reference helpdesk ticket
> > 95358].
> >
> > (3) *I am not trying to using point2grid*, I am trying to use MET
> > regridding utilities to map a *gridded* netCDF to a target model
> > *gridded*
> > netCDF. Unless there is another better tool in MET packages that
I
> > am not
> > aware of, I assume regrid_data_plane is for general mapping
purpose.
> >
> > (4) For each hourly period, orbital satellites overpassed only a
small
> > area
> > around the globe. I have asked NESDIS aerosol to use fill-values
for
> > non-observed grids and with compress level=1 hoping regard-
data_plane
> > can
> > skip those "-9999.f" grid quickly. I may be wrong. In reality, I
may
> > only
> > use files from 18Z to 23Z for verification on CONUS, Alaska, and
> > Hawaii.
> > The reason to keep it global coverage is to simplify NESDIS
production
> > processes and be flexible for future applications. Please study my
> > problem,
> > if changes on the data production side [NESDIS aerosol group] can
> > speed-up
> > the regridding processes, please let me know.
> >
> > Thank you for your help.
> >
> > Ho-Chun Huang
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct., Rm. 2792
> >
> > College Park, MD 20740
> >
> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >
> > 301-683-3958
> >
> >
> > On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Howard,
> > >
> > > OK, please find input data for testing here:
> > >
> > >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> > >
> > > When you untar this file, you'll find the contents of Ho-Chun's
> > > directory:
> > > venus:/gpfs/dell2/emc/modeling/save/Ho-Chun.Huang/plot/viirs
> > >
> > > In addition, I copied the input/output directories from
> > > /gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/VIIRS_AOD into
that
> > > viirs
> > > directory. Also, I included the file named "aqm.aot.148.grid"
which
> > > defines
> > > the target grid.
> > >
> > > If you're able to take a look at this data and see how it runs,
> > > that'd be
> > > great! Ho-Chun found that it took over 12 hours!? to run on
WCOSS.
> > >
> > > Thanks,
> > > John
> > >
> > > On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > Howard,
> > > >
> > > > I'll go grab this data and copy it over to dakota. Ho-Chun,
sorry
> > > > for the
> > > > delay in getting started with this. I'll go grab it now.
> > > >
> > > > John
> > > >
> > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
>
> > > >>
> > > >> I don't have access to venus.
> > > >> Would you upload the files (config, script, log, and data
files)
> > > >> to MET
> > > >> ftp server?
> > > >>
> > > >> Which MET version are you using?
> > > >>
> > > >> The GOES-AOD data was supported by regrid_data_plane (V8.x).
It's
> > > >> moved
> > > >> to point2grid (V9.0). The regrid_data_plane still processes
GOES-
> > > >> AOD
> > > data
> > > >> in different way (interpolated based on the neighbor data by
using
> > > >> width
> > > >> and shape, not by the physical location). The output is
different
> > > >> from
> > > >> point2grid. VIIRS AOD is not supported by MET, yet. There
might be
> > > >> a
> > > >> workaround.
> > > >>
> > > >> Cheers,
> > > >> Howard
> > > >>
> > > >> On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov wrote:
> > > >> > To Whom It May Concern:
> > > >> >
> > > >> > I have an LatLon 0.03x0.03 satellite observed AOD at
20200519
> > > >> > 20Z
> > > >> > (global
> > > >> > but most with fill_value). I want to use regrid_data_plane
to
> > > >> > regrid
> > > >> > it to
> > > >> > CMAQ Lambert Conformal Grid. But it seems to hang during
the
> > > >> > Interpolation
> > > >> > step.
> > > >> >
> > > >> > Can you find out what went wrong? Sometimes it is more
than an
> > > >> > hour
> > > >> > waiting before I kill it.
> > > >> >
> > > >> > runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > >> > Chun.Huang/plot/viirs/
> > > >> >
> > > >> > with command > bash run.viirs_aqm_aod.sh viirs aqm 20200519
> > > >> > 20200519
> > > >
> > > >> > regrid.log 2>&1 &
> > > >> >
> > > >> > You can check the "regrid.log" in the script directory for
> > > >> > input,
> > > >> > model_grid, and output file location on venus. I will keep
it
> > > >> > going
> > > >> > from May 29 19:15Z.
> > > >> >
> > > >> > One thing I do not understand is that the output file is
already
> > > >> > generated
> > > >> > while the job is still
> > > >> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > >> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
> > > >> > AOD_AQM_aqm_20200519_20.nc
> > > >> >
> > > >> > Ho-Chun Huang
> > > >> >
> > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > >> >
> > > >> > 5830 University Research Ct., Rm. 2792
> > > >> >
> > > >> > College Park, MD 20740
> > > >> >
> > > >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > >> >
> > > >> > 301-683-3958
> > > >>
> > > >>
> > > >>
> > > >>
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: hang on regrid_data_plane
From: John Halley Gotway
Time: Thu Jun 04 09:39:24 2020
Howard,
I recall that with the GOES files, you set up regrid_data_plane to
determine the grid mapping the first time it's run, and then reuse
that
mapping information in subsequent calls. I really don't know anything
about
the VIIRS data. Searching online, it looks like the VIIRS data comes
in
swaths, whereas the GOES data is stationary. So perhaps the approach
of
pre-computing the grid mappings won't work. I wonder if the swaths are
repeated though.
Ho-Chun, do you know if the geometry of the VIIRS data repeats day
after
day? If we can figure out a way of computing the grid mapping
information
once ahead of time and then reuse that information, I'm guessing the
tool
would run much faster.
Thanks,
John
On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
>
> I will investigate why it takes too long.
>
> I ran regrid_data_plane for just one variable (AOD_H_Quality,
dimension is
> 6000 by 12000) and it took about 4 hours at dakota.
>
> DEBUG 1: Writing output file: VIIRS-L3-
AOD_AQM_aqm_20200519_20.regrided.nc
>
> real 239m2.552s
> user 206m42.332s
> sys 32m18.028s
>
> The VIIRS-L3-AOD_AQM_20200519_200000.nc does not have the time
variable
> "t". So point2grid does not handle this case with workaround.
>
> Cheers,
> Howard
>
> On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov wrote:
> > Hi, John and Howard:
> >
> > (1) You should be able to see my MET version in the run script :
9.0
> >
> > (2) Background information, we used to discuss between NCAR MET,
> > NESDIS,
> > and EMC about using VIIRS pixel AOD data per granule and mapping
pixel
> > observed AOD into a selected model grid. It has been determined
that
> > the
> > data volume is too large for MET to handle VIIRS granule pixel
data
> > ingestion. EMC has asked NESDIS aerosol groups to provide *L3
> > "gridded"
> > nefCDF* files for global and regional AQM model
> > evaluation/verification.
> > I am testing the regional one VIIRS-L3-
AOD_*AQM*_20200519_200000.nc,
> > while Partha Bhattacharjee is testing VIIRS-L3-
> > AOD_*GEFS*_20200519_120000.nc
> > for global GEFS-aerosol verification [reference helpdesk ticket
> > 95358].
> >
> > (3) *I am not trying to using point2grid*, I am trying to use MET
> > regridding utilities to map a *gridded* netCDF to a target model
> > *gridded*
> > netCDF. Unless there is another better tool in MET packages that
I
> > am not
> > aware of, I assume regrid_data_plane is for general mapping
purpose.
> >
> > (4) For each hourly period, orbital satellites overpassed only a
small
> > area
> > around the globe. I have asked NESDIS aerosol to use fill-values
for
> > non-observed grids and with compress level=1 hoping regard-
data_plane
> > can
> > skip those "-9999.f" grid quickly. I may be wrong. In reality, I
may
> > only
> > use files from 18Z to 23Z for verification on CONUS, Alaska, and
> > Hawaii.
> > The reason to keep it global coverage is to simplify NESDIS
production
> > processes and be flexible for future applications. Please study my
> > problem,
> > if changes on the data production side [NESDIS aerosol group] can
> > speed-up
> > the regridding processes, please let me know.
> >
> > Thank you for your help.
> >
> > Ho-Chun Huang
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct., Rm. 2792
> >
> > College Park, MD 20740
> >
> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >
> > 301-683-3958
> >
> >
> > On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Howard,
> > >
> > > OK, please find input data for testing here:
> > >
> > >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> > >
> > > When you untar this file, you'll find the contents of Ho-Chun's
> > > directory:
> > > venus:/gpfs/dell2/emc/modeling/save/Ho-Chun.Huang/plot/viirs
> > >
> > > In addition, I copied the input/output directories from
> > > /gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/VIIRS_AOD into
that
> > > viirs
> > > directory. Also, I included the file named "aqm.aot.148.grid"
which
> > > defines
> > > the target grid.
> > >
> > > If you're able to take a look at this data and see how it runs,
> > > that'd be
> > > great! Ho-Chun found that it took over 12 hours!? to run on
WCOSS.
> > >
> > > Thanks,
> > > John
> > >
> > > On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > Howard,
> > > >
> > > > I'll go grab this data and copy it over to dakota. Ho-Chun,
sorry
> > > > for the
> > > > delay in getting started with this. I'll go grab it now.
> > > >
> > > > John
> > > >
> > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
>
> > > >>
> > > >> I don't have access to venus.
> > > >> Would you upload the files (config, script, log, and data
files)
> > > >> to MET
> > > >> ftp server?
> > > >>
> > > >> Which MET version are you using?
> > > >>
> > > >> The GOES-AOD data was supported by regrid_data_plane (V8.x).
It's
> > > >> moved
> > > >> to point2grid (V9.0). The regrid_data_plane still processes
GOES-
> > > >> AOD
> > > data
> > > >> in different way (interpolated based on the neighbor data by
using
> > > >> width
> > > >> and shape, not by the physical location). The output is
different
> > > >> from
> > > >> point2grid. VIIRS AOD is not supported by MET, yet. There
might be
> > > >> a
> > > >> workaround.
> > > >>
> > > >> Cheers,
> > > >> Howard
> > > >>
> > > >> On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov wrote:
> > > >> > To Whom It May Concern:
> > > >> >
> > > >> > I have an LatLon 0.03x0.03 satellite observed AOD at
20200519
> > > >> > 20Z
> > > >> > (global
> > > >> > but most with fill_value). I want to use regrid_data_plane
to
> > > >> > regrid
> > > >> > it to
> > > >> > CMAQ Lambert Conformal Grid. But it seems to hang during
the
> > > >> > Interpolation
> > > >> > step.
> > > >> >
> > > >> > Can you find out what went wrong? Sometimes it is more
than an
> > > >> > hour
> > > >> > waiting before I kill it.
> > > >> >
> > > >> > runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > >> > Chun.Huang/plot/viirs/
> > > >> >
> > > >> > with command > bash run.viirs_aqm_aod.sh viirs aqm 20200519
> > > >> > 20200519
> > > >
> > > >> > regrid.log 2>&1 &
> > > >> >
> > > >> > You can check the "regrid.log" in the script directory for
> > > >> > input,
> > > >> > model_grid, and output file location on venus. I will keep
it
> > > >> > going
> > > >> > from May 29 19:15Z.
> > > >> >
> > > >> > One thing I do not understand is that the output file is
already
> > > >> > generated
> > > >> > while the job is still
> > > >> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > >> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
> > > >> > AOD_AQM_aqm_20200519_20.nc
> > > >> >
> > > >> > Ho-Chun Huang
> > > >> >
> > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > >> >
> > > >> > 5830 University Research Ct., Rm. 2792
> > > >> >
> > > >> > College Park, MD 20740
> > > >> >
> > > >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > >> >
> > > >> > 301-683-3958
> > > >>
> > > >>
> > > >>
> > > >>
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: hang on regrid_data_plane
From: Ho-Chun Huang - NOAA Affiliate
Time: Thu Jun 04 09:53:07 2020
Hi, Howard and John:
The VIIRS L3 AOD is on fixed LatLon 0.03x0.03 grid, so its location is
fixed everyday. Only the area of observed data will change for each
hours,
in general moving northward and westward.
This is graphic based on direct read of the input VIIRS L3 AOD file,
for
your reference,
I want to emphasize again, the input VIIRS AOD is gridded and *NOT*
pixel
data, you do not need to recompute the geo-location for each -field
variables.
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Thu, Jun 4, 2020 at 11:39 AM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Howard,
>
> I recall that with the GOES files, you set up regrid_data_plane to
> determine the grid mapping the first time it's run, and then reuse
that
> mapping information in subsequent calls. I really don't know
anything about
> the VIIRS data. Searching online, it looks like the VIIRS data comes
in
> swaths, whereas the GOES data is stationary. So perhaps the approach
of
> pre-computing the grid mappings won't work. I wonder if the swaths
are
> repeated though.
>
> Ho-Chun, do you know if the geometry of the VIIRS data repeats day
after
> day? If we can figure out a way of computing the grid mapping
information
> once ahead of time and then reuse that information, I'm guessing the
tool
> would run much faster.
>
> Thanks,
> John
>
> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT <met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
> >
> > I will investigate why it takes too long.
> >
> > I ran regrid_data_plane for just one variable (AOD_H_Quality,
dimension
> is
> > 6000 by 12000) and it took about 4 hours at dakota.
> >
> > DEBUG 1: Writing output file:
> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
> >
> > real 239m2.552s
> > user 206m42.332s
> > sys 32m18.028s
> >
> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does not have the time
variable
> > "t". So point2grid does not handle this case with workaround.
> >
> > Cheers,
> > Howard
> >
> > On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov wrote:
> > > Hi, John and Howard:
> > >
> > > (1) You should be able to see my MET version in the run script :
9.0
> > >
> > > (2) Background information, we used to discuss between NCAR MET,
> > > NESDIS,
> > > and EMC about using VIIRS pixel AOD data per granule and mapping
pixel
> > > observed AOD into a selected model grid. It has been
determined that
> > > the
> > > data volume is too large for MET to handle VIIRS granule pixel
data
> > > ingestion. EMC has asked NESDIS aerosol groups to provide *L3
> > > "gridded"
> > > nefCDF* files for global and regional AQM model
> > > evaluation/verification.
> > > I am testing the regional one VIIRS-L3-
AOD_*AQM*_20200519_200000.nc,
> > > while Partha Bhattacharjee is testing VIIRS-L3-
> > > AOD_*GEFS*_20200519_120000.nc
> > > for global GEFS-aerosol verification [reference helpdesk ticket
> > > 95358].
> > >
> > > (3) *I am not trying to using point2grid*, I am trying to use
MET
> > > regridding utilities to map a *gridded* netCDF to a target model
> > > *gridded*
> > > netCDF. Unless there is another better tool in MET packages
that I
> > > am not
> > > aware of, I assume regrid_data_plane is for general mapping
purpose.
> > >
> > > (4) For each hourly period, orbital satellites overpassed only a
small
> > > area
> > > around the globe. I have asked NESDIS aerosol to use fill-values
for
> > > non-observed grids and with compress level=1 hoping regard-
data_plane
> > > can
> > > skip those "-9999.f" grid quickly. I may be wrong. In reality,
I may
> > > only
> > > use files from 18Z to 23Z for verification on CONUS, Alaska, and
> > > Hawaii.
> > > The reason to keep it global coverage is to simplify NESDIS
production
> > > processes and be flexible for future applications. Please study
my
> > > problem,
> > > if changes on the data production side [NESDIS aerosol group]
can
> > > speed-up
> > > the regridding processes, please let me know.
> > >
> > > Thank you for your help.
> > >
> > > Ho-Chun Huang
> > >
> > > IMSG at NOAA/NWS/NCEP/EMC
> > >
> > > 5830 University Research Ct., Rm. 2792
> > >
> > > College Park, MD 20740
> > >
> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > >
> > > 301-683-3958
> > >
> > >
> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > > Howard,
> > > >
> > > > OK, please find input data for testing here:
> > > >
> > > >
> >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> > > >
> > > > When you untar this file, you'll find the contents of Ho-
Chun's
> > > > directory:
> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-Chun.Huang/plot/viirs
> > > >
> > > > In addition, I copied the input/output directories from
> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/VIIRS_AOD into
that
> > > > viirs
> > > > directory. Also, I included the file named "aqm.aot.148.grid"
which
> > > > defines
> > > > the target grid.
> > > >
> > > > If you're able to take a look at this data and see how it
runs,
> > > > that'd be
> > > > great! Ho-Chun found that it took over 12 hours!? to run on
WCOSS.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway
<johnhg at ucar.edu>
> > > > wrote:
> > > >
> > > > > Howard,
> > > > >
> > > > > I'll go grab this data and copy it over to dakota. Ho-Chun,
sorry
> > > > > for the
> > > > > delay in getting started with this. I'll go grab it now.
> > > > >
> > > > > John
> > > > >
> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > >>
> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
> > > > >>
> > > > >> I don't have access to venus.
> > > > >> Would you upload the files (config, script, log, and data
files)
> > > > >> to MET
> > > > >> ftp server?
> > > > >>
> > > > >> Which MET version are you using?
> > > > >>
> > > > >> The GOES-AOD data was supported by regrid_data_plane
(V8.x). It's
> > > > >> moved
> > > > >> to point2grid (V9.0). The regrid_data_plane still processes
GOES-
> > > > >> AOD
> > > > data
> > > > >> in different way (interpolated based on the neighbor data
by using
> > > > >> width
> > > > >> and shape, not by the physical location). The output is
different
> > > > >> from
> > > > >> point2grid. VIIRS AOD is not supported by MET, yet. There
might be
> > > > >> a
> > > > >> workaround.
> > > > >>
> > > > >> Cheers,
> > > > >> Howard
> > > > >>
> > > > >> On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov wrote:
> > > > >> > To Whom It May Concern:
> > > > >> >
> > > > >> > I have an LatLon 0.03x0.03 satellite observed AOD at
20200519
> > > > >> > 20Z
> > > > >> > (global
> > > > >> > but most with fill_value). I want to use
regrid_data_plane to
> > > > >> > regrid
> > > > >> > it to
> > > > >> > CMAQ Lambert Conformal Grid. But it seems to hang during
the
> > > > >> > Interpolation
> > > > >> > step.
> > > > >> >
> > > > >> > Can you find out what went wrong? Sometimes it is more
than an
> > > > >> > hour
> > > > >> > waiting before I kill it.
> > > > >> >
> > > > >> > runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > > >> > Chun.Huang/plot/viirs/
> > > > >> >
> > > > >> > with command > bash run.viirs_aqm_aod.sh viirs aqm
20200519
> > > > >> > 20200519
> > > > >
> > > > >> > regrid.log 2>&1 &
> > > > >> >
> > > > >> > You can check the "regrid.log" in the script directory
for
> > > > >> > input,
> > > > >> > model_grid, and output file location on venus. I will
keep it
> > > > >> > going
> > > > >> > from May 29 19:15Z.
> > > > >> >
> > > > >> > One thing I do not understand is that the output file is
already
> > > > >> > generated
> > > > >> > while the job is still
> > > > >> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > >> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
> > > > >> > AOD_AQM_aqm_20200519_20.nc
> > > > >> >
> > > > >> > Ho-Chun Huang
> > > > >> >
> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > > >> >
> > > > >> > 5830 University Research Ct., Rm. 2792
> > > > >> >
> > > > >> > College Park, MD 20740
> > > > >> >
> > > > >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > >> >
> > > > >> > 301-683-3958
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > >
> > > >
> >
> >
> >
> >
>
>
------------------------------------------------
Subject: hang on regrid_data_plane
From: Ho-Chun Huang - NOAA Affiliate
Time: Thu Jun 04 09:57:39 2020
Hi,
The regrid_data_plane already obtained the LatLon projection
information
from input obs file
DEBUG 1: Reading data file:
/gpfs/dell2/emc/modeling/noscrub/Ho-
Chun.Huang/VIIRS_AOD/AOD/20200519/VIIRS-L3-AOD_AQM_20200519_200000.nc
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcMet".
DEBUG 4:
DEBUG 4: Latitude/Longitude Grid Data:
DEBUG 4: lat_ll: -89.985
DEBUG 4: lon_ll: 179.985
DEBUG 4: delta_lat: 0.03
DEBUG 4: delta_lon: 0.03
DEBUG 4: Nlat: 6000
DEBUG 4: Nlon: 12000
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Thu, Jun 4, 2020 at 11:52 AM Ho-Chun Huang - NOAA Affiliate <
ho-chun.huang at noaa.gov> wrote:
> Hi, Howard and John:
>
> The VIIRS L3 AOD is on fixed LatLon 0.03x0.03 grid, so its location
is
> fixed everyday. Only the area of observed data will change for each
hours,
> in general moving northward and westward.
>
> This is graphic based on direct read of the input VIIRS L3 AOD file,
for
> your reference,
>
> I want to emphasize again, the input VIIRS AOD is gridded and *NOT*
pixel
> data, you do not need to recompute the geo-location for each -field
> variables.
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Thu, Jun 4, 2020 at 11:39 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Howard,
>>
>> I recall that with the GOES files, you set up regrid_data_plane to
>> determine the grid mapping the first time it's run, and then reuse
that
>> mapping information in subsequent calls. I really don't know
anything
>> about
>> the VIIRS data. Searching online, it looks like the VIIRS data
comes in
>> swaths, whereas the GOES data is stationary. So perhaps the
approach of
>> pre-computing the grid mappings won't work. I wonder if the swaths
are
>> repeated though.
>>
>> Ho-Chun, do you know if the geometry of the VIIRS data repeats day
after
>> day? If we can figure out a way of computing the grid mapping
information
>> once ahead of time and then reuse that information, I'm guessing
the tool
>> would run much faster.
>>
>> Thanks,
>> John
>>
>> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT
<met_help at ucar.edu>
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
>> >
>> > I will investigate why it takes too long.
>> >
>> > I ran regrid_data_plane for just one variable (AOD_H_Quality,
dimension
>> is
>> > 6000 by 12000) and it took about 4 hours at dakota.
>> >
>> > DEBUG 1: Writing output file:
>> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
>> >
>> > real 239m2.552s
>> > user 206m42.332s
>> > sys 32m18.028s
>> >
>> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does not have the time
variable
>> > "t". So point2grid does not handle this case with workaround.
>> >
>> > Cheers,
>> > Howard
>> >
>> > On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov wrote:
>> > > Hi, John and Howard:
>> > >
>> > > (1) You should be able to see my MET version in the run script
: 9.0
>> > >
>> > > (2) Background information, we used to discuss between NCAR
MET,
>> > > NESDIS,
>> > > and EMC about using VIIRS pixel AOD data per granule and
mapping pixel
>> > > observed AOD into a selected model grid. It has been
determined that
>> > > the
>> > > data volume is too large for MET to handle VIIRS granule pixel
data
>> > > ingestion. EMC has asked NESDIS aerosol groups to provide *L3
>> > > "gridded"
>> > > nefCDF* files for global and regional AQM model
>> > > evaluation/verification.
>> > > I am testing the regional one VIIRS-L3-
AOD_*AQM*_20200519_200000.nc,
>> > > while Partha Bhattacharjee is testing VIIRS-L3-
>> > > AOD_*GEFS*_20200519_120000.nc
>> > > for global GEFS-aerosol verification [reference helpdesk ticket
>> > > 95358].
>> > >
>> > > (3) *I am not trying to using point2grid*, I am trying to use
MET
>> > > regridding utilities to map a *gridded* netCDF to a target
model
>> > > *gridded*
>> > > netCDF. Unless there is another better tool in MET packages
that I
>> > > am not
>> > > aware of, I assume regrid_data_plane is for general mapping
purpose.
>> > >
>> > > (4) For each hourly period, orbital satellites overpassed only
a small
>> > > area
>> > > around the globe. I have asked NESDIS aerosol to use fill-
values for
>> > > non-observed grids and with compress level=1 hoping regard-
data_plane
>> > > can
>> > > skip those "-9999.f" grid quickly. I may be wrong. In
reality, I may
>> > > only
>> > > use files from 18Z to 23Z for verification on CONUS, Alaska,
and
>> > > Hawaii.
>> > > The reason to keep it global coverage is to simplify NESDIS
production
>> > > processes and be flexible for future applications. Please study
my
>> > > problem,
>> > > if changes on the data production side [NESDIS aerosol group]
can
>> > > speed-up
>> > > the regridding processes, please let me know.
>> > >
>> > > Thank you for your help.
>> > >
>> > > Ho-Chun Huang
>> > >
>> > > IMSG at NOAA/NWS/NCEP/EMC
>> > >
>> > > 5830 University Research Ct., Rm. 2792
>> > >
>> > > College Park, MD 20740
>> > >
>> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>> > >
>> > > 301-683-3958
>> > >
>> > >
>> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway via RT
>> > > <met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > > Howard,
>> > > >
>> > > > OK, please find input data for testing here:
>> > > >
>> > > >
>> >
>>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
>> > > >
>> > > > When you untar this file, you'll find the contents of Ho-
Chun's
>> > > > directory:
>> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-Chun.Huang/plot/viirs
>> > > >
>> > > > In addition, I copied the input/output directories from
>> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/VIIRS_AOD into
that
>> > > > viirs
>> > > > directory. Also, I included the file named "aqm.aot.148.grid"
which
>> > > > defines
>> > > > the target grid.
>> > > >
>> > > > If you're able to take a look at this data and see how it
runs,
>> > > > that'd be
>> > > > great! Ho-Chun found that it took over 12 hours!? to run on
WCOSS.
>> > > >
>> > > > Thanks,
>> > > > John
>> > > >
>> > > > On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway
<johnhg at ucar.edu>
>> > > > wrote:
>> > > >
>> > > > > Howard,
>> > > > >
>> > > > > I'll go grab this data and copy it over to dakota. Ho-Chun,
sorry
>> > > > > for the
>> > > > > delay in getting started with this. I'll go grab it now.
>> > > > >
>> > > > > John
>> > > > >
>> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT
>> > > > > <met_help at ucar.edu>
>> > > > > wrote:
>> > > > >
>> > > > >>
>> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
>> > > > >>
>> > > > >> I don't have access to venus.
>> > > > >> Would you upload the files (config, script, log, and data
files)
>> > > > >> to MET
>> > > > >> ftp server?
>> > > > >>
>> > > > >> Which MET version are you using?
>> > > > >>
>> > > > >> The GOES-AOD data was supported by regrid_data_plane
(V8.x). It's
>> > > > >> moved
>> > > > >> to point2grid (V9.0). The regrid_data_plane still
processes GOES-
>> > > > >> AOD
>> > > > data
>> > > > >> in different way (interpolated based on the neighbor data
by
>> using
>> > > > >> width
>> > > > >> and shape, not by the physical location). The output is
different
>> > > > >> from
>> > > > >> point2grid. VIIRS AOD is not supported by MET, yet. There
might
>> be
>> > > > >> a
>> > > > >> workaround.
>> > > > >>
>> > > > >> Cheers,
>> > > > >> Howard
>> > > > >>
>> > > > >> On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov wrote:
>> > > > >> > To Whom It May Concern:
>> > > > >> >
>> > > > >> > I have an LatLon 0.03x0.03 satellite observed AOD at
20200519
>> > > > >> > 20Z
>> > > > >> > (global
>> > > > >> > but most with fill_value). I want to use
regrid_data_plane to
>> > > > >> > regrid
>> > > > >> > it to
>> > > > >> > CMAQ Lambert Conformal Grid. But it seems to hang
during the
>> > > > >> > Interpolation
>> > > > >> > step.
>> > > > >> >
>> > > > >> > Can you find out what went wrong? Sometimes it is more
than an
>> > > > >> > hour
>> > > > >> > waiting before I kill it.
>> > > > >> >
>> > > > >> > runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-
>> > > > >> > Chun.Huang/plot/viirs/
>> > > > >> >
>> > > > >> > with command > bash run.viirs_aqm_aod.sh viirs aqm
20200519
>> > > > >> > 20200519
>> > > > >
>> > > > >> > regrid.log 2>&1 &
>> > > > >> >
>> > > > >> > You can check the "regrid.log" in the script directory
for
>> > > > >> > input,
>> > > > >> > model_grid, and output file location on venus. I will
keep it
>> > > > >> > going
>> > > > >> > from May 29 19:15Z.
>> > > > >> >
>> > > > >> > One thing I do not understand is that the output file is
>> already
>> > > > >> > generated
>> > > > >> > while the job is still
>> > > > >> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
>> > > > >> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
>> > > > >> > AOD_AQM_aqm_20200519_20.nc
>> > > > >> >
>> > > > >> > Ho-Chun Huang
>> > > > >> >
>> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
>> > > > >> >
>> > > > >> > 5830 University Research Ct., Rm. 2792
>> > > > >> >
>> > > > >> > College Park, MD 20740
>> > > > >> >
>> > > > >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>> > > > >> >
>> > > > >> > 301-683-3958
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >>
>> > > >
>> > > >
>> >
>> >
>> >
>> >
>>
>>
------------------------------------------------
Subject: hang on regrid_data_plane
From: Ho-Chun Huang - NOAA Affiliate
Time: Thu Jun 04 10:00:22 2020
Hi,
Sorry for the wrong figure, attached is the direct read from NESDIS
VIIRS
L3 AOD file.
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Thu, Jun 4, 2020 at 11:57 AM Ho-Chun Huang - NOAA Affiliate <
ho-chun.huang at noaa.gov> wrote:
> Hi,
>
> The regrid_data_plane already obtained the LatLon projection
information
> from input obs file
>
> DEBUG 1: Reading data file:
> /gpfs/dell2/emc/modeling/noscrub/Ho-
Chun.Huang/VIIRS_AOD/AOD/20200519/VIIRS-L3-AOD_AQM_20200519_200000.nc
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcMet".
> DEBUG 4:
> DEBUG 4: Latitude/Longitude Grid Data:
> DEBUG 4: lat_ll: -89.985
> DEBUG 4: lon_ll: 179.985
> DEBUG 4: delta_lat: 0.03
> DEBUG 4: delta_lon: 0.03
> DEBUG 4: Nlat: 6000
> DEBUG 4: Nlon: 12000
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Thu, Jun 4, 2020 at 11:52 AM Ho-Chun Huang - NOAA Affiliate <
> ho-chun.huang at noaa.gov> wrote:
>
>> Hi, Howard and John:
>>
>> The VIIRS L3 AOD is on fixed LatLon 0.03x0.03 grid, so its location
is
>> fixed everyday. Only the area of observed data will change for
each hours,
>> in general moving northward and westward.
>>
>> This is graphic based on direct read of the input VIIRS L3 AOD
file, for
>> your reference,
>>
>> I want to emphasize again, the input VIIRS AOD is gridded and *NOT*
>> pixel data, you do not need to recompute the geo-location for each
-field
>> variables.
>>
>> Ho-Chun Huang
>>
>> IMSG at NOAA/NWS/NCEP/EMC
>>
>> 5830 University Research Ct., Rm. 2792
>>
>> College Park, MD 20740
>>
>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>
>> 301-683-3958
>>
>>
>> On Thu, Jun 4, 2020 at 11:39 AM John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>>> Howard,
>>>
>>> I recall that with the GOES files, you set up regrid_data_plane to
>>> determine the grid mapping the first time it's run, and then reuse
that
>>> mapping information in subsequent calls. I really don't know
anything
>>> about
>>> the VIIRS data. Searching online, it looks like the VIIRS data
comes in
>>> swaths, whereas the GOES data is stationary. So perhaps the
approach of
>>> pre-computing the grid mappings won't work. I wonder if the swaths
are
>>> repeated though.
>>>
>>> Ho-Chun, do you know if the geometry of the VIIRS data repeats day
after
>>> day? If we can figure out a way of computing the grid mapping
information
>>> once ahead of time and then reuse that information, I'm guessing
the tool
>>> would run much faster.
>>>
>>> Thanks,
>>> John
>>>
>>> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT
<met_help at ucar.edu>
>>> wrote:
>>>
>>> >
>>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
>>> >
>>> > I will investigate why it takes too long.
>>> >
>>> > I ran regrid_data_plane for just one variable (AOD_H_Quality,
>>> dimension is
>>> > 6000 by 12000) and it took about 4 hours at dakota.
>>> >
>>> > DEBUG 1: Writing output file:
>>> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
>>> >
>>> > real 239m2.552s
>>> > user 206m42.332s
>>> > sys 32m18.028s
>>> >
>>> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does not have the time
>>> variable
>>> > "t". So point2grid does not handle this case with workaround.
>>> >
>>> > Cheers,
>>> > Howard
>>> >
>>> > On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov wrote:
>>> > > Hi, John and Howard:
>>> > >
>>> > > (1) You should be able to see my MET version in the run script
: 9.0
>>> > >
>>> > > (2) Background information, we used to discuss between NCAR
MET,
>>> > > NESDIS,
>>> > > and EMC about using VIIRS pixel AOD data per granule and
mapping
>>> pixel
>>> > > observed AOD into a selected model grid. It has been
determined
>>> that
>>> > > the
>>> > > data volume is too large for MET to handle VIIRS granule pixel
data
>>> > > ingestion. EMC has asked NESDIS aerosol groups to provide *L3
>>> > > "gridded"
>>> > > nefCDF* files for global and regional AQM model
>>> > > evaluation/verification.
>>> > > I am testing the regional one VIIRS-L3-
AOD_*AQM*_20200519_200000.nc,
>>> > > while Partha Bhattacharjee is testing VIIRS-L3-
>>> > > AOD_*GEFS*_20200519_120000.nc
>>> > > for global GEFS-aerosol verification [reference helpdesk
ticket
>>> > > 95358].
>>> > >
>>> > > (3) *I am not trying to using point2grid*, I am trying to use
MET
>>> > > regridding utilities to map a *gridded* netCDF to a target
model
>>> > > *gridded*
>>> > > netCDF. Unless there is another better tool in MET packages
that I
>>> > > am not
>>> > > aware of, I assume regrid_data_plane is for general mapping
purpose.
>>> > >
>>> > > (4) For each hourly period, orbital satellites overpassed only
a
>>> small
>>> > > area
>>> > > around the globe. I have asked NESDIS aerosol to use fill-
values for
>>> > > non-observed grids and with compress level=1 hoping regard-
data_plane
>>> > > can
>>> > > skip those "-9999.f" grid quickly. I may be wrong. In
reality, I
>>> may
>>> > > only
>>> > > use files from 18Z to 23Z for verification on CONUS, Alaska,
and
>>> > > Hawaii.
>>> > > The reason to keep it global coverage is to simplify NESDIS
>>> production
>>> > > processes and be flexible for future applications. Please
study my
>>> > > problem,
>>> > > if changes on the data production side [NESDIS aerosol group]
can
>>> > > speed-up
>>> > > the regridding processes, please let me know.
>>> > >
>>> > > Thank you for your help.
>>> > >
>>> > > Ho-Chun Huang
>>> > >
>>> > > IMSG at NOAA/NWS/NCEP/EMC
>>> > >
>>> > > 5830 University Research Ct., Rm. 2792
>>> > >
>>> > > College Park, MD 20740
>>> > >
>>> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>> > >
>>> > > 301-683-3958
>>> > >
>>> > >
>>> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway via RT
>>> > > <met_help at ucar.edu>
>>> > > wrote:
>>> > >
>>> > > > Howard,
>>> > > >
>>> > > > OK, please find input data for testing here:
>>> > > >
>>> > > >
>>> >
>>>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
>>> > > >
>>> > > > When you untar this file, you'll find the contents of Ho-
Chun's
>>> > > > directory:
>>> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-Chun.Huang/plot/viirs
>>> > > >
>>> > > > In addition, I copied the input/output directories from
>>> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/VIIRS_AOD
into that
>>> > > > viirs
>>> > > > directory. Also, I included the file named
"aqm.aot.148.grid" which
>>> > > > defines
>>> > > > the target grid.
>>> > > >
>>> > > > If you're able to take a look at this data and see how it
runs,
>>> > > > that'd be
>>> > > > great! Ho-Chun found that it took over 12 hours!? to run on
WCOSS.
>>> > > >
>>> > > > Thanks,
>>> > > > John
>>> > > >
>>> > > > On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway
<johnhg at ucar.edu
>>> >
>>> > > > wrote:
>>> > > >
>>> > > > > Howard,
>>> > > > >
>>> > > > > I'll go grab this data and copy it over to dakota. Ho-
Chun, sorry
>>> > > > > for the
>>> > > > > delay in getting started with this. I'll go grab it now.
>>> > > > >
>>> > > > > John
>>> > > > >
>>> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT
>>> > > > > <met_help at ucar.edu>
>>> > > > > wrote:
>>> > > > >
>>> > > > >>
>>> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
>>> > > > >>
>>> > > > >> I don't have access to venus.
>>> > > > >> Would you upload the files (config, script, log, and data
files)
>>> > > > >> to MET
>>> > > > >> ftp server?
>>> > > > >>
>>> > > > >> Which MET version are you using?
>>> > > > >>
>>> > > > >> The GOES-AOD data was supported by regrid_data_plane
(V8.x).
>>> It's
>>> > > > >> moved
>>> > > > >> to point2grid (V9.0). The regrid_data_plane still
processes
>>> GOES-
>>> > > > >> AOD
>>> > > > data
>>> > > > >> in different way (interpolated based on the neighbor data
by
>>> using
>>> > > > >> width
>>> > > > >> and shape, not by the physical location). The output is
>>> different
>>> > > > >> from
>>> > > > >> point2grid. VIIRS AOD is not supported by MET, yet. There
might
>>> be
>>> > > > >> a
>>> > > > >> workaround.
>>> > > > >>
>>> > > > >> Cheers,
>>> > > > >> Howard
>>> > > > >>
>>> > > > >> On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov
wrote:
>>> > > > >> > To Whom It May Concern:
>>> > > > >> >
>>> > > > >> > I have an LatLon 0.03x0.03 satellite observed AOD at
20200519
>>> > > > >> > 20Z
>>> > > > >> > (global
>>> > > > >> > but most with fill_value). I want to use
regrid_data_plane to
>>> > > > >> > regrid
>>> > > > >> > it to
>>> > > > >> > CMAQ Lambert Conformal Grid. But it seems to hang
during the
>>> > > > >> > Interpolation
>>> > > > >> > step.
>>> > > > >> >
>>> > > > >> > Can you find out what went wrong? Sometimes it is more
than
>>> an
>>> > > > >> > hour
>>> > > > >> > waiting before I kill it.
>>> > > > >> >
>>> > > > >> > runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-
>>> > > > >> > Chun.Huang/plot/viirs/
>>> > > > >> >
>>> > > > >> > with command > bash run.viirs_aqm_aod.sh viirs aqm
20200519
>>> > > > >> > 20200519
>>> > > > >
>>> > > > >> > regrid.log 2>&1 &
>>> > > > >> >
>>> > > > >> > You can check the "regrid.log" in the script directory
for
>>> > > > >> > input,
>>> > > > >> > model_grid, and output file location on venus. I will
keep it
>>> > > > >> > going
>>> > > > >> > from May 29 19:15Z.
>>> > > > >> >
>>> > > > >> > One thing I do not understand is that the output file
is
>>> already
>>> > > > >> > generated
>>> > > > >> > while the job is still
>>> > > > >> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
>>> > > > >> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
>>> > > > >> > AOD_AQM_aqm_20200519_20.nc
>>> > > > >> >
>>> > > > >> > Ho-Chun Huang
>>> > > > >> >
>>> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
>>> > > > >> >
>>> > > > >> > 5830 University Research Ct., Rm. 2792
>>> > > > >> >
>>> > > > >> > College Park, MD 20740
>>> > > > >> >
>>> > > > >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>> > > > >> >
>>> > > > >> > 301-683-3958
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > >
>>> > > >
>>> >
>>> >
>>> >
>>> >
>>>
>>>
------------------------------------------------
Subject: hang on regrid_data_plane
From: John Halley Gotway
Time: Thu Jun 04 10:45:38 2020
Ho-Chun,
Great, thanks for clarifying. So the task is for us to figure out why
this
standard regridding step is running so slowly. Howard, seems like we
should
profile this to see what's taking so long. Could serve as a good
opportunity to make the code more efficient.
Thanks,
John
On Thu, Jun 4, 2020 at 10:00 AM Ho-Chun Huang - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
>
> Hi,
>
> Sorry for the wrong figure, attached is the direct read from NESDIS
VIIRS
> L3 AOD file.
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Thu, Jun 4, 2020 at 11:57 AM Ho-Chun Huang - NOAA Affiliate <
> ho-chun.huang at noaa.gov> wrote:
>
> > Hi,
> >
> > The regrid_data_plane already obtained the LatLon projection
information
> > from input obs file
> >
> > DEBUG 1: Reading data file:
> >
> /gpfs/dell2/emc/modeling/noscrub/Ho-
Chun.Huang/VIIRS_AOD/AOD/20200519/VIIRS-L3-AOD_AQM_20200519_200000.nc
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcMet".
> > DEBUG 4:
> > DEBUG 4: Latitude/Longitude Grid Data:
> > DEBUG 4: lat_ll: -89.985
> > DEBUG 4: lon_ll: 179.985
> > DEBUG 4: delta_lat: 0.03
> > DEBUG 4: delta_lon: 0.03
> > DEBUG 4: Nlat: 6000
> > DEBUG 4: Nlon: 12000
> >
> > Ho-Chun Huang
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct., Rm. 2792
> >
> > College Park, MD 20740
> >
> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >
> > 301-683-3958
> >
> >
> > On Thu, Jun 4, 2020 at 11:52 AM Ho-Chun Huang - NOAA Affiliate <
> > ho-chun.huang at noaa.gov> wrote:
> >
> >> Hi, Howard and John:
> >>
> >> The VIIRS L3 AOD is on fixed LatLon 0.03x0.03 grid, so its
location is
> >> fixed everyday. Only the area of observed data will change for
each
> hours,
> >> in general moving northward and westward.
> >>
> >> This is graphic based on direct read of the input VIIRS L3 AOD
file, for
> >> your reference,
> >>
> >> I want to emphasize again, the input VIIRS AOD is gridded and
*NOT*
> >> pixel data, you do not need to recompute the geo-location for
each
> -field
> >> variables.
> >>
> >> Ho-Chun Huang
> >>
> >> IMSG at NOAA/NWS/NCEP/EMC
> >>
> >> 5830 University Research Ct., Rm. 2792
> >>
> >> College Park, MD 20740
> >>
> >> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >>
> >> 301-683-3958
> >>
> >>
> >> On Thu, Jun 4, 2020 at 11:39 AM John Halley Gotway via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >>> Howard,
> >>>
> >>> I recall that with the GOES files, you set up regrid_data_plane
to
> >>> determine the grid mapping the first time it's run, and then
reuse that
> >>> mapping information in subsequent calls. I really don't know
anything
> >>> about
> >>> the VIIRS data. Searching online, it looks like the VIIRS data
comes in
> >>> swaths, whereas the GOES data is stationary. So perhaps the
approach of
> >>> pre-computing the grid mappings won't work. I wonder if the
swaths are
> >>> repeated though.
> >>>
> >>> Ho-Chun, do you know if the geometry of the VIIRS data repeats
day
> after
> >>> day? If we can figure out a way of computing the grid mapping
> information
> >>> once ahead of time and then reuse that information, I'm guessing
the
> tool
> >>> would run much faster.
> >>>
> >>> Thanks,
> >>> John
> >>>
> >>> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT
<met_help at ucar.edu>
> >>> wrote:
> >>>
> >>> >
> >>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
>
> >>> >
> >>> > I will investigate why it takes too long.
> >>> >
> >>> > I ran regrid_data_plane for just one variable (AOD_H_Quality,
> >>> dimension is
> >>> > 6000 by 12000) and it took about 4 hours at dakota.
> >>> >
> >>> > DEBUG 1: Writing output file:
> >>> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
> >>> >
> >>> > real 239m2.552s
> >>> > user 206m42.332s
> >>> > sys 32m18.028s
> >>> >
> >>> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does not have the time
> >>> variable
> >>> > "t". So point2grid does not handle this case with workaround.
> >>> >
> >>> > Cheers,
> >>> > Howard
> >>> >
> >>> > On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov wrote:
> >>> > > Hi, John and Howard:
> >>> > >
> >>> > > (1) You should be able to see my MET version in the run
script :
> 9.0
> >>> > >
> >>> > > (2) Background information, we used to discuss between NCAR
MET,
> >>> > > NESDIS,
> >>> > > and EMC about using VIIRS pixel AOD data per granule and
mapping
> >>> pixel
> >>> > > observed AOD into a selected model grid. It has been
determined
> >>> that
> >>> > > the
> >>> > > data volume is too large for MET to handle VIIRS granule
pixel data
> >>> > > ingestion. EMC has asked NESDIS aerosol groups to provide
*L3
> >>> > > "gridded"
> >>> > > nefCDF* files for global and regional AQM model
> >>> > > evaluation/verification.
> >>> > > I am testing the regional one VIIRS-L3-AOD_*AQM*_
> 20200519_200000.nc,
> >>> > > while Partha Bhattacharjee is testing VIIRS-L3-
> >>> > > AOD_*GEFS*_20200519_120000.nc
> >>> > > for global GEFS-aerosol verification [reference helpdesk
ticket
> >>> > > 95358].
> >>> > >
> >>> > > (3) *I am not trying to using point2grid*, I am trying to
use MET
> >>> > > regridding utilities to map a *gridded* netCDF to a target
model
> >>> > > *gridded*
> >>> > > netCDF. Unless there is another better tool in MET
packages that
> I
> >>> > > am not
> >>> > > aware of, I assume regrid_data_plane is for general mapping
> purpose.
> >>> > >
> >>> > > (4) For each hourly period, orbital satellites overpassed
only a
> >>> small
> >>> > > area
> >>> > > around the globe. I have asked NESDIS aerosol to use fill-
values
> for
> >>> > > non-observed grids and with compress level=1 hoping
> regard-data_plane
> >>> > > can
> >>> > > skip those "-9999.f" grid quickly. I may be wrong. In
reality, I
> >>> may
> >>> > > only
> >>> > > use files from 18Z to 23Z for verification on CONUS, Alaska,
and
> >>> > > Hawaii.
> >>> > > The reason to keep it global coverage is to simplify NESDIS
> >>> production
> >>> > > processes and be flexible for future applications. Please
study my
> >>> > > problem,
> >>> > > if changes on the data production side [NESDIS aerosol
group] can
> >>> > > speed-up
> >>> > > the regridding processes, please let me know.
> >>> > >
> >>> > > Thank you for your help.
> >>> > >
> >>> > > Ho-Chun Huang
> >>> > >
> >>> > > IMSG at NOAA/NWS/NCEP/EMC
> >>> > >
> >>> > > 5830 University Research Ct., Rm. 2792
> >>> > >
> >>> > > College Park, MD 20740
> >>> > >
> >>> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >>> > >
> >>> > > 301-683-3958
> >>> > >
> >>> > >
> >>> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway via RT
> >>> > > <met_help at ucar.edu>
> >>> > > wrote:
> >>> > >
> >>> > > > Howard,
> >>> > > >
> >>> > > > OK, please find input data for testing here:
> >>> > > >
> >>> > > >
> >>> >
> >>>
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> >>> > > >
> >>> > > > When you untar this file, you'll find the contents of Ho-
Chun's
> >>> > > > directory:
> >>> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-
Chun.Huang/plot/viirs
> >>> > > >
> >>> > > > In addition, I copied the input/output directories from
> >>> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/VIIRS_AOD
into
> that
> >>> > > > viirs
> >>> > > > directory. Also, I included the file named
"aqm.aot.148.grid"
> which
> >>> > > > defines
> >>> > > > the target grid.
> >>> > > >
> >>> > > > If you're able to take a look at this data and see how it
runs,
> >>> > > > that'd be
> >>> > > > great! Ho-Chun found that it took over 12 hours!? to run
on
> WCOSS.
> >>> > > >
> >>> > > > Thanks,
> >>> > > > John
> >>> > > >
> >>> > > > On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway <
> johnhg at ucar.edu
> >>> >
> >>> > > > wrote:
> >>> > > >
> >>> > > > > Howard,
> >>> > > > >
> >>> > > > > I'll go grab this data and copy it over to dakota. Ho-
Chun,
> sorry
> >>> > > > > for the
> >>> > > > > delay in getting started with this. I'll go grab it now.
> >>> > > > >
> >>> > > > > John
> >>> > > > >
> >>> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT
> >>> > > > > <met_help at ucar.edu>
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > >>
> >>> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> >
> >>> > > > >>
> >>> > > > >> I don't have access to venus.
> >>> > > > >> Would you upload the files (config, script, log, and
data
> files)
> >>> > > > >> to MET
> >>> > > > >> ftp server?
> >>> > > > >>
> >>> > > > >> Which MET version are you using?
> >>> > > > >>
> >>> > > > >> The GOES-AOD data was supported by regrid_data_plane
(V8.x).
> >>> It's
> >>> > > > >> moved
> >>> > > > >> to point2grid (V9.0). The regrid_data_plane still
processes
> >>> GOES-
> >>> > > > >> AOD
> >>> > > > data
> >>> > > > >> in different way (interpolated based on the neighbor
data by
> >>> using
> >>> > > > >> width
> >>> > > > >> and shape, not by the physical location). The output is
> >>> different
> >>> > > > >> from
> >>> > > > >> point2grid. VIIRS AOD is not supported by MET, yet.
There
> might
> >>> be
> >>> > > > >> a
> >>> > > > >> workaround.
> >>> > > > >>
> >>> > > > >> Cheers,
> >>> > > > >> Howard
> >>> > > > >>
> >>> > > > >> On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov
wrote:
> >>> > > > >> > To Whom It May Concern:
> >>> > > > >> >
> >>> > > > >> > I have an LatLon 0.03x0.03 satellite observed AOD at
> 20200519
> >>> > > > >> > 20Z
> >>> > > > >> > (global
> >>> > > > >> > but most with fill_value). I want to use
regrid_data_plane
> to
> >>> > > > >> > regrid
> >>> > > > >> > it to
> >>> > > > >> > CMAQ Lambert Conformal Grid. But it seems to hang
during
> the
> >>> > > > >> > Interpolation
> >>> > > > >> > step.
> >>> > > > >> >
> >>> > > > >> > Can you find out what went wrong? Sometimes it is
more than
> >>> an
> >>> > > > >> > hour
> >>> > > > >> > waiting before I kill it.
> >>> > > > >> >
> >>> > > > >> > runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-
> >>> > > > >> > Chun.Huang/plot/viirs/
> >>> > > > >> >
> >>> > > > >> > with command > bash run.viirs_aqm_aod.sh viirs aqm
20200519
> >>> > > > >> > 20200519
> >>> > > > >
> >>> > > > >> > regrid.log 2>&1 &
> >>> > > > >> >
> >>> > > > >> > You can check the "regrid.log" in the script
directory for
> >>> > > > >> > input,
> >>> > > > >> > model_grid, and output file location on venus. I
will keep
> it
> >>> > > > >> > going
> >>> > > > >> > from May 29 19:15Z.
> >>> > > > >> >
> >>> > > > >> > One thing I do not understand is that the output file
is
> >>> already
> >>> > > > >> > generated
> >>> > > > >> > while the job is still
> >>> > > > >> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
> >>> > > > >> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
> >>> > > > >> > AOD_AQM_aqm_20200519_20.nc
> >>> > > > >> >
> >>> > > > >> > Ho-Chun Huang
> >>> > > > >> >
> >>> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
> >>> > > > >> >
> >>> > > > >> > 5830 University Research Ct., Rm. 2792
> >>> > > > >> >
> >>> > > > >> > College Park, MD 20740
> >>> > > > >> >
> >>> > > > >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >>> > > > >> >
> >>> > > > >> > 301-683-3958
> >>> > > > >>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >>
> >>> > > >
> >>> > > >
> >>> >
> >>> >
> >>> >
> >>> >
> >>>
> >>>
>
>
------------------------------------------------
Subject: hang on regrid_data_plane
From: Ho-Chun Huang - NOAA Affiliate
Time: Thu Jun 04 10:54:47 2020
Hi, John and Howard:
*Ho-Chun, do you know if the geometry of the VIIRS data repeats day
after
day?*
There are two satellites currently providing VIIRS AOD observations,
SUOMI-NPP and NOAA-20. Eash satellite overpasses the same area twice
per
day, AOD only observed during daytime overpass. The overpass for local
time
should be similar per satellite, NOAA-20 is ~ 50 mins ahead of SUOMI-
NPP.
Although I think daily granule pixel location per satellite may shift
slightly day after day (still need to verify this statement with
NESDIS
personal)? Thus, for L3 gridded AOD data, *the area of coverage* of
orbital satellites should be similar *for the same hour of day*.
Maybe +-
few grid boxes toward the north and west for predefined search area.
Thus,
if you put a predefined search box with an adequate buffer zone on 4
sides
for specific UTC, maybe it will work.
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Thu, Jun 4, 2020 at 12:00 PM Ho-Chun Huang - NOAA Affiliate <
ho-chun.huang at noaa.gov> wrote:
> Hi,
>
> Sorry for the wrong figure, attached is the direct read from NESDIS
VIIRS
> L3 AOD file.
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Thu, Jun 4, 2020 at 11:57 AM Ho-Chun Huang - NOAA Affiliate <
> ho-chun.huang at noaa.gov> wrote:
>
>> Hi,
>>
>> The regrid_data_plane already obtained the LatLon projection
information
>> from input obs file
>>
>> DEBUG 1: Reading data file:
>> /gpfs/dell2/emc/modeling/noscrub/Ho-
Chun.Huang/VIIRS_AOD/AOD/20200519/VIIRS-L3-AOD_AQM_20200519_200000.nc
>> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>> Met2dDataFile object of type "FileType_NcMet".
>> DEBUG 4:
>> DEBUG 4: Latitude/Longitude Grid Data:
>> DEBUG 4: lat_ll: -89.985
>> DEBUG 4: lon_ll: 179.985
>> DEBUG 4: delta_lat: 0.03
>> DEBUG 4: delta_lon: 0.03
>> DEBUG 4: Nlat: 6000
>> DEBUG 4: Nlon: 12000
>>
>> Ho-Chun Huang
>>
>> IMSG at NOAA/NWS/NCEP/EMC
>>
>> 5830 University Research Ct., Rm. 2792
>>
>> College Park, MD 20740
>>
>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>
>> 301-683-3958
>>
>>
>> On Thu, Jun 4, 2020 at 11:52 AM Ho-Chun Huang - NOAA Affiliate <
>> ho-chun.huang at noaa.gov> wrote:
>>
>>> Hi, Howard and John:
>>>
>>> The VIIRS L3 AOD is on fixed LatLon 0.03x0.03 grid, so its
location is
>>> fixed everyday. Only the area of observed data will change for
each hours,
>>> in general moving northward and westward.
>>>
>>> This is graphic based on direct read of the input VIIRS L3 AOD
file, for
>>> your reference,
>>>
>>> I want to emphasize again, the input VIIRS AOD is gridded and
*NOT*
>>> pixel data, you do not need to recompute the geo-location for each
-field
>>> variables.
>>>
>>> Ho-Chun Huang
>>>
>>> IMSG at NOAA/NWS/NCEP/EMC
>>>
>>> 5830 University Research Ct., Rm. 2792
>>>
>>> College Park, MD 20740
>>>
>>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>>
>>> 301-683-3958
>>>
>>>
>>> On Thu, Jun 4, 2020 at 11:39 AM John Halley Gotway via RT <
>>> met_help at ucar.edu> wrote:
>>>
>>>> Howard,
>>>>
>>>> I recall that with the GOES files, you set up regrid_data_plane
to
>>>> determine the grid mapping the first time it's run, and then
reuse that
>>>> mapping information in subsequent calls. I really don't know
anything
>>>> about
>>>> the VIIRS data. Searching online, it looks like the VIIRS data
comes in
>>>> swaths, whereas the GOES data is stationary. So perhaps the
approach of
>>>> pre-computing the grid mappings won't work. I wonder if the
swaths are
>>>> repeated though.
>>>>
>>>> Ho-Chun, do you know if the geometry of the VIIRS data repeats
day after
>>>> day? If we can figure out a way of computing the grid mapping
>>>> information
>>>> once ahead of time and then reuse that information, I'm guessing
the
>>>> tool
>>>> would run much faster.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT
<met_help at ucar.edu>
>>>> wrote:
>>>>
>>>> >
>>>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
>>>> >
>>>> > I will investigate why it takes too long.
>>>> >
>>>> > I ran regrid_data_plane for just one variable (AOD_H_Quality,
>>>> dimension is
>>>> > 6000 by 12000) and it took about 4 hours at dakota.
>>>> >
>>>> > DEBUG 1: Writing output file:
>>>> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
>>>> >
>>>> > real 239m2.552s
>>>> > user 206m42.332s
>>>> > sys 32m18.028s
>>>> >
>>>> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does not have the time
>>>> variable
>>>> > "t". So point2grid does not handle this case with workaround.
>>>> >
>>>> > Cheers,
>>>> > Howard
>>>> >
>>>> > On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov wrote:
>>>> > > Hi, John and Howard:
>>>> > >
>>>> > > (1) You should be able to see my MET version in the run
script : 9.0
>>>> > >
>>>> > > (2) Background information, we used to discuss between NCAR
MET,
>>>> > > NESDIS,
>>>> > > and EMC about using VIIRS pixel AOD data per granule and
mapping
>>>> pixel
>>>> > > observed AOD into a selected model grid. It has been
determined
>>>> that
>>>> > > the
>>>> > > data volume is too large for MET to handle VIIRS granule
pixel data
>>>> > > ingestion. EMC has asked NESDIS aerosol groups to provide
*L3
>>>> > > "gridded"
>>>> > > nefCDF* files for global and regional AQM model
>>>> > > evaluation/verification.
>>>> > > I am testing the regional one VIIRS-L3-
AOD_*AQM*_20200519_200000.nc
>>>> ,
>>>> > > while Partha Bhattacharjee is testing VIIRS-L3-
>>>> > > AOD_*GEFS*_20200519_120000.nc
>>>> > > for global GEFS-aerosol verification [reference helpdesk
ticket
>>>> > > 95358].
>>>> > >
>>>> > > (3) *I am not trying to using point2grid*, I am trying to use
MET
>>>> > > regridding utilities to map a *gridded* netCDF to a target
model
>>>> > > *gridded*
>>>> > > netCDF. Unless there is another better tool in MET packages
that I
>>>> > > am not
>>>> > > aware of, I assume regrid_data_plane is for general mapping
purpose.
>>>> > >
>>>> > > (4) For each hourly period, orbital satellites overpassed
only a
>>>> small
>>>> > > area
>>>> > > around the globe. I have asked NESDIS aerosol to use fill-
values for
>>>> > > non-observed grids and with compress level=1 hoping
>>>> regard-data_plane
>>>> > > can
>>>> > > skip those "-9999.f" grid quickly. I may be wrong. In
reality, I
>>>> may
>>>> > > only
>>>> > > use files from 18Z to 23Z for verification on CONUS, Alaska,
and
>>>> > > Hawaii.
>>>> > > The reason to keep it global coverage is to simplify NESDIS
>>>> production
>>>> > > processes and be flexible for future applications. Please
study my
>>>> > > problem,
>>>> > > if changes on the data production side [NESDIS aerosol group]
can
>>>> > > speed-up
>>>> > > the regridding processes, please let me know.
>>>> > >
>>>> > > Thank you for your help.
>>>> > >
>>>> > > Ho-Chun Huang
>>>> > >
>>>> > > IMSG at NOAA/NWS/NCEP/EMC
>>>> > >
>>>> > > 5830 University Research Ct., Rm. 2792
>>>> > >
>>>> > > College Park, MD 20740
>>>> > >
>>>> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>>> > >
>>>> > > 301-683-3958
>>>> > >
>>>> > >
>>>> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway via RT
>>>> > > <met_help at ucar.edu>
>>>> > > wrote:
>>>> > >
>>>> > > > Howard,
>>>> > > >
>>>> > > > OK, please find input data for testing here:
>>>> > > >
>>>> > > >
>>>> >
>>>>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
>>>> > > >
>>>> > > > When you untar this file, you'll find the contents of Ho-
Chun's
>>>> > > > directory:
>>>> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-
Chun.Huang/plot/viirs
>>>> > > >
>>>> > > > In addition, I copied the input/output directories from
>>>> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/VIIRS_AOD
into that
>>>> > > > viirs
>>>> > > > directory. Also, I included the file named
"aqm.aot.148.grid"
>>>> which
>>>> > > > defines
>>>> > > > the target grid.
>>>> > > >
>>>> > > > If you're able to take a look at this data and see how it
runs,
>>>> > > > that'd be
>>>> > > > great! Ho-Chun found that it took over 12 hours!? to run on
WCOSS.
>>>> > > >
>>>> > > > Thanks,
>>>> > > > John
>>>> > > >
>>>> > > > On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway <
>>>> johnhg at ucar.edu>
>>>> > > > wrote:
>>>> > > >
>>>> > > > > Howard,
>>>> > > > >
>>>> > > > > I'll go grab this data and copy it over to dakota. Ho-
Chun,
>>>> sorry
>>>> > > > > for the
>>>> > > > > delay in getting started with this. I'll go grab it now.
>>>> > > > >
>>>> > > > > John
>>>> > > > >
>>>> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT
>>>> > > > > <met_help at ucar.edu>
>>>> > > > > wrote:
>>>> > > > >
>>>> > > > >>
>>>> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
>>>> >
>>>> > > > >>
>>>> > > > >> I don't have access to venus.
>>>> > > > >> Would you upload the files (config, script, log, and
data
>>>> files)
>>>> > > > >> to MET
>>>> > > > >> ftp server?
>>>> > > > >>
>>>> > > > >> Which MET version are you using?
>>>> > > > >>
>>>> > > > >> The GOES-AOD data was supported by regrid_data_plane
(V8.x).
>>>> It's
>>>> > > > >> moved
>>>> > > > >> to point2grid (V9.0). The regrid_data_plane still
processes
>>>> GOES-
>>>> > > > >> AOD
>>>> > > > data
>>>> > > > >> in different way (interpolated based on the neighbor
data by
>>>> using
>>>> > > > >> width
>>>> > > > >> and shape, not by the physical location). The output is
>>>> different
>>>> > > > >> from
>>>> > > > >> point2grid. VIIRS AOD is not supported by MET, yet.
There
>>>> might be
>>>> > > > >> a
>>>> > > > >> workaround.
>>>> > > > >>
>>>> > > > >> Cheers,
>>>> > > > >> Howard
>>>> > > > >>
>>>> > > > >> On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov
wrote:
>>>> > > > >> > To Whom It May Concern:
>>>> > > > >> >
>>>> > > > >> > I have an LatLon 0.03x0.03 satellite observed AOD at
20200519
>>>> > > > >> > 20Z
>>>> > > > >> > (global
>>>> > > > >> > but most with fill_value). I want to use
regrid_data_plane
>>>> to
>>>> > > > >> > regrid
>>>> > > > >> > it to
>>>> > > > >> > CMAQ Lambert Conformal Grid. But it seems to hang
during the
>>>> > > > >> > Interpolation
>>>> > > > >> > step.
>>>> > > > >> >
>>>> > > > >> > Can you find out what went wrong? Sometimes it is
more than
>>>> an
>>>> > > > >> > hour
>>>> > > > >> > waiting before I kill it.
>>>> > > > >> >
>>>> > > > >> > runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-
>>>> > > > >> > Chun.Huang/plot/viirs/
>>>> > > > >> >
>>>> > > > >> > with command > bash run.viirs_aqm_aod.sh viirs aqm
20200519
>>>> > > > >> > 20200519
>>>> > > > >
>>>> > > > >> > regrid.log 2>&1 &
>>>> > > > >> >
>>>> > > > >> > You can check the "regrid.log" in the script directory
for
>>>> > > > >> > input,
>>>> > > > >> > model_grid, and output file location on venus. I will
keep
>>>> it
>>>> > > > >> > going
>>>> > > > >> > from May 29 19:15Z.
>>>> > > > >> >
>>>> > > > >> > One thing I do not understand is that the output file
is
>>>> already
>>>> > > > >> > generated
>>>> > > > >> > while the job is still
>>>> > > > >> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
>>>> > > > >> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
>>>> > > > >> > AOD_AQM_aqm_20200519_20.nc
>>>> > > > >> >
>>>> > > > >> > Ho-Chun Huang
>>>> > > > >> >
>>>> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
>>>> > > > >> >
>>>> > > > >> > 5830 University Research Ct., Rm. 2792
>>>> > > > >> >
>>>> > > > >> > College Park, MD 20740
>>>> > > > >> >
>>>> > > > >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>>>> > > > >> >
>>>> > > > >> > 301-683-3958
>>>> > > > >>
>>>> > > > >>
>>>> > > > >>
>>>> > > > >>
>>>> > > >
>>>> > > >
>>>> >
>>>> >
>>>> >
>>>> >
>>>>
>>>>
------------------------------------------------
Subject: hang on regrid_data_plane
From: John Halley Gotway
Time: Mon Jun 08 15:05:35 2020
Howard and Ho-Chun,
I'm wondering if we have any updates on this ticket? Howard, have you
been
able to find a reason for this running so slowly?
Thanks,
John
On Thu, Jun 4, 2020 at 10:55 AM Ho-Chun Huang - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
>
> Hi, John and Howard:
>
> *Ho-Chun, do you know if the geometry of the VIIRS data repeats day
after
> day?*
>
>
> There are two satellites currently providing VIIRS AOD observations,
> SUOMI-NPP and NOAA-20. Eash satellite overpasses the same area
twice per
> day, AOD only observed during daytime overpass. The overpass for
local time
> should be similar per satellite, NOAA-20 is ~ 50 mins ahead of
SUOMI-NPP.
>
> Although I think daily granule pixel location per satellite may
shift
> slightly day after day (still need to verify this statement with
NESDIS
> personal)? Thus, for L3 gridded AOD data, *the area of coverage* of
> orbital satellites should be similar *for the same hour of day*.
Maybe +-
> few grid boxes toward the north and west for predefined search area.
Thus,
> if you put a predefined search box with an adequate buffer zone on 4
sides
> for specific UTC, maybe it will work.
>
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Thu, Jun 4, 2020 at 12:00 PM Ho-Chun Huang - NOAA Affiliate <
> ho-chun.huang at noaa.gov> wrote:
>
> > Hi,
> >
> > Sorry for the wrong figure, attached is the direct read from
NESDIS VIIRS
> > L3 AOD file.
> >
> > Ho-Chun Huang
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct., Rm. 2792
> >
> > College Park, MD 20740
> >
> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >
> > 301-683-3958
> >
> >
> > On Thu, Jun 4, 2020 at 11:57 AM Ho-Chun Huang - NOAA Affiliate <
> > ho-chun.huang at noaa.gov> wrote:
> >
> >> Hi,
> >>
> >> The regrid_data_plane already obtained the LatLon projection
information
> >> from input obs file
> >>
> >> DEBUG 1: Reading data file:
> >>
> /gpfs/dell2/emc/modeling/noscrub/Ho-
Chun.Huang/VIIRS_AOD/AOD/20200519/VIIRS-L3-AOD_AQM_20200519_200000.nc
> >> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> >> Met2dDataFile object of type "FileType_NcMet".
> >> DEBUG 4:
> >> DEBUG 4: Latitude/Longitude Grid Data:
> >> DEBUG 4: lat_ll: -89.985
> >> DEBUG 4: lon_ll: 179.985
> >> DEBUG 4: delta_lat: 0.03
> >> DEBUG 4: delta_lon: 0.03
> >> DEBUG 4: Nlat: 6000
> >> DEBUG 4: Nlon: 12000
> >>
> >> Ho-Chun Huang
> >>
> >> IMSG at NOAA/NWS/NCEP/EMC
> >>
> >> 5830 University Research Ct., Rm. 2792
> >>
> >> College Park, MD 20740
> >>
> >> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >>
> >> 301-683-3958
> >>
> >>
> >> On Thu, Jun 4, 2020 at 11:52 AM Ho-Chun Huang - NOAA Affiliate <
> >> ho-chun.huang at noaa.gov> wrote:
> >>
> >>> Hi, Howard and John:
> >>>
> >>> The VIIRS L3 AOD is on fixed LatLon 0.03x0.03 grid, so its
location is
> >>> fixed everyday. Only the area of observed data will change for
each
> hours,
> >>> in general moving northward and westward.
> >>>
> >>> This is graphic based on direct read of the input VIIRS L3 AOD
file,
> for
> >>> your reference,
> >>>
> >>> I want to emphasize again, the input VIIRS AOD is gridded and
*NOT*
> >>> pixel data, you do not need to recompute the geo-location for
each
> -field
> >>> variables.
> >>>
> >>> Ho-Chun Huang
> >>>
> >>> IMSG at NOAA/NWS/NCEP/EMC
> >>>
> >>> 5830 University Research Ct., Rm. 2792
> >>>
> >>> College Park, MD 20740
> >>>
> >>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >>>
> >>> 301-683-3958
> >>>
> >>>
> >>> On Thu, Jun 4, 2020 at 11:39 AM John Halley Gotway via RT <
> >>> met_help at ucar.edu> wrote:
> >>>
> >>>> Howard,
> >>>>
> >>>> I recall that with the GOES files, you set up regrid_data_plane
to
> >>>> determine the grid mapping the first time it's run, and then
reuse
> that
> >>>> mapping information in subsequent calls. I really don't know
anything
> >>>> about
> >>>> the VIIRS data. Searching online, it looks like the VIIRS data
comes
> in
> >>>> swaths, whereas the GOES data is stationary. So perhaps the
approach
> of
> >>>> pre-computing the grid mappings won't work. I wonder if the
swaths are
> >>>> repeated though.
> >>>>
> >>>> Ho-Chun, do you know if the geometry of the VIIRS data repeats
day
> after
> >>>> day? If we can figure out a way of computing the grid mapping
> >>>> information
> >>>> once ahead of time and then reuse that information, I'm
guessing the
> >>>> tool
> >>>> would run much faster.
> >>>>
> >>>> Thanks,
> >>>> John
> >>>>
> >>>> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT
<met_help at ucar.edu>
> >>>> wrote:
> >>>>
> >>>> >
> >>>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
>
> >>>> >
> >>>> > I will investigate why it takes too long.
> >>>> >
> >>>> > I ran regrid_data_plane for just one variable (AOD_H_Quality,
> >>>> dimension is
> >>>> > 6000 by 12000) and it took about 4 hours at dakota.
> >>>> >
> >>>> > DEBUG 1: Writing output file:
> >>>> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
> >>>> >
> >>>> > real 239m2.552s
> >>>> > user 206m42.332s
> >>>> > sys 32m18.028s
> >>>> >
> >>>> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does not have the
time
> >>>> variable
> >>>> > "t". So point2grid does not handle this case with workaround.
> >>>> >
> >>>> > Cheers,
> >>>> > Howard
> >>>> >
> >>>> > On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov wrote:
> >>>> > > Hi, John and Howard:
> >>>> > >
> >>>> > > (1) You should be able to see my MET version in the run
script :
> 9.0
> >>>> > >
> >>>> > > (2) Background information, we used to discuss between NCAR
MET,
> >>>> > > NESDIS,
> >>>> > > and EMC about using VIIRS pixel AOD data per granule and
mapping
> >>>> pixel
> >>>> > > observed AOD into a selected model grid. It has been
determined
> >>>> that
> >>>> > > the
> >>>> > > data volume is too large for MET to handle VIIRS granule
pixel
> data
> >>>> > > ingestion. EMC has asked NESDIS aerosol groups to provide
*L3
> >>>> > > "gridded"
> >>>> > > nefCDF* files for global and regional AQM model
> >>>> > > evaluation/verification.
> >>>> > > I am testing the regional one VIIRS-L3-AOD_*AQM*_
> 20200519_200000.nc
> >>>> ,
> >>>> > > while Partha Bhattacharjee is testing VIIRS-L3-
> >>>> > > AOD_*GEFS*_20200519_120000.nc
> >>>> > > for global GEFS-aerosol verification [reference helpdesk
ticket
> >>>> > > 95358].
> >>>> > >
> >>>> > > (3) *I am not trying to using point2grid*, I am trying to
use MET
> >>>> > > regridding utilities to map a *gridded* netCDF to a target
model
> >>>> > > *gridded*
> >>>> > > netCDF. Unless there is another better tool in MET
packages
> that I
> >>>> > > am not
> >>>> > > aware of, I assume regrid_data_plane is for general mapping
> purpose.
> >>>> > >
> >>>> > > (4) For each hourly period, orbital satellites overpassed
only a
> >>>> small
> >>>> > > area
> >>>> > > around the globe. I have asked NESDIS aerosol to use fill-
values
> for
> >>>> > > non-observed grids and with compress level=1 hoping
> >>>> regard-data_plane
> >>>> > > can
> >>>> > > skip those "-9999.f" grid quickly. I may be wrong. In
reality, I
> >>>> may
> >>>> > > only
> >>>> > > use files from 18Z to 23Z for verification on CONUS,
Alaska, and
> >>>> > > Hawaii.
> >>>> > > The reason to keep it global coverage is to simplify NESDIS
> >>>> production
> >>>> > > processes and be flexible for future applications. Please
study my
> >>>> > > problem,
> >>>> > > if changes on the data production side [NESDIS aerosol
group] can
> >>>> > > speed-up
> >>>> > > the regridding processes, please let me know.
> >>>> > >
> >>>> > > Thank you for your help.
> >>>> > >
> >>>> > > Ho-Chun Huang
> >>>> > >
> >>>> > > IMSG at NOAA/NWS/NCEP/EMC
> >>>> > >
> >>>> > > 5830 University Research Ct., Rm. 2792
> >>>> > >
> >>>> > > College Park, MD 20740
> >>>> > >
> >>>> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >>>> > >
> >>>> > > 301-683-3958
> >>>> > >
> >>>> > >
> >>>> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway via RT
> >>>> > > <met_help at ucar.edu>
> >>>> > > wrote:
> >>>> > >
> >>>> > > > Howard,
> >>>> > > >
> >>>> > > > OK, please find input data for testing here:
> >>>> > > >
> >>>> > > >
> >>>> >
> >>>>
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> >>>> > > >
> >>>> > > > When you untar this file, you'll find the contents of Ho-
Chun's
> >>>> > > > directory:
> >>>> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-
Chun.Huang/plot/viirs
> >>>> > > >
> >>>> > > > In addition, I copied the input/output directories from
> >>>> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-Chun.Huang/VIIRS_AOD
into
> that
> >>>> > > > viirs
> >>>> > > > directory. Also, I included the file named
"aqm.aot.148.grid"
> >>>> which
> >>>> > > > defines
> >>>> > > > the target grid.
> >>>> > > >
> >>>> > > > If you're able to take a look at this data and see how it
runs,
> >>>> > > > that'd be
> >>>> > > > great! Ho-Chun found that it took over 12 hours!? to run
on
> WCOSS.
> >>>> > > >
> >>>> > > > Thanks,
> >>>> > > > John
> >>>> > > >
> >>>> > > > On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway <
> >>>> johnhg at ucar.edu>
> >>>> > > > wrote:
> >>>> > > >
> >>>> > > > > Howard,
> >>>> > > > >
> >>>> > > > > I'll go grab this data and copy it over to dakota. Ho-
Chun,
> >>>> sorry
> >>>> > > > > for the
> >>>> > > > > delay in getting started with this. I'll go grab it
now.
> >>>> > > > >
> >>>> > > > > John
> >>>> > > > >
> >>>> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT
> >>>> > > > > <met_help at ucar.edu>
> >>>> > > > > wrote:
> >>>> > > > >
> >>>> > > > >>
> >>>> > > > >> <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> >>>> >
> >>>> > > > >>
> >>>> > > > >> I don't have access to venus.
> >>>> > > > >> Would you upload the files (config, script, log, and
data
> >>>> files)
> >>>> > > > >> to MET
> >>>> > > > >> ftp server?
> >>>> > > > >>
> >>>> > > > >> Which MET version are you using?
> >>>> > > > >>
> >>>> > > > >> The GOES-AOD data was supported by regrid_data_plane
(V8.x).
> >>>> It's
> >>>> > > > >> moved
> >>>> > > > >> to point2grid (V9.0). The regrid_data_plane still
processes
> >>>> GOES-
> >>>> > > > >> AOD
> >>>> > > > data
> >>>> > > > >> in different way (interpolated based on the neighbor
data by
> >>>> using
> >>>> > > > >> width
> >>>> > > > >> and shape, not by the physical location). The output
is
> >>>> different
> >>>> > > > >> from
> >>>> > > > >> point2grid. VIIRS AOD is not supported by MET, yet.
There
> >>>> might be
> >>>> > > > >> a
> >>>> > > > >> workaround.
> >>>> > > > >>
> >>>> > > > >> Cheers,
> >>>> > > > >> Howard
> >>>> > > > >>
> >>>> > > > >> On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov
wrote:
> >>>> > > > >> > To Whom It May Concern:
> >>>> > > > >> >
> >>>> > > > >> > I have an LatLon 0.03x0.03 satellite observed AOD at
> 20200519
> >>>> > > > >> > 20Z
> >>>> > > > >> > (global
> >>>> > > > >> > but most with fill_value). I want to use
regrid_data_plane
> >>>> to
> >>>> > > > >> > regrid
> >>>> > > > >> > it to
> >>>> > > > >> > CMAQ Lambert Conformal Grid. But it seems to hang
during
> the
> >>>> > > > >> > Interpolation
> >>>> > > > >> > step.
> >>>> > > > >> >
> >>>> > > > >> > Can you find out what went wrong? Sometimes it is
more
> than
> >>>> an
> >>>> > > > >> > hour
> >>>> > > > >> > waiting before I kill it.
> >>>> > > > >> >
> >>>> > > > >> > runscript in venus:/gpfs/dell2/emc/modeling/save/Ho-
> >>>> > > > >> > Chun.Huang/plot/viirs/
> >>>> > > > >> >
> >>>> > > > >> > with command > bash run.viirs_aqm_aod.sh viirs aqm
> 20200519
> >>>> > > > >> > 20200519
> >>>> > > > >
> >>>> > > > >> > regrid.log 2>&1 &
> >>>> > > > >> >
> >>>> > > > >> > You can check the "regrid.log" in the script
directory for
> >>>> > > > >> > input,
> >>>> > > > >> > model_grid, and output file location on venus. I
will keep
> >>>> it
> >>>> > > > >> > going
> >>>> > > > >> > from May 29 19:15Z.
> >>>> > > > >> >
> >>>> > > > >> > One thing I do not understand is that the output
file is
> >>>> already
> >>>> > > > >> > generated
> >>>> > > > >> > while the job is still
> >>>> > > > >> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
> >>>> > > > >> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
> >>>> > > > >> > AOD_AQM_aqm_20200519_20.nc
> >>>> > > > >> >
> >>>> > > > >> > Ho-Chun Huang
> >>>> > > > >> >
> >>>> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
> >>>> > > > >> >
> >>>> > > > >> > 5830 University Research Ct., Rm. 2792
> >>>> > > > >> >
> >>>> > > > >> > College Park, MD 20740
> >>>> > > > >> >
> >>>> > > > >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >>>> > > > >> >
> >>>> > > > >> > 301-683-3958
> >>>> > > > >>
> >>>> > > > >>
> >>>> > > > >>
> >>>> > > > >>
> >>>> > > >
> >>>> > > >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>>
> >>>>
>
>
------------------------------------------------
Subject: hang on regrid_data_plane
From: Howard Soh
Time: Mon Jun 08 15:35:27 2020
The first cause is reading the variable 6000 times (dimension is 10000
by 6000). The variables are compressed with with level 9 (highest). It
taook more than 1 second to read & decompress a variable at dakota.
if 1 seconds, 1 * 6000 = 6000 seconds = 100 minutes
if 1.5 seconds, 1.5 * 6000 = 9000 seconds = 150 minutes.
The first step is reading a variable one time instead of 6000 times.
But the output did not match, so I'm working on making the same
result. Once this is done, I will check if there is more room to speed
up (some steps are executed 60M times).
Cheers,
Howard
On Mon Jun 08 15:05:35 2020, johnhg wrote:
> Howard and Ho-Chun,
>
> I'm wondering if we have any updates on this ticket? Howard, have
you
> been
> able to find a reason for this running so slowly?
>
> Thanks,
> John
>
> On Thu, Jun 4, 2020 at 10:55 AM Ho-Chun Huang - NOAA Affiliate via
RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
> >
> > Hi, John and Howard:
> >
> > *Ho-Chun, do you know if the geometry of the VIIRS data repeats
day
> > after
> > day?*
> >
> >
> > There are two satellites currently providing VIIRS AOD
observations,
> > SUOMI-NPP and NOAA-20. Eash satellite overpasses the same area
twice
> > per
> > day, AOD only observed during daytime overpass. The overpass for
> > local time
> > should be similar per satellite, NOAA-20 is ~ 50 mins ahead of
SUOMI-
> > NPP.
> >
> > Although I think daily granule pixel location per satellite may
shift
> > slightly day after day (still need to verify this statement with
> > NESDIS
> > personal)? Thus, for L3 gridded AOD data, *the area of coverage*
of
> > orbital satellites should be similar *for the same hour of day*.
> > Maybe +-
> > few grid boxes toward the north and west for predefined search
area.
> > Thus,
> > if you put a predefined search box with an adequate buffer zone on
4
> > sides
> > for specific UTC, maybe it will work.
> >
> >
> > Ho-Chun Huang
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct., Rm. 2792
> >
> > College Park, MD 20740
> >
> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >
> > 301-683-3958
> >
> >
> > On Thu, Jun 4, 2020 at 12:00 PM Ho-Chun Huang - NOAA Affiliate <
> > ho-chun.huang at noaa.gov> wrote:
> >
> > > Hi,
> > >
> > > Sorry for the wrong figure, attached is the direct read from
NESDIS
> > > VIIRS
> > > L3 AOD file.
> > >
> > > Ho-Chun Huang
> > >
> > > IMSG at NOAA/NWS/NCEP/EMC
> > >
> > > 5830 University Research Ct., Rm. 2792
> > >
> > > College Park, MD 20740
> > >
> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > >
> > > 301-683-3958
> > >
> > >
> > > On Thu, Jun 4, 2020 at 11:57 AM Ho-Chun Huang - NOAA Affiliate <
> > > ho-chun.huang at noaa.gov> wrote:
> > >
> > >> Hi,
> > >>
> > >> The regrid_data_plane already obtained the LatLon projection
> > >> information
> > >> from input obs file
> > >>
> > >> DEBUG 1: Reading data file:
> > >>
> > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > Chun.Huang/VIIRS_AOD/AOD/20200519/VIIRS-L3-
AOD_AQM_20200519_200000.nc
> > >> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > >> new
> > >> Met2dDataFile object of type "FileType_NcMet".
> > >> DEBUG 4:
> > >> DEBUG 4: Latitude/Longitude Grid Data:
> > >> DEBUG 4: lat_ll: -89.985
> > >> DEBUG 4: lon_ll: 179.985
> > >> DEBUG 4: delta_lat: 0.03
> > >> DEBUG 4: delta_lon: 0.03
> > >> DEBUG 4: Nlat: 6000
> > >> DEBUG 4: Nlon: 12000
> > >>
> > >> Ho-Chun Huang
> > >>
> > >> IMSG at NOAA/NWS/NCEP/EMC
> > >>
> > >> 5830 University Research Ct., Rm. 2792
> > >>
> > >> College Park, MD 20740
> > >>
> > >> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > >>
> > >> 301-683-3958
> > >>
> > >>
> > >> On Thu, Jun 4, 2020 at 11:52 AM Ho-Chun Huang - NOAA Affiliate
<
> > >> ho-chun.huang at noaa.gov> wrote:
> > >>
> > >>> Hi, Howard and John:
> > >>>
> > >>> The VIIRS L3 AOD is on fixed LatLon 0.03x0.03 grid, so its
> > >>> location is
> > >>> fixed everyday. Only the area of observed data will change
for
> > >>> each
> > hours,
> > >>> in general moving northward and westward.
> > >>>
> > >>> This is graphic based on direct read of the input VIIRS L3 AOD
> > >>> file,
> > for
> > >>> your reference,
> > >>>
> > >>> I want to emphasize again, the input VIIRS AOD is gridded and
> > >>> *NOT*
> > >>> pixel data, you do not need to recompute the geo-location for
> > >>> each
> > -field
> > >>> variables.
> > >>>
> > >>> Ho-Chun Huang
> > >>>
> > >>> IMSG at NOAA/NWS/NCEP/EMC
> > >>>
> > >>> 5830 University Research Ct., Rm. 2792
> > >>>
> > >>> College Park, MD 20740
> > >>>
> > >>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > >>>
> > >>> 301-683-3958
> > >>>
> > >>>
> > >>> On Thu, Jun 4, 2020 at 11:39 AM John Halley Gotway via RT <
> > >>> met_help at ucar.edu> wrote:
> > >>>
> > >>>> Howard,
> > >>>>
> > >>>> I recall that with the GOES files, you set up
regrid_data_plane
> > >>>> to
> > >>>> determine the grid mapping the first time it's run, and then
> > >>>> reuse
> > that
> > >>>> mapping information in subsequent calls. I really don't know
> > >>>> anything
> > >>>> about
> > >>>> the VIIRS data. Searching online, it looks like the VIIRS
data
> > >>>> comes
> > in
> > >>>> swaths, whereas the GOES data is stationary. So perhaps the
> > >>>> approach
> > of
> > >>>> pre-computing the grid mappings won't work. I wonder if the
> > >>>> swaths are
> > >>>> repeated though.
> > >>>>
> > >>>> Ho-Chun, do you know if the geometry of the VIIRS data
repeats
> > >>>> day
> > after
> > >>>> day? If we can figure out a way of computing the grid mapping
> > >>>> information
> > >>>> once ahead of time and then reuse that information, I'm
guessing
> > >>>> the
> > >>>> tool
> > >>>> would run much faster.
> > >>>>
> > >>>> Thanks,
> > >>>> John
> > >>>>
> > >>>> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT
> > >>>> <met_help at ucar.edu>
> > >>>> wrote:
> > >>>>
> > >>>> >
> > >>>> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > >>>> > >
> > >>>> >
> > >>>> > I will investigate why it takes too long.
> > >>>> >
> > >>>> > I ran regrid_data_plane for just one variable
(AOD_H_Quality,
> > >>>> dimension is
> > >>>> > 6000 by 12000) and it took about 4 hours at dakota.
> > >>>> >
> > >>>> > DEBUG 1: Writing output file:
> > >>>> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
> > >>>> >
> > >>>> > real 239m2.552s
> > >>>> > user 206m42.332s
> > >>>> > sys 32m18.028s
> > >>>> >
> > >>>> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does not have the
time
> > >>>> variable
> > >>>> > "t". So point2grid does not handle this case with
workaround.
> > >>>> >
> > >>>> > Cheers,
> > >>>> > Howard
> > >>>> >
> > >>>> > On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov wrote:
> > >>>> > > Hi, John and Howard:
> > >>>> > >
> > >>>> > > (1) You should be able to see my MET version in the run
> > >>>> > > script :
> > 9.0
> > >>>> > >
> > >>>> > > (2) Background information, we used to discuss between
NCAR
> > >>>> > > MET,
> > >>>> > > NESDIS,
> > >>>> > > and EMC about using VIIRS pixel AOD data per granule and
> > >>>> > > mapping
> > >>>> pixel
> > >>>> > > observed AOD into a selected model grid. It has been
> > >>>> > > determined
> > >>>> that
> > >>>> > > the
> > >>>> > > data volume is too large for MET to handle VIIRS granule
> > >>>> > > pixel
> > data
> > >>>> > > ingestion. EMC has asked NESDIS aerosol groups to
provide
> > >>>> > > *L3
> > >>>> > > "gridded"
> > >>>> > > nefCDF* files for global and regional AQM model
> > >>>> > > evaluation/verification.
> > >>>> > > I am testing the regional one VIIRS-L3-AOD_*AQM*_
> > 20200519_200000.nc
> > >>>> ,
> > >>>> > > while Partha Bhattacharjee is testing VIIRS-L3-
> > >>>> > > AOD_*GEFS*_20200519_120000.nc
> > >>>> > > for global GEFS-aerosol verification [reference helpdesk
> > >>>> > > ticket
> > >>>> > > 95358].
> > >>>> > >
> > >>>> > > (3) *I am not trying to using point2grid*, I am trying to
> > >>>> > > use MET
> > >>>> > > regridding utilities to map a *gridded* netCDF to a
target
> > >>>> > > model
> > >>>> > > *gridded*
> > >>>> > > netCDF. Unless there is another better tool in MET
> > >>>> > > packages
> > that I
> > >>>> > > am not
> > >>>> > > aware of, I assume regrid_data_plane is for general
mapping
> > purpose.
> > >>>> > >
> > >>>> > > (4) For each hourly period, orbital satellites overpassed
> > >>>> > > only a
> > >>>> small
> > >>>> > > area
> > >>>> > > around the globe. I have asked NESDIS aerosol to use
fill-
> > >>>> > > values
> > for
> > >>>> > > non-observed grids and with compress level=1 hoping
> > >>>> regard-data_plane
> > >>>> > > can
> > >>>> > > skip those "-9999.f" grid quickly. I may be wrong. In
> > >>>> > > reality, I
> > >>>> may
> > >>>> > > only
> > >>>> > > use files from 18Z to 23Z for verification on CONUS,
Alaska,
> > >>>> > > and
> > >>>> > > Hawaii.
> > >>>> > > The reason to keep it global coverage is to simplify
NESDIS
> > >>>> production
> > >>>> > > processes and be flexible for future applications. Please
> > >>>> > > study my
> > >>>> > > problem,
> > >>>> > > if changes on the data production side [NESDIS aerosol
> > >>>> > > group] can
> > >>>> > > speed-up
> > >>>> > > the regridding processes, please let me know.
> > >>>> > >
> > >>>> > > Thank you for your help.
> > >>>> > >
> > >>>> > > Ho-Chun Huang
> > >>>> > >
> > >>>> > > IMSG at NOAA/NWS/NCEP/EMC
> > >>>> > >
> > >>>> > > 5830 University Research Ct., Rm. 2792
> > >>>> > >
> > >>>> > > College Park, MD 20740
> > >>>> > >
> > >>>> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > >>>> > >
> > >>>> > > 301-683-3958
> > >>>> > >
> > >>>> > >
> > >>>> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway via RT
> > >>>> > > <met_help at ucar.edu>
> > >>>> > > wrote:
> > >>>> > >
> > >>>> > > > Howard,
> > >>>> > > >
> > >>>> > > > OK, please find input data for testing here:
> > >>>> > > >
> > >>>> > > >
> > >>>> >
> > >>>>
> >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> > >>>> > > >
> > >>>> > > > When you untar this file, you'll find the contents of
Ho-
> > >>>> > > > Chun's
> > >>>> > > > directory:
> > >>>> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-
> > >>>> > > > Chun.Huang/plot/viirs
> > >>>> > > >
> > >>>> > > > In addition, I copied the input/output directories from
> > >>>> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
Chun.Huang/VIIRS_AOD
> > >>>> > > > into
> > that
> > >>>> > > > viirs
> > >>>> > > > directory. Also, I included the file named
> > >>>> > > > "aqm.aot.148.grid"
> > >>>> which
> > >>>> > > > defines
> > >>>> > > > the target grid.
> > >>>> > > >
> > >>>> > > > If you're able to take a look at this data and see how
it
> > >>>> > > > runs,
> > >>>> > > > that'd be
> > >>>> > > > great! Ho-Chun found that it took over 12 hours!? to
run
> > >>>> > > > on
> > WCOSS.
> > >>>> > > >
> > >>>> > > > Thanks,
> > >>>> > > > John
> > >>>> > > >
> > >>>> > > > On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway <
> > >>>> johnhg at ucar.edu>
> > >>>> > > > wrote:
> > >>>> > > >
> > >>>> > > > > Howard,
> > >>>> > > > >
> > >>>> > > > > I'll go grab this data and copy it over to dakota.
Ho-
> > >>>> > > > > Chun,
> > >>>> sorry
> > >>>> > > > > for the
> > >>>> > > > > delay in getting started with this. I'll go grab it
now.
> > >>>> > > > >
> > >>>> > > > > John
> > >>>> > > > >
> > >>>> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT
> > >>>> > > > > <met_help at ucar.edu>
> > >>>> > > > > wrote:
> > >>>> > > > >
> > >>>> > > > >>
> > >>>> > > > >> <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > >>>> >
> > >>>> > > > >>
> > >>>> > > > >> I don't have access to venus.
> > >>>> > > > >> Would you upload the files (config, script, log, and
> > >>>> > > > >> data
> > >>>> files)
> > >>>> > > > >> to MET
> > >>>> > > > >> ftp server?
> > >>>> > > > >>
> > >>>> > > > >> Which MET version are you using?
> > >>>> > > > >>
> > >>>> > > > >> The GOES-AOD data was supported by regrid_data_plane
> > >>>> > > > >> (V8.x).
> > >>>> It's
> > >>>> > > > >> moved
> > >>>> > > > >> to point2grid (V9.0). The regrid_data_plane still
> > >>>> > > > >> processes
> > >>>> GOES-
> > >>>> > > > >> AOD
> > >>>> > > > data
> > >>>> > > > >> in different way (interpolated based on the neighbor
> > >>>> > > > >> data by
> > >>>> using
> > >>>> > > > >> width
> > >>>> > > > >> and shape, not by the physical location). The output
is
> > >>>> different
> > >>>> > > > >> from
> > >>>> > > > >> point2grid. VIIRS AOD is not supported by MET, yet.
> > >>>> > > > >> There
> > >>>> might be
> > >>>> > > > >> a
> > >>>> > > > >> workaround.
> > >>>> > > > >>
> > >>>> > > > >> Cheers,
> > >>>> > > > >> Howard
> > >>>> > > > >>
> > >>>> > > > >> On Fri May 29 13:24:55 2020, ho-chun.huang at noaa.gov
> > >>>> > > > >> wrote:
> > >>>> > > > >> > To Whom It May Concern:
> > >>>> > > > >> >
> > >>>> > > > >> > I have an LatLon 0.03x0.03 satellite observed AOD
at
> > 20200519
> > >>>> > > > >> > 20Z
> > >>>> > > > >> > (global
> > >>>> > > > >> > but most with fill_value). I want to use
> > >>>> > > > >> > regrid_data_plane
> > >>>> to
> > >>>> > > > >> > regrid
> > >>>> > > > >> > it to
> > >>>> > > > >> > CMAQ Lambert Conformal Grid. But it seems to hang
> > >>>> > > > >> > during
> > the
> > >>>> > > > >> > Interpolation
> > >>>> > > > >> > step.
> > >>>> > > > >> >
> > >>>> > > > >> > Can you find out what went wrong? Sometimes it is
> > >>>> > > > >> > more
> > than
> > >>>> an
> > >>>> > > > >> > hour
> > >>>> > > > >> > waiting before I kill it.
> > >>>> > > > >> >
> > >>>> > > > >> > runscript in
venus:/gpfs/dell2/emc/modeling/save/Ho-
> > >>>> > > > >> > Chun.Huang/plot/viirs/
> > >>>> > > > >> >
> > >>>> > > > >> > with command > bash run.viirs_aqm_aod.sh viirs aqm
> > 20200519
> > >>>> > > > >> > 20200519
> > >>>> > > > >
> > >>>> > > > >> > regrid.log 2>&1 &
> > >>>> > > > >> >
> > >>>> > > > >> > You can check the "regrid.log" in the script
> > >>>> > > > >> > directory for
> > >>>> > > > >> > input,
> > >>>> > > > >> > model_grid, and output file location on venus. I
> > >>>> > > > >> > will keep
> > >>>> it
> > >>>> > > > >> > going
> > >>>> > > > >> > from May 29 19:15Z.
> > >>>> > > > >> >
> > >>>> > > > >> > One thing I do not understand is that the output
file
> > >>>> > > > >> > is
> > >>>> already
> > >>>> > > > >> > generated
> > >>>> > > > >> > while the job is still
> > >>>> > > > >> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
> > >>>> > > > >> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-L3-
> > >>>> > > > >> > AOD_AQM_aqm_20200519_20.nc
> > >>>> > > > >> >
> > >>>> > > > >> > Ho-Chun Huang
> > >>>> > > > >> >
> > >>>> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > >>>> > > > >> >
> > >>>> > > > >> > 5830 University Research Ct., Rm. 2792
> > >>>> > > > >> >
> > >>>> > > > >> > College Park, MD 20740
> > >>>> > > > >> >
> > >>>> > > > >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > >>>> > > > >> >
> > >>>> > > > >> > 301-683-3958
> > >>>> > > > >>
> > >>>> > > > >>
> > >>>> > > > >>
> > >>>> > > > >>
> > >>>> > > >
> > >>>> > > >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>> >
> > >>>>
> > >>>>
> >
> >
------------------------------------------------
Subject: hang on regrid_data_plane
From: Ho-Chun Huang - NOAA Affiliate
Time: Mon Jun 08 15:55:29 2020
Hi,
I thought I asked the compress level=1 at the beginning. I will
double
check with NESIDS again, but can you tell me now at what nc compress
level
that regrid_data_plane does not need to decompress the variable for
each
grid. If it can resolved from the data source level, it won't waste
your
time for the special case.
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Mon, Jun 8, 2020 at 5:35 PM Howard Soh via RT <met_help at ucar.edu>
wrote:
> The first cause is reading the variable 6000 times (dimension is
10000 by
> 6000). The variables are compressed with with level 9 (highest). It
taook
> more than 1 second to read & decompress a variable at dakota.
> if 1 seconds, 1 * 6000 = 6000 seconds = 100 minutes
> if 1.5 seconds, 1.5 * 6000 = 9000 seconds = 150 minutes.
>
> The first step is reading a variable one time instead of 6000 times.
But
> the output did not match, so I'm working on making the same result.
Once
> this is done, I will check if there is more room to speed up (some
steps
> are executed 60M times).
>
> Cheers,
> Howard
>
> On Mon Jun 08 15:05:35 2020, johnhg wrote:
> > Howard and Ho-Chun,
> >
> > I'm wondering if we have any updates on this ticket? Howard, have
you
> > been
> > able to find a reason for this running so slowly?
> >
> > Thanks,
> > John
> >
> > On Thu, Jun 4, 2020 at 10:55 AM Ho-Chun Huang - NOAA Affiliate via
RT
> > <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
> > >
> > > Hi, John and Howard:
> > >
> > > *Ho-Chun, do you know if the geometry of the VIIRS data repeats
day
> > > after
> > > day?*
> > >
> > >
> > > There are two satellites currently providing VIIRS AOD
observations,
> > > SUOMI-NPP and NOAA-20. Eash satellite overpasses the same area
twice
> > > per
> > > day, AOD only observed during daytime overpass. The overpass for
> > > local time
> > > should be similar per satellite, NOAA-20 is ~ 50 mins ahead of
SUOMI-
> > > NPP.
> > >
> > > Although I think daily granule pixel location per satellite may
shift
> > > slightly day after day (still need to verify this statement with
> > > NESDIS
> > > personal)? Thus, for L3 gridded AOD data, *the area of
coverage* of
> > > orbital satellites should be similar *for the same hour of day*.
> > > Maybe +-
> > > few grid boxes toward the north and west for predefined search
area.
> > > Thus,
> > > if you put a predefined search box with an adequate buffer zone
on 4
> > > sides
> > > for specific UTC, maybe it will work.
> > >
> > >
> > > Ho-Chun Huang
> > >
> > > IMSG at NOAA/NWS/NCEP/EMC
> > >
> > > 5830 University Research Ct., Rm. 2792
> > >
> > > College Park, MD 20740
> > >
> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > >
> > > 301-683-3958
> > >
> > >
> > > On Thu, Jun 4, 2020 at 12:00 PM Ho-Chun Huang - NOAA Affiliate <
> > > ho-chun.huang at noaa.gov> wrote:
> > >
> > > > Hi,
> > > >
> > > > Sorry for the wrong figure, attached is the direct read from
NESDIS
> > > > VIIRS
> > > > L3 AOD file.
> > > >
> > > > Ho-Chun Huang
> > > >
> > > > IMSG at NOAA/NWS/NCEP/EMC
> > > >
> > > > 5830 University Research Ct., Rm. 2792
> > > >
> > > > College Park, MD 20740
> > > >
> > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > >
> > > > 301-683-3958
> > > >
> > > >
> > > > On Thu, Jun 4, 2020 at 11:57 AM Ho-Chun Huang - NOAA Affiliate
<
> > > > ho-chun.huang at noaa.gov> wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> The regrid_data_plane already obtained the LatLon projection
> > > >> information
> > > >> from input obs file
> > > >>
> > > >> DEBUG 1: Reading data file:
> > > >>
> > > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > Chun.Huang/VIIRS_AOD/AOD/20200519/VIIRS-L3-
AOD_AQM_20200519_200000.nc
> > > >> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > >> new
> > > >> Met2dDataFile object of type "FileType_NcMet".
> > > >> DEBUG 4:
> > > >> DEBUG 4: Latitude/Longitude Grid Data:
> > > >> DEBUG 4: lat_ll: -89.985
> > > >> DEBUG 4: lon_ll: 179.985
> > > >> DEBUG 4: delta_lat: 0.03
> > > >> DEBUG 4: delta_lon: 0.03
> > > >> DEBUG 4: Nlat: 6000
> > > >> DEBUG 4: Nlon: 12000
> > > >>
> > > >> Ho-Chun Huang
> > > >>
> > > >> IMSG at NOAA/NWS/NCEP/EMC
> > > >>
> > > >> 5830 University Research Ct., Rm. 2792
> > > >>
> > > >> College Park, MD 20740
> > > >>
> > > >> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > >>
> > > >> 301-683-3958
> > > >>
> > > >>
> > > >> On Thu, Jun 4, 2020 at 11:52 AM Ho-Chun Huang - NOAA
Affiliate <
> > > >> ho-chun.huang at noaa.gov> wrote:
> > > >>
> > > >>> Hi, Howard and John:
> > > >>>
> > > >>> The VIIRS L3 AOD is on fixed LatLon 0.03x0.03 grid, so its
> > > >>> location is
> > > >>> fixed everyday. Only the area of observed data will change
for
> > > >>> each
> > > hours,
> > > >>> in general moving northward and westward.
> > > >>>
> > > >>> This is graphic based on direct read of the input VIIRS L3
AOD
> > > >>> file,
> > > for
> > > >>> your reference,
> > > >>>
> > > >>> I want to emphasize again, the input VIIRS AOD is gridded
and
> > > >>> *NOT*
> > > >>> pixel data, you do not need to recompute the geo-location
for
> > > >>> each
> > > -field
> > > >>> variables.
> > > >>>
> > > >>> Ho-Chun Huang
> > > >>>
> > > >>> IMSG at NOAA/NWS/NCEP/EMC
> > > >>>
> > > >>> 5830 University Research Ct., Rm. 2792
> > > >>>
> > > >>> College Park, MD 20740
> > > >>>
> > > >>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > >>>
> > > >>> 301-683-3958
> > > >>>
> > > >>>
> > > >>> On Thu, Jun 4, 2020 at 11:39 AM John Halley Gotway via RT <
> > > >>> met_help at ucar.edu> wrote:
> > > >>>
> > > >>>> Howard,
> > > >>>>
> > > >>>> I recall that with the GOES files, you set up
regrid_data_plane
> > > >>>> to
> > > >>>> determine the grid mapping the first time it's run, and
then
> > > >>>> reuse
> > > that
> > > >>>> mapping information in subsequent calls. I really don't
know
> > > >>>> anything
> > > >>>> about
> > > >>>> the VIIRS data. Searching online, it looks like the VIIRS
data
> > > >>>> comes
> > > in
> > > >>>> swaths, whereas the GOES data is stationary. So perhaps the
> > > >>>> approach
> > > of
> > > >>>> pre-computing the grid mappings won't work. I wonder if the
> > > >>>> swaths are
> > > >>>> repeated though.
> > > >>>>
> > > >>>> Ho-Chun, do you know if the geometry of the VIIRS data
repeats
> > > >>>> day
> > > after
> > > >>>> day? If we can figure out a way of computing the grid
mapping
> > > >>>> information
> > > >>>> once ahead of time and then reuse that information, I'm
guessing
> > > >>>> the
> > > >>>> tool
> > > >>>> would run much faster.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> John
> > > >>>>
> > > >>>> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT
> > > >>>> <met_help at ucar.edu>
> > > >>>> wrote:
> > > >>>>
> > > >>>> >
> > > >>>> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > >>>> > >
> > > >>>> >
> > > >>>> > I will investigate why it takes too long.
> > > >>>> >
> > > >>>> > I ran regrid_data_plane for just one variable
(AOD_H_Quality,
> > > >>>> dimension is
> > > >>>> > 6000 by 12000) and it took about 4 hours at dakota.
> > > >>>> >
> > > >>>> > DEBUG 1: Writing output file:
> > > >>>> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
> > > >>>> >
> > > >>>> > real 239m2.552s
> > > >>>> > user 206m42.332s
> > > >>>> > sys 32m18.028s
> > > >>>> >
> > > >>>> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does not have the
time
> > > >>>> variable
> > > >>>> > "t". So point2grid does not handle this case with
workaround.
> > > >>>> >
> > > >>>> > Cheers,
> > > >>>> > Howard
> > > >>>> >
> > > >>>> > On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov
wrote:
> > > >>>> > > Hi, John and Howard:
> > > >>>> > >
> > > >>>> > > (1) You should be able to see my MET version in the run
> > > >>>> > > script :
> > > 9.0
> > > >>>> > >
> > > >>>> > > (2) Background information, we used to discuss between
NCAR
> > > >>>> > > MET,
> > > >>>> > > NESDIS,
> > > >>>> > > and EMC about using VIIRS pixel AOD data per granule
and
> > > >>>> > > mapping
> > > >>>> pixel
> > > >>>> > > observed AOD into a selected model grid. It has been
> > > >>>> > > determined
> > > >>>> that
> > > >>>> > > the
> > > >>>> > > data volume is too large for MET to handle VIIRS
granule
> > > >>>> > > pixel
> > > data
> > > >>>> > > ingestion. EMC has asked NESDIS aerosol groups to
provide
> > > >>>> > > *L3
> > > >>>> > > "gridded"
> > > >>>> > > nefCDF* files for global and regional AQM model
> > > >>>> > > evaluation/verification.
> > > >>>> > > I am testing the regional one VIIRS-L3-AOD_*AQM*_
> > > 20200519_200000.nc
> > > >>>> ,
> > > >>>> > > while Partha Bhattacharjee is testing VIIRS-L3-
> > > >>>> > > AOD_*GEFS*_20200519_120000.nc
> > > >>>> > > for global GEFS-aerosol verification [reference
helpdesk
> > > >>>> > > ticket
> > > >>>> > > 95358].
> > > >>>> > >
> > > >>>> > > (3) *I am not trying to using point2grid*, I am trying
to
> > > >>>> > > use MET
> > > >>>> > > regridding utilities to map a *gridded* netCDF to a
target
> > > >>>> > > model
> > > >>>> > > *gridded*
> > > >>>> > > netCDF. Unless there is another better tool in MET
> > > >>>> > > packages
> > > that I
> > > >>>> > > am not
> > > >>>> > > aware of, I assume regrid_data_plane is for general
mapping
> > > purpose.
> > > >>>> > >
> > > >>>> > > (4) For each hourly period, orbital satellites
overpassed
> > > >>>> > > only a
> > > >>>> small
> > > >>>> > > area
> > > >>>> > > around the globe. I have asked NESDIS aerosol to use
fill-
> > > >>>> > > values
> > > for
> > > >>>> > > non-observed grids and with compress level=1 hoping
> > > >>>> regard-data_plane
> > > >>>> > > can
> > > >>>> > > skip those "-9999.f" grid quickly. I may be wrong. In
> > > >>>> > > reality, I
> > > >>>> may
> > > >>>> > > only
> > > >>>> > > use files from 18Z to 23Z for verification on CONUS,
Alaska,
> > > >>>> > > and
> > > >>>> > > Hawaii.
> > > >>>> > > The reason to keep it global coverage is to simplify
NESDIS
> > > >>>> production
> > > >>>> > > processes and be flexible for future applications.
Please
> > > >>>> > > study my
> > > >>>> > > problem,
> > > >>>> > > if changes on the data production side [NESDIS aerosol
> > > >>>> > > group] can
> > > >>>> > > speed-up
> > > >>>> > > the regridding processes, please let me know.
> > > >>>> > >
> > > >>>> > > Thank you for your help.
> > > >>>> > >
> > > >>>> > > Ho-Chun Huang
> > > >>>> > >
> > > >>>> > > IMSG at NOAA/NWS/NCEP/EMC
> > > >>>> > >
> > > >>>> > > 5830 University Research Ct., Rm. 2792
> > > >>>> > >
> > > >>>> > > College Park, MD 20740
> > > >>>> > >
> > > >>>> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > >>>> > >
> > > >>>> > > 301-683-3958
> > > >>>> > >
> > > >>>> > >
> > > >>>> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway via
RT
> > > >>>> > > <met_help at ucar.edu>
> > > >>>> > > wrote:
> > > >>>> > >
> > > >>>> > > > Howard,
> > > >>>> > > >
> > > >>>> > > > OK, please find input data for testing here:
> > > >>>> > > >
> > > >>>> > > >
> > > >>>> >
> > > >>>>
> > >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> > > >>>> > > >
> > > >>>> > > > When you untar this file, you'll find the contents of
Ho-
> > > >>>> > > > Chun's
> > > >>>> > > > directory:
> > > >>>> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > >>>> > > > Chun.Huang/plot/viirs
> > > >>>> > > >
> > > >>>> > > > In addition, I copied the input/output directories
from
> > > >>>> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
Chun.Huang/VIIRS_AOD
> > > >>>> > > > into
> > > that
> > > >>>> > > > viirs
> > > >>>> > > > directory. Also, I included the file named
> > > >>>> > > > "aqm.aot.148.grid"
> > > >>>> which
> > > >>>> > > > defines
> > > >>>> > > > the target grid.
> > > >>>> > > >
> > > >>>> > > > If you're able to take a look at this data and see
how it
> > > >>>> > > > runs,
> > > >>>> > > > that'd be
> > > >>>> > > > great! Ho-Chun found that it took over 12 hours!? to
run
> > > >>>> > > > on
> > > WCOSS.
> > > >>>> > > >
> > > >>>> > > > Thanks,
> > > >>>> > > > John
> > > >>>> > > >
> > > >>>> > > > On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway <
> > > >>>> johnhg at ucar.edu>
> > > >>>> > > > wrote:
> > > >>>> > > >
> > > >>>> > > > > Howard,
> > > >>>> > > > >
> > > >>>> > > > > I'll go grab this data and copy it over to dakota.
Ho-
> > > >>>> > > > > Chun,
> > > >>>> sorry
> > > >>>> > > > > for the
> > > >>>> > > > > delay in getting started with this. I'll go grab it
now.
> > > >>>> > > > >
> > > >>>> > > > > John
> > > >>>> > > > >
> > > >>>> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT
> > > >>>> > > > > <met_help at ucar.edu>
> > > >>>> > > > > wrote:
> > > >>>> > > > >
> > > >>>> > > > >>
> > > >>>> > > > >> <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > >>>> >
> > > >>>> > > > >>
> > > >>>> > > > >> I don't have access to venus.
> > > >>>> > > > >> Would you upload the files (config, script, log,
and
> > > >>>> > > > >> data
> > > >>>> files)
> > > >>>> > > > >> to MET
> > > >>>> > > > >> ftp server?
> > > >>>> > > > >>
> > > >>>> > > > >> Which MET version are you using?
> > > >>>> > > > >>
> > > >>>> > > > >> The GOES-AOD data was supported by
regrid_data_plane
> > > >>>> > > > >> (V8.x).
> > > >>>> It's
> > > >>>> > > > >> moved
> > > >>>> > > > >> to point2grid (V9.0). The regrid_data_plane still
> > > >>>> > > > >> processes
> > > >>>> GOES-
> > > >>>> > > > >> AOD
> > > >>>> > > > data
> > > >>>> > > > >> in different way (interpolated based on the
neighbor
> > > >>>> > > > >> data by
> > > >>>> using
> > > >>>> > > > >> width
> > > >>>> > > > >> and shape, not by the physical location). The
output is
> > > >>>> different
> > > >>>> > > > >> from
> > > >>>> > > > >> point2grid. VIIRS AOD is not supported by MET,
yet.
> > > >>>> > > > >> There
> > > >>>> might be
> > > >>>> > > > >> a
> > > >>>> > > > >> workaround.
> > > >>>> > > > >>
> > > >>>> > > > >> Cheers,
> > > >>>> > > > >> Howard
> > > >>>> > > > >>
> > > >>>> > > > >> On Fri May 29 13:24:55 2020, ho-
chun.huang at noaa.gov
> > > >>>> > > > >> wrote:
> > > >>>> > > > >> > To Whom It May Concern:
> > > >>>> > > > >> >
> > > >>>> > > > >> > I have an LatLon 0.03x0.03 satellite observed
AOD at
> > > 20200519
> > > >>>> > > > >> > 20Z
> > > >>>> > > > >> > (global
> > > >>>> > > > >> > but most with fill_value). I want to use
> > > >>>> > > > >> > regrid_data_plane
> > > >>>> to
> > > >>>> > > > >> > regrid
> > > >>>> > > > >> > it to
> > > >>>> > > > >> > CMAQ Lambert Conformal Grid. But it seems to
hang
> > > >>>> > > > >> > during
> > > the
> > > >>>> > > > >> > Interpolation
> > > >>>> > > > >> > step.
> > > >>>> > > > >> >
> > > >>>> > > > >> > Can you find out what went wrong? Sometimes it
is
> > > >>>> > > > >> > more
> > > than
> > > >>>> an
> > > >>>> > > > >> > hour
> > > >>>> > > > >> > waiting before I kill it.
> > > >>>> > > > >> >
> > > >>>> > > > >> > runscript in
venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > >>>> > > > >> > Chun.Huang/plot/viirs/
> > > >>>> > > > >> >
> > > >>>> > > > >> > with command > bash run.viirs_aqm_aod.sh viirs
aqm
> > > 20200519
> > > >>>> > > > >> > 20200519
> > > >>>> > > > >
> > > >>>> > > > >> > regrid.log 2>&1 &
> > > >>>> > > > >> >
> > > >>>> > > > >> > You can check the "regrid.log" in the script
> > > >>>> > > > >> > directory for
> > > >>>> > > > >> > input,
> > > >>>> > > > >> > model_grid, and output file location on venus.
I
> > > >>>> > > > >> > will keep
> > > >>>> it
> > > >>>> > > > >> > going
> > > >>>> > > > >> > from May 29 19:15Z.
> > > >>>> > > > >> >
> > > >>>> > > > >> > One thing I do not understand is that the output
file
> > > >>>> > > > >> > is
> > > >>>> already
> > > >>>> > > > >> > generated
> > > >>>> > > > >> > while the job is still
> > > >>>> > > > >> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > >>>> > > > >> > Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-
L3-
> > > >>>> > > > >> > AOD_AQM_aqm_20200519_20.nc
> > > >>>> > > > >> >
> > > >>>> > > > >> > Ho-Chun Huang
> > > >>>> > > > >> >
> > > >>>> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > >>>> > > > >> >
> > > >>>> > > > >> > 5830 University Research Ct., Rm. 2792
> > > >>>> > > > >> >
> > > >>>> > > > >> > College Park, MD 20740
> > > >>>> > > > >> >
> > > >>>> > > > >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > >>>> > > > >> >
> > > >>>> > > > >> > 301-683-3958
> > > >>>> > > > >>
> > > >>>> > > > >>
> > > >>>> > > > >>
> > > >>>> > > > >>
> > > >>>> > > >
> > > >>>> > > >
> > > >>>> >
> > > >>>> >
> > > >>>> >
> > > >>>> >
> > > >>>>
> > > >>>>
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: hang on regrid_data_plane
From: Howard Soh
Time: Tue Jun 09 10:33:35 2020
The compression is a variable based. NetCDF API reads a whole data,
decompresses & slices. With a quick test, it seems to be no big
difference betwend compress level 1 and 2.
File size (by nccopy)
8.1G VIIRS-L3-AOD_AQM_20200519_200000.c0.nc (no compression)
86M VIIRS-L3-AOD_AQM_20200519_200000.c1.nc (compress 1)
86M VIIRS-L3-AOD_AQM_20200519_200000.c2.nc (compress 2)
54M VIIRS-L3-AOD_AQM_20200519_200000.nc (compress 9)
Wall time comparison with plot_data_plane (which reads 6000 times) at
dakota (time is not accurate because the system load is not
consistent):
c0: 0m44.899s
c1: 57m33.530s
c2: 57m22.974s
Cheers,
Howard
On Mon Jun 08 15:55:29 2020, ho-chun.huang at noaa.gov wrote:
> Hi,
>
> I thought I asked the compress level=1 at the beginning. I will
> double
> check with NESIDS again, but can you tell me now at what nc compress
> level
> that regrid_data_plane does not need to decompress the variable for
> each
> grid. If it can resolved from the data source level, it won't waste
> your
> time for the special case.
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Mon, Jun 8, 2020 at 5:35 PM Howard Soh via RT <met_help at ucar.edu>
> wrote:
>
> > The first cause is reading the variable 6000 times (dimension is
> > 10000 by
> > 6000). The variables are compressed with with level 9 (highest).
It
> > taook
> > more than 1 second to read & decompress a variable at dakota.
> > if 1 seconds, 1 * 6000 = 6000 seconds = 100 minutes
> > if 1.5 seconds, 1.5 * 6000 = 9000 seconds = 150 minutes.
> >
> > The first step is reading a variable one time instead of 6000
times.
> > But
> > the output did not match, so I'm working on making the same
result.
> > Once
> > this is done, I will check if there is more room to speed up (some
> > steps
> > are executed 60M times).
> >
> > Cheers,
> > Howard
> >
> > On Mon Jun 08 15:05:35 2020, johnhg wrote:
> > > Howard and Ho-Chun,
> > >
> > > I'm wondering if we have any updates on this ticket? Howard,
have
> > > you
> > > been
> > > able to find a reason for this running so slowly?
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Jun 4, 2020 at 10:55 AM Ho-Chun Huang - NOAA Affiliate
via
> > > RT
> > > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
>
> > > >
> > > > Hi, John and Howard:
> > > >
> > > > *Ho-Chun, do you know if the geometry of the VIIRS data
repeats
> > > > day
> > > > after
> > > > day?*
> > > >
> > > >
> > > > There are two satellites currently providing VIIRS AOD
> > > > observations,
> > > > SUOMI-NPP and NOAA-20. Eash satellite overpasses the same
area
> > > > twice
> > > > per
> > > > day, AOD only observed during daytime overpass. The overpass
for
> > > > local time
> > > > should be similar per satellite, NOAA-20 is ~ 50 mins ahead of
> > > > SUOMI-
> > > > NPP.
> > > >
> > > > Although I think daily granule pixel location per satellite
may
> > > > shift
> > > > slightly day after day (still need to verify this statement
with
> > > > NESDIS
> > > > personal)? Thus, for L3 gridded AOD data, *the area of
coverage*
> > > > of
> > > > orbital satellites should be similar *for the same hour of
day*.
> > > > Maybe +-
> > > > few grid boxes toward the north and west for predefined search
> > > > area.
> > > > Thus,
> > > > if you put a predefined search box with an adequate buffer
zone
> > > > on 4
> > > > sides
> > > > for specific UTC, maybe it will work.
> > > >
> > > >
> > > > Ho-Chun Huang
> > > >
> > > > IMSG at NOAA/NWS/NCEP/EMC
> > > >
> > > > 5830 University Research Ct., Rm. 2792
> > > >
> > > > College Park, MD 20740
> > > >
> > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > >
> > > > 301-683-3958
> > > >
> > > >
> > > > On Thu, Jun 4, 2020 at 12:00 PM Ho-Chun Huang - NOAA Affiliate
<
> > > > ho-chun.huang at noaa.gov> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Sorry for the wrong figure, attached is the direct read from
> > > > > NESDIS
> > > > > VIIRS
> > > > > L3 AOD file.
> > > > >
> > > > > Ho-Chun Huang
> > > > >
> > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > >
> > > > > 5830 University Research Ct., Rm. 2792
> > > > >
> > > > > College Park, MD 20740
> > > > >
> > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > >
> > > > > 301-683-3958
> > > > >
> > > > >
> > > > > On Thu, Jun 4, 2020 at 11:57 AM Ho-Chun Huang - NOAA
Affiliate
> > > > > <
> > > > > ho-chun.huang at noaa.gov> wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> The regrid_data_plane already obtained the LatLon
projection
> > > > >> information
> > > > >> from input obs file
> > > > >>
> > > > >> DEBUG 1: Reading data file:
> > > > >>
> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > Chun.Huang/VIIRS_AOD/AOD/20200519/VIIRS-L3-
> > > > AOD_AQM_20200519_200000.nc
> > > > >> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
> > > > >> created
> > > > >> new
> > > > >> Met2dDataFile object of type "FileType_NcMet".
> > > > >> DEBUG 4:
> > > > >> DEBUG 4: Latitude/Longitude Grid Data:
> > > > >> DEBUG 4: lat_ll: -89.985
> > > > >> DEBUG 4: lon_ll: 179.985
> > > > >> DEBUG 4: delta_lat: 0.03
> > > > >> DEBUG 4: delta_lon: 0.03
> > > > >> DEBUG 4: Nlat: 6000
> > > > >> DEBUG 4: Nlon: 12000
> > > > >>
> > > > >> Ho-Chun Huang
> > > > >>
> > > > >> IMSG at NOAA/NWS/NCEP/EMC
> > > > >>
> > > > >> 5830 University Research Ct., Rm. 2792
> > > > >>
> > > > >> College Park, MD 20740
> > > > >>
> > > > >> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > >>
> > > > >> 301-683-3958
> > > > >>
> > > > >>
> > > > >> On Thu, Jun 4, 2020 at 11:52 AM Ho-Chun Huang - NOAA
Affiliate
> > > > >> <
> > > > >> ho-chun.huang at noaa.gov> wrote:
> > > > >>
> > > > >>> Hi, Howard and John:
> > > > >>>
> > > > >>> The VIIRS L3 AOD is on fixed LatLon 0.03x0.03 grid, so its
> > > > >>> location is
> > > > >>> fixed everyday. Only the area of observed data will
change
> > > > >>> for
> > > > >>> each
> > > > hours,
> > > > >>> in general moving northward and westward.
> > > > >>>
> > > > >>> This is graphic based on direct read of the input VIIRS L3
> > > > >>> AOD
> > > > >>> file,
> > > > for
> > > > >>> your reference,
> > > > >>>
> > > > >>> I want to emphasize again, the input VIIRS AOD is gridded
and
> > > > >>> *NOT*
> > > > >>> pixel data, you do not need to recompute the geo-location
for
> > > > >>> each
> > > > -field
> > > > >>> variables.
> > > > >>>
> > > > >>> Ho-Chun Huang
> > > > >>>
> > > > >>> IMSG at NOAA/NWS/NCEP/EMC
> > > > >>>
> > > > >>> 5830 University Research Ct., Rm. 2792
> > > > >>>
> > > > >>> College Park, MD 20740
> > > > >>>
> > > > >>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > >>>
> > > > >>> 301-683-3958
> > > > >>>
> > > > >>>
> > > > >>> On Thu, Jun 4, 2020 at 11:39 AM John Halley Gotway via RT
<
> > > > >>> met_help at ucar.edu> wrote:
> > > > >>>
> > > > >>>> Howard,
> > > > >>>>
> > > > >>>> I recall that with the GOES files, you set up
> > > > >>>> regrid_data_plane
> > > > >>>> to
> > > > >>>> determine the grid mapping the first time it's run, and
then
> > > > >>>> reuse
> > > > that
> > > > >>>> mapping information in subsequent calls. I really don't
know
> > > > >>>> anything
> > > > >>>> about
> > > > >>>> the VIIRS data. Searching online, it looks like the VIIRS
> > > > >>>> data
> > > > >>>> comes
> > > > in
> > > > >>>> swaths, whereas the GOES data is stationary. So perhaps
the
> > > > >>>> approach
> > > > of
> > > > >>>> pre-computing the grid mappings won't work. I wonder if
the
> > > > >>>> swaths are
> > > > >>>> repeated though.
> > > > >>>>
> > > > >>>> Ho-Chun, do you know if the geometry of the VIIRS data
> > > > >>>> repeats
> > > > >>>> day
> > > > after
> > > > >>>> day? If we can figure out a way of computing the grid
> > > > >>>> mapping
> > > > >>>> information
> > > > >>>> once ahead of time and then reuse that information, I'm
> > > > >>>> guessing
> > > > >>>> the
> > > > >>>> tool
> > > > >>>> would run much faster.
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> John
> > > > >>>>
> > > > >>>> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT
> > > > >>>> <met_help at ucar.edu>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>> >
> > > > >>>> > <URL:
> > > > >>>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > >>>> > >
> > > > >>>> >
> > > > >>>> > I will investigate why it takes too long.
> > > > >>>> >
> > > > >>>> > I ran regrid_data_plane for just one variable
> > > > >>>> > (AOD_H_Quality,
> > > > >>>> dimension is
> > > > >>>> > 6000 by 12000) and it took about 4 hours at dakota.
> > > > >>>> >
> > > > >>>> > DEBUG 1: Writing output file:
> > > > >>>> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
> > > > >>>> >
> > > > >>>> > real 239m2.552s
> > > > >>>> > user 206m42.332s
> > > > >>>> > sys 32m18.028s
> > > > >>>> >
> > > > >>>> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does not have
the
> > > > >>>> > time
> > > > >>>> variable
> > > > >>>> > "t". So point2grid does not handle this case with
> > > > >>>> > workaround.
> > > > >>>> >
> > > > >>>> > Cheers,
> > > > >>>> > Howard
> > > > >>>> >
> > > > >>>> > On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov
wrote:
> > > > >>>> > > Hi, John and Howard:
> > > > >>>> > >
> > > > >>>> > > (1) You should be able to see my MET version in the
run
> > > > >>>> > > script :
> > > > 9.0
> > > > >>>> > >
> > > > >>>> > > (2) Background information, we used to discuss
between
> > > > >>>> > > NCAR
> > > > >>>> > > MET,
> > > > >>>> > > NESDIS,
> > > > >>>> > > and EMC about using VIIRS pixel AOD data per granule
and
> > > > >>>> > > mapping
> > > > >>>> pixel
> > > > >>>> > > observed AOD into a selected model grid. It has
been
> > > > >>>> > > determined
> > > > >>>> that
> > > > >>>> > > the
> > > > >>>> > > data volume is too large for MET to handle VIIRS
granule
> > > > >>>> > > pixel
> > > > data
> > > > >>>> > > ingestion. EMC has asked NESDIS aerosol groups to
> > > > >>>> > > provide
> > > > >>>> > > *L3
> > > > >>>> > > "gridded"
> > > > >>>> > > nefCDF* files for global and regional AQM model
> > > > >>>> > > evaluation/verification.
> > > > >>>> > > I am testing the regional one VIIRS-L3-AOD_*AQM*_
> > > > 20200519_200000.nc
> > > > >>>> ,
> > > > >>>> > > while Partha Bhattacharjee is testing VIIRS-L3-
> > > > >>>> > > AOD_*GEFS*_20200519_120000.nc
> > > > >>>> > > for global GEFS-aerosol verification [reference
helpdesk
> > > > >>>> > > ticket
> > > > >>>> > > 95358].
> > > > >>>> > >
> > > > >>>> > > (3) *I am not trying to using point2grid*, I am
trying
> > > > >>>> > > to
> > > > >>>> > > use MET
> > > > >>>> > > regridding utilities to map a *gridded* netCDF to a
> > > > >>>> > > target
> > > > >>>> > > model
> > > > >>>> > > *gridded*
> > > > >>>> > > netCDF. Unless there is another better tool in MET
> > > > >>>> > > packages
> > > > that I
> > > > >>>> > > am not
> > > > >>>> > > aware of, I assume regrid_data_plane is for general
> > > > >>>> > > mapping
> > > > purpose.
> > > > >>>> > >
> > > > >>>> > > (4) For each hourly period, orbital satellites
> > > > >>>> > > overpassed
> > > > >>>> > > only a
> > > > >>>> small
> > > > >>>> > > area
> > > > >>>> > > around the globe. I have asked NESDIS aerosol to use
> > > > >>>> > > fill-
> > > > >>>> > > values
> > > > for
> > > > >>>> > > non-observed grids and with compress level=1 hoping
> > > > >>>> regard-data_plane
> > > > >>>> > > can
> > > > >>>> > > skip those "-9999.f" grid quickly. I may be wrong.
In
> > > > >>>> > > reality, I
> > > > >>>> may
> > > > >>>> > > only
> > > > >>>> > > use files from 18Z to 23Z for verification on CONUS,
> > > > >>>> > > Alaska,
> > > > >>>> > > and
> > > > >>>> > > Hawaii.
> > > > >>>> > > The reason to keep it global coverage is to simplify
> > > > >>>> > > NESDIS
> > > > >>>> production
> > > > >>>> > > processes and be flexible for future applications.
> > > > >>>> > > Please
> > > > >>>> > > study my
> > > > >>>> > > problem,
> > > > >>>> > > if changes on the data production side [NESDIS
aerosol
> > > > >>>> > > group] can
> > > > >>>> > > speed-up
> > > > >>>> > > the regridding processes, please let me know.
> > > > >>>> > >
> > > > >>>> > > Thank you for your help.
> > > > >>>> > >
> > > > >>>> > > Ho-Chun Huang
> > > > >>>> > >
> > > > >>>> > > IMSG at NOAA/NWS/NCEP/EMC
> > > > >>>> > >
> > > > >>>> > > 5830 University Research Ct., Rm. 2792
> > > > >>>> > >
> > > > >>>> > > College Park, MD 20740
> > > > >>>> > >
> > > > >>>> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > >>>> > >
> > > > >>>> > > 301-683-3958
> > > > >>>> > >
> > > > >>>> > >
> > > > >>>> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway via
RT
> > > > >>>> > > <met_help at ucar.edu>
> > > > >>>> > > wrote:
> > > > >>>> > >
> > > > >>>> > > > Howard,
> > > > >>>> > > >
> > > > >>>> > > > OK, please find input data for testing here:
> > > > >>>> > > >
> > > > >>>> > > >
> > > > >>>> >
> > > > >>>>
> > > >
> >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> > > > >>>> > > >
> > > > >>>> > > > When you untar this file, you'll find the contents
of
> > > > >>>> > > > Ho-
> > > > >>>> > > > Chun's
> > > > >>>> > > > directory:
> > > > >>>> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > > >>>> > > > Chun.Huang/plot/viirs
> > > > >>>> > > >
> > > > >>>> > > > In addition, I copied the input/output directories
> > > > >>>> > > > from
> > > > >>>> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > >>>> > > > Chun.Huang/VIIRS_AOD
> > > > >>>> > > > into
> > > > that
> > > > >>>> > > > viirs
> > > > >>>> > > > directory. Also, I included the file named
> > > > >>>> > > > "aqm.aot.148.grid"
> > > > >>>> which
> > > > >>>> > > > defines
> > > > >>>> > > > the target grid.
> > > > >>>> > > >
> > > > >>>> > > > If you're able to take a look at this data and see
how
> > > > >>>> > > > it
> > > > >>>> > > > runs,
> > > > >>>> > > > that'd be
> > > > >>>> > > > great! Ho-Chun found that it took over 12 hours!?
to
> > > > >>>> > > > run
> > > > >>>> > > > on
> > > > WCOSS.
> > > > >>>> > > >
> > > > >>>> > > > Thanks,
> > > > >>>> > > > John
> > > > >>>> > > >
> > > > >>>> > > > On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway <
> > > > >>>> johnhg at ucar.edu>
> > > > >>>> > > > wrote:
> > > > >>>> > > >
> > > > >>>> > > > > Howard,
> > > > >>>> > > > >
> > > > >>>> > > > > I'll go grab this data and copy it over to
dakota.
> > > > >>>> > > > > Ho-
> > > > >>>> > > > > Chun,
> > > > >>>> sorry
> > > > >>>> > > > > for the
> > > > >>>> > > > > delay in getting started with this. I'll go grab
it
> > > > >>>> > > > > now.
> > > > >>>> > > > >
> > > > >>>> > > > > John
> > > > >>>> > > > >
> > > > >>>> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via RT
> > > > >>>> > > > > <met_help at ucar.edu>
> > > > >>>> > > > > wrote:
> > > > >>>> > > > >
> > > > >>>> > > > >>
> > > > >>>> > > > >> <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > >>>> >
> > > > >>>> > > > >>
> > > > >>>> > > > >> I don't have access to venus.
> > > > >>>> > > > >> Would you upload the files (config, script, log,
> > > > >>>> > > > >> and
> > > > >>>> > > > >> data
> > > > >>>> files)
> > > > >>>> > > > >> to MET
> > > > >>>> > > > >> ftp server?
> > > > >>>> > > > >>
> > > > >>>> > > > >> Which MET version are you using?
> > > > >>>> > > > >>
> > > > >>>> > > > >> The GOES-AOD data was supported by
> > > > >>>> > > > >> regrid_data_plane
> > > > >>>> > > > >> (V8.x).
> > > > >>>> It's
> > > > >>>> > > > >> moved
> > > > >>>> > > > >> to point2grid (V9.0). The regrid_data_plane
still
> > > > >>>> > > > >> processes
> > > > >>>> GOES-
> > > > >>>> > > > >> AOD
> > > > >>>> > > > data
> > > > >>>> > > > >> in different way (interpolated based on the
> > > > >>>> > > > >> neighbor
> > > > >>>> > > > >> data by
> > > > >>>> using
> > > > >>>> > > > >> width
> > > > >>>> > > > >> and shape, not by the physical location). The
> > > > >>>> > > > >> output is
> > > > >>>> different
> > > > >>>> > > > >> from
> > > > >>>> > > > >> point2grid. VIIRS AOD is not supported by MET,
yet.
> > > > >>>> > > > >> There
> > > > >>>> might be
> > > > >>>> > > > >> a
> > > > >>>> > > > >> workaround.
> > > > >>>> > > > >>
> > > > >>>> > > > >> Cheers,
> > > > >>>> > > > >> Howard
> > > > >>>> > > > >>
> > > > >>>> > > > >> On Fri May 29 13:24:55 2020, ho-
chun.huang at noaa.gov
> > > > >>>> > > > >> wrote:
> > > > >>>> > > > >> > To Whom It May Concern:
> > > > >>>> > > > >> >
> > > > >>>> > > > >> > I have an LatLon 0.03x0.03 satellite observed
AOD
> > > > >>>> > > > >> > at
> > > > 20200519
> > > > >>>> > > > >> > 20Z
> > > > >>>> > > > >> > (global
> > > > >>>> > > > >> > but most with fill_value). I want to use
> > > > >>>> > > > >> > regrid_data_plane
> > > > >>>> to
> > > > >>>> > > > >> > regrid
> > > > >>>> > > > >> > it to
> > > > >>>> > > > >> > CMAQ Lambert Conformal Grid. But it seems to
> > > > >>>> > > > >> > hang
> > > > >>>> > > > >> > during
> > > > the
> > > > >>>> > > > >> > Interpolation
> > > > >>>> > > > >> > step.
> > > > >>>> > > > >> >
> > > > >>>> > > > >> > Can you find out what went wrong? Sometimes
it
> > > > >>>> > > > >> > is
> > > > >>>> > > > >> > more
> > > > than
> > > > >>>> an
> > > > >>>> > > > >> > hour
> > > > >>>> > > > >> > waiting before I kill it.
> > > > >>>> > > > >> >
> > > > >>>> > > > >> > runscript in
> > > > >>>> > > > >> > venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > > >>>> > > > >> > Chun.Huang/plot/viirs/
> > > > >>>> > > > >> >
> > > > >>>> > > > >> > with command > bash run.viirs_aqm_aod.sh viirs
> > > > >>>> > > > >> > aqm
> > > > 20200519
> > > > >>>> > > > >> > 20200519
> > > > >>>> > > > >
> > > > >>>> > > > >> > regrid.log 2>&1 &
> > > > >>>> > > > >> >
> > > > >>>> > > > >> > You can check the "regrid.log" in the script
> > > > >>>> > > > >> > directory for
> > > > >>>> > > > >> > input,
> > > > >>>> > > > >> > model_grid, and output file location on venus.
I
> > > > >>>> > > > >> > will keep
> > > > >>>> it
> > > > >>>> > > > >> > going
> > > > >>>> > > > >> > from May 29 19:15Z.
> > > > >>>> > > > >> >
> > > > >>>> > > > >> > One thing I do not understand is that the
output
> > > > >>>> > > > >> > file
> > > > >>>> > > > >> > is
> > > > >>>> already
> > > > >>>> > > > >> > generated
> > > > >>>> > > > >> > while the job is still
> > > > >>>> > > > >> > running. /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > >>>> > > > >> >
Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-
> > > > >>>> > > > >> > L3-
> > > > >>>> > > > >> > AOD_AQM_aqm_20200519_20.nc
> > > > >>>> > > > >> >
> > > > >>>> > > > >> > Ho-Chun Huang
> > > > >>>> > > > >> >
> > > > >>>> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > > >>>> > > > >> >
> > > > >>>> > > > >> > 5830 University Research Ct., Rm. 2792
> > > > >>>> > > > >> >
> > > > >>>> > > > >> > College Park, MD 20740
> > > > >>>> > > > >> >
> > > > >>>> > > > >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > >>>> > > > >> >
> > > > >>>> > > > >> > 301-683-3958
> > > > >>>> > > > >>
> > > > >>>> > > > >>
> > > > >>>> > > > >>
> > > > >>>> > > > >>
> > > > >>>> > > >
> > > > >>>> > > >
> > > > >>>> >
> > > > >>>> >
> > > > >>>> >
> > > > >>>> >
> > > > >>>>
> > > > >>>>
> > > >
> > > >
> >
> >
> >
> >
------------------------------------------------
Subject: hang on regrid_data_plane
From: Ho-Chun Huang - NOAA Affiliate
Time: Tue Jun 09 10:53:07 2020
Hi, Howard:
Thanks. I see, as long as there is a compression applied to the
netcdf
file, regrid_data_plane has to decompress each grid [for 6000 times].
C0
to C1 is a big reduction and I hope to be able to use compression
level 1
for VIIRS L3 AOD file.
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Tue, Jun 9, 2020 at 12:33 PM Howard Soh via RT <met_help at ucar.edu>
wrote:
> The compression is a variable based. NetCDF API reads a whole data,
> decompresses & slices. With a quick test, it seems to be no big
difference
> betwend compress level 1 and 2.
>
> File size (by nccopy)
> 8.1G VIIRS-L3-AOD_AQM_20200519_200000.c0.nc (no compression)
> 86M VIIRS-L3-AOD_AQM_20200519_200000.c1.nc (compress 1)
> 86M VIIRS-L3-AOD_AQM_20200519_200000.c2.nc (compress 2)
> 54M VIIRS-L3-AOD_AQM_20200519_200000.nc (compress 9)
>
> Wall time comparison with plot_data_plane (which reads 6000 times)
at
> dakota (time is not accurate because the system load is not
consistent):
>
> c0: 0m44.899s
> c1: 57m33.530s
> c2: 57m22.974s
>
> Cheers,
> Howard
>
> On Mon Jun 08 15:55:29 2020, ho-chun.huang at noaa.gov wrote:
> > Hi,
> >
> > I thought I asked the compress level=1 at the beginning. I will
> > double
> > check with NESIDS again, but can you tell me now at what nc
compress
> > level
> > that regrid_data_plane does not need to decompress the variable
for
> > each
> > grid. If it can resolved from the data source level, it won't
waste
> > your
> > time for the special case.
> >
> > Ho-Chun Huang
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct., Rm. 2792
> >
> > College Park, MD 20740
> >
> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >
> > 301-683-3958
> >
> >
> > On Mon, Jun 8, 2020 at 5:35 PM Howard Soh via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > The first cause is reading the variable 6000 times (dimension is
> > > 10000 by
> > > 6000). The variables are compressed with with level 9 (highest).
It
> > > taook
> > > more than 1 second to read & decompress a variable at dakota.
> > > if 1 seconds, 1 * 6000 = 6000 seconds = 100 minutes
> > > if 1.5 seconds, 1.5 * 6000 = 9000 seconds = 150 minutes.
> > >
> > > The first step is reading a variable one time instead of 6000
times.
> > > But
> > > the output did not match, so I'm working on making the same
result.
> > > Once
> > > this is done, I will check if there is more room to speed up
(some
> > > steps
> > > are executed 60M times).
> > >
> > > Cheers,
> > > Howard
> > >
> > > On Mon Jun 08 15:05:35 2020, johnhg wrote:
> > > > Howard and Ho-Chun,
> > > >
> > > > I'm wondering if we have any updates on this ticket? Howard,
have
> > > > you
> > > > been
> > > > able to find a reason for this running so slowly?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Jun 4, 2020 at 10:55 AM Ho-Chun Huang - NOAA Affiliate
via
> > > > RT
> > > > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411 >
> > > > >
> > > > > Hi, John and Howard:
> > > > >
> > > > > *Ho-Chun, do you know if the geometry of the VIIRS data
repeats
> > > > > day
> > > > > after
> > > > > day?*
> > > > >
> > > > >
> > > > > There are two satellites currently providing VIIRS AOD
> > > > > observations,
> > > > > SUOMI-NPP and NOAA-20. Eash satellite overpasses the same
area
> > > > > twice
> > > > > per
> > > > > day, AOD only observed during daytime overpass. The overpass
for
> > > > > local time
> > > > > should be similar per satellite, NOAA-20 is ~ 50 mins ahead
of
> > > > > SUOMI-
> > > > > NPP.
> > > > >
> > > > > Although I think daily granule pixel location per satellite
may
> > > > > shift
> > > > > slightly day after day (still need to verify this statement
with
> > > > > NESDIS
> > > > > personal)? Thus, for L3 gridded AOD data, *the area of
coverage*
> > > > > of
> > > > > orbital satellites should be similar *for the same hour of
day*.
> > > > > Maybe +-
> > > > > few grid boxes toward the north and west for predefined
search
> > > > > area.
> > > > > Thus,
> > > > > if you put a predefined search box with an adequate buffer
zone
> > > > > on 4
> > > > > sides
> > > > > for specific UTC, maybe it will work.
> > > > >
> > > > >
> > > > > Ho-Chun Huang
> > > > >
> > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > >
> > > > > 5830 University Research Ct., Rm. 2792
> > > > >
> > > > > College Park, MD 20740
> > > > >
> > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > >
> > > > > 301-683-3958
> > > > >
> > > > >
> > > > > On Thu, Jun 4, 2020 at 12:00 PM Ho-Chun Huang - NOAA
Affiliate <
> > > > > ho-chun.huang at noaa.gov> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Sorry for the wrong figure, attached is the direct read
from
> > > > > > NESDIS
> > > > > > VIIRS
> > > > > > L3 AOD file.
> > > > > >
> > > > > > Ho-Chun Huang
> > > > > >
> > > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > >
> > > > > > 5830 University Research Ct., Rm. 2792
> > > > > >
> > > > > > College Park, MD 20740
> > > > > >
> > > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > >
> > > > > > 301-683-3958
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 4, 2020 at 11:57 AM Ho-Chun Huang - NOAA
Affiliate
> > > > > > <
> > > > > > ho-chun.huang at noaa.gov> wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> The regrid_data_plane already obtained the LatLon
projection
> > > > > >> information
> > > > > >> from input obs file
> > > > > >>
> > > > > >> DEBUG 1: Reading data file:
> > > > > >>
> > > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > Chun.Huang/VIIRS_AOD/AOD/20200519/VIIRS-L3-
> > > > > AOD_AQM_20200519_200000.nc
> > > > > >> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
> > > > > >> created
> > > > > >> new
> > > > > >> Met2dDataFile object of type "FileType_NcMet".
> > > > > >> DEBUG 4:
> > > > > >> DEBUG 4: Latitude/Longitude Grid Data:
> > > > > >> DEBUG 4: lat_ll: -89.985
> > > > > >> DEBUG 4: lon_ll: 179.985
> > > > > >> DEBUG 4: delta_lat: 0.03
> > > > > >> DEBUG 4: delta_lon: 0.03
> > > > > >> DEBUG 4: Nlat: 6000
> > > > > >> DEBUG 4: Nlon: 12000
> > > > > >>
> > > > > >> Ho-Chun Huang
> > > > > >>
> > > > > >> IMSG at NOAA/NWS/NCEP/EMC
> > > > > >>
> > > > > >> 5830 University Research Ct., Rm. 2792
> > > > > >>
> > > > > >> College Park, MD 20740
> > > > > >>
> > > > > >> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > >>
> > > > > >> 301-683-3958
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Jun 4, 2020 at 11:52 AM Ho-Chun Huang - NOAA
Affiliate
> > > > > >> <
> > > > > >> ho-chun.huang at noaa.gov> wrote:
> > > > > >>
> > > > > >>> Hi, Howard and John:
> > > > > >>>
> > > > > >>> The VIIRS L3 AOD is on fixed LatLon 0.03x0.03 grid, so
its
> > > > > >>> location is
> > > > > >>> fixed everyday. Only the area of observed data will
change
> > > > > >>> for
> > > > > >>> each
> > > > > hours,
> > > > > >>> in general moving northward and westward.
> > > > > >>>
> > > > > >>> This is graphic based on direct read of the input VIIRS
L3
> > > > > >>> AOD
> > > > > >>> file,
> > > > > for
> > > > > >>> your reference,
> > > > > >>>
> > > > > >>> I want to emphasize again, the input VIIRS AOD is
gridded and
> > > > > >>> *NOT*
> > > > > >>> pixel data, you do not need to recompute the geo-
location for
> > > > > >>> each
> > > > > -field
> > > > > >>> variables.
> > > > > >>>
> > > > > >>> Ho-Chun Huang
> > > > > >>>
> > > > > >>> IMSG at NOAA/NWS/NCEP/EMC
> > > > > >>>
> > > > > >>> 5830 University Research Ct., Rm. 2792
> > > > > >>>
> > > > > >>> College Park, MD 20740
> > > > > >>>
> > > > > >>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > >>>
> > > > > >>> 301-683-3958
> > > > > >>>
> > > > > >>>
> > > > > >>> On Thu, Jun 4, 2020 at 11:39 AM John Halley Gotway via
RT <
> > > > > >>> met_help at ucar.edu> wrote:
> > > > > >>>
> > > > > >>>> Howard,
> > > > > >>>>
> > > > > >>>> I recall that with the GOES files, you set up
> > > > > >>>> regrid_data_plane
> > > > > >>>> to
> > > > > >>>> determine the grid mapping the first time it's run, and
then
> > > > > >>>> reuse
> > > > > that
> > > > > >>>> mapping information in subsequent calls. I really don't
know
> > > > > >>>> anything
> > > > > >>>> about
> > > > > >>>> the VIIRS data. Searching online, it looks like the
VIIRS
> > > > > >>>> data
> > > > > >>>> comes
> > > > > in
> > > > > >>>> swaths, whereas the GOES data is stationary. So perhaps
the
> > > > > >>>> approach
> > > > > of
> > > > > >>>> pre-computing the grid mappings won't work. I wonder if
the
> > > > > >>>> swaths are
> > > > > >>>> repeated though.
> > > > > >>>>
> > > > > >>>> Ho-Chun, do you know if the geometry of the VIIRS data
> > > > > >>>> repeats
> > > > > >>>> day
> > > > > after
> > > > > >>>> day? If we can figure out a way of computing the grid
> > > > > >>>> mapping
> > > > > >>>> information
> > > > > >>>> once ahead of time and then reuse that information, I'm
> > > > > >>>> guessing
> > > > > >>>> the
> > > > > >>>> tool
> > > > > >>>> would run much faster.
> > > > > >>>>
> > > > > >>>> Thanks,
> > > > > >>>> John
> > > > > >>>>
> > > > > >>>> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT
> > > > > >>>> <met_help at ucar.edu>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>> >
> > > > > >>>> > <URL:
> > > > > >>>> >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > >>>> > >
> > > > > >>>> >
> > > > > >>>> > I will investigate why it takes too long.
> > > > > >>>> >
> > > > > >>>> > I ran regrid_data_plane for just one variable
> > > > > >>>> > (AOD_H_Quality,
> > > > > >>>> dimension is
> > > > > >>>> > 6000 by 12000) and it took about 4 hours at dakota.
> > > > > >>>> >
> > > > > >>>> > DEBUG 1: Writing output file:
> > > > > >>>> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
> > > > > >>>> >
> > > > > >>>> > real 239m2.552s
> > > > > >>>> > user 206m42.332s
> > > > > >>>> > sys 32m18.028s
> > > > > >>>> >
> > > > > >>>> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does not have
the
> > > > > >>>> > time
> > > > > >>>> variable
> > > > > >>>> > "t". So point2grid does not handle this case with
> > > > > >>>> > workaround.
> > > > > >>>> >
> > > > > >>>> > Cheers,
> > > > > >>>> > Howard
> > > > > >>>> >
> > > > > >>>> > On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov
wrote:
> > > > > >>>> > > Hi, John and Howard:
> > > > > >>>> > >
> > > > > >>>> > > (1) You should be able to see my MET version in the
run
> > > > > >>>> > > script :
> > > > > 9.0
> > > > > >>>> > >
> > > > > >>>> > > (2) Background information, we used to discuss
between
> > > > > >>>> > > NCAR
> > > > > >>>> > > MET,
> > > > > >>>> > > NESDIS,
> > > > > >>>> > > and EMC about using VIIRS pixel AOD data per
granule and
> > > > > >>>> > > mapping
> > > > > >>>> pixel
> > > > > >>>> > > observed AOD into a selected model grid. It has
been
> > > > > >>>> > > determined
> > > > > >>>> that
> > > > > >>>> > > the
> > > > > >>>> > > data volume is too large for MET to handle VIIRS
granule
> > > > > >>>> > > pixel
> > > > > data
> > > > > >>>> > > ingestion. EMC has asked NESDIS aerosol groups to
> > > > > >>>> > > provide
> > > > > >>>> > > *L3
> > > > > >>>> > > "gridded"
> > > > > >>>> > > nefCDF* files for global and regional AQM model
> > > > > >>>> > > evaluation/verification.
> > > > > >>>> > > I am testing the regional one VIIRS-L3-AOD_*AQM*_
> > > > > 20200519_200000.nc
> > > > > >>>> ,
> > > > > >>>> > > while Partha Bhattacharjee is testing VIIRS-L3-
> > > > > >>>> > > AOD_*GEFS*_20200519_120000.nc
> > > > > >>>> > > for global GEFS-aerosol verification [reference
helpdesk
> > > > > >>>> > > ticket
> > > > > >>>> > > 95358].
> > > > > >>>> > >
> > > > > >>>> > > (3) *I am not trying to using point2grid*, I am
trying
> > > > > >>>> > > to
> > > > > >>>> > > use MET
> > > > > >>>> > > regridding utilities to map a *gridded* netCDF to a
> > > > > >>>> > > target
> > > > > >>>> > > model
> > > > > >>>> > > *gridded*
> > > > > >>>> > > netCDF. Unless there is another better tool in
MET
> > > > > >>>> > > packages
> > > > > that I
> > > > > >>>> > > am not
> > > > > >>>> > > aware of, I assume regrid_data_plane is for general
> > > > > >>>> > > mapping
> > > > > purpose.
> > > > > >>>> > >
> > > > > >>>> > > (4) For each hourly period, orbital satellites
> > > > > >>>> > > overpassed
> > > > > >>>> > > only a
> > > > > >>>> small
> > > > > >>>> > > area
> > > > > >>>> > > around the globe. I have asked NESDIS aerosol to
use
> > > > > >>>> > > fill-
> > > > > >>>> > > values
> > > > > for
> > > > > >>>> > > non-observed grids and with compress level=1 hoping
> > > > > >>>> regard-data_plane
> > > > > >>>> > > can
> > > > > >>>> > > skip those "-9999.f" grid quickly. I may be wrong.
In
> > > > > >>>> > > reality, I
> > > > > >>>> may
> > > > > >>>> > > only
> > > > > >>>> > > use files from 18Z to 23Z for verification on
CONUS,
> > > > > >>>> > > Alaska,
> > > > > >>>> > > and
> > > > > >>>> > > Hawaii.
> > > > > >>>> > > The reason to keep it global coverage is to
simplify
> > > > > >>>> > > NESDIS
> > > > > >>>> production
> > > > > >>>> > > processes and be flexible for future applications.
> > > > > >>>> > > Please
> > > > > >>>> > > study my
> > > > > >>>> > > problem,
> > > > > >>>> > > if changes on the data production side [NESDIS
aerosol
> > > > > >>>> > > group] can
> > > > > >>>> > > speed-up
> > > > > >>>> > > the regridding processes, please let me know.
> > > > > >>>> > >
> > > > > >>>> > > Thank you for your help.
> > > > > >>>> > >
> > > > > >>>> > > Ho-Chun Huang
> > > > > >>>> > >
> > > > > >>>> > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > >>>> > >
> > > > > >>>> > > 5830 University Research Ct., Rm. 2792
> > > > > >>>> > >
> > > > > >>>> > > College Park, MD 20740
> > > > > >>>> > >
> > > > > >>>> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > >>>> > >
> > > > > >>>> > > 301-683-3958
> > > > > >>>> > >
> > > > > >>>> > >
> > > > > >>>> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway
via RT
> > > > > >>>> > > <met_help at ucar.edu>
> > > > > >>>> > > wrote:
> > > > > >>>> > >
> > > > > >>>> > > > Howard,
> > > > > >>>> > > >
> > > > > >>>> > > > OK, please find input data for testing here:
> > > > > >>>> > > >
> > > > > >>>> > > >
> > > > > >>>> >
> > > > > >>>>
> > > > >
> > >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> > > > > >>>> > > >
> > > > > >>>> > > > When you untar this file, you'll find the
contents of
> > > > > >>>> > > > Ho-
> > > > > >>>> > > > Chun's
> > > > > >>>> > > > directory:
> > > > > >>>> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > > > >>>> > > > Chun.Huang/plot/viirs
> > > > > >>>> > > >
> > > > > >>>> > > > In addition, I copied the input/output
directories
> > > > > >>>> > > > from
> > > > > >>>> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > >>>> > > > Chun.Huang/VIIRS_AOD
> > > > > >>>> > > > into
> > > > > that
> > > > > >>>> > > > viirs
> > > > > >>>> > > > directory. Also, I included the file named
> > > > > >>>> > > > "aqm.aot.148.grid"
> > > > > >>>> which
> > > > > >>>> > > > defines
> > > > > >>>> > > > the target grid.
> > > > > >>>> > > >
> > > > > >>>> > > > If you're able to take a look at this data and
see how
> > > > > >>>> > > > it
> > > > > >>>> > > > runs,
> > > > > >>>> > > > that'd be
> > > > > >>>> > > > great! Ho-Chun found that it took over 12 hours!?
to
> > > > > >>>> > > > run
> > > > > >>>> > > > on
> > > > > WCOSS.
> > > > > >>>> > > >
> > > > > >>>> > > > Thanks,
> > > > > >>>> > > > John
> > > > > >>>> > > >
> > > > > >>>> > > > On Wed, Jun 3, 2020 at 4:45 PM John Halley Gotway
<
> > > > > >>>> johnhg at ucar.edu>
> > > > > >>>> > > > wrote:
> > > > > >>>> > > >
> > > > > >>>> > > > > Howard,
> > > > > >>>> > > > >
> > > > > >>>> > > > > I'll go grab this data and copy it over to
dakota.
> > > > > >>>> > > > > Ho-
> > > > > >>>> > > > > Chun,
> > > > > >>>> sorry
> > > > > >>>> > > > > for the
> > > > > >>>> > > > > delay in getting started with this. I'll go
grab it
> > > > > >>>> > > > > now.
> > > > > >>>> > > > >
> > > > > >>>> > > > > John
> > > > > >>>> > > > >
> > > > > >>>> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via
RT
> > > > > >>>> > > > > <met_help at ucar.edu>
> > > > > >>>> > > > > wrote:
> > > > > >>>> > > > >
> > > > > >>>> > > > >>
> > > > > >>>> > > > >> <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > >>>> >
> > > > > >>>> > > > >>
> > > > > >>>> > > > >> I don't have access to venus.
> > > > > >>>> > > > >> Would you upload the files (config, script,
log,
> > > > > >>>> > > > >> and
> > > > > >>>> > > > >> data
> > > > > >>>> files)
> > > > > >>>> > > > >> to MET
> > > > > >>>> > > > >> ftp server?
> > > > > >>>> > > > >>
> > > > > >>>> > > > >> Which MET version are you using?
> > > > > >>>> > > > >>
> > > > > >>>> > > > >> The GOES-AOD data was supported by
> > > > > >>>> > > > >> regrid_data_plane
> > > > > >>>> > > > >> (V8.x).
> > > > > >>>> It's
> > > > > >>>> > > > >> moved
> > > > > >>>> > > > >> to point2grid (V9.0). The regrid_data_plane
still
> > > > > >>>> > > > >> processes
> > > > > >>>> GOES-
> > > > > >>>> > > > >> AOD
> > > > > >>>> > > > data
> > > > > >>>> > > > >> in different way (interpolated based on the
> > > > > >>>> > > > >> neighbor
> > > > > >>>> > > > >> data by
> > > > > >>>> using
> > > > > >>>> > > > >> width
> > > > > >>>> > > > >> and shape, not by the physical location). The
> > > > > >>>> > > > >> output is
> > > > > >>>> different
> > > > > >>>> > > > >> from
> > > > > >>>> > > > >> point2grid. VIIRS AOD is not supported by MET,
yet.
> > > > > >>>> > > > >> There
> > > > > >>>> might be
> > > > > >>>> > > > >> a
> > > > > >>>> > > > >> workaround.
> > > > > >>>> > > > >>
> > > > > >>>> > > > >> Cheers,
> > > > > >>>> > > > >> Howard
> > > > > >>>> > > > >>
> > > > > >>>> > > > >> On Fri May 29 13:24:55 2020, ho-
chun.huang at noaa.gov
> > > > > >>>> > > > >> wrote:
> > > > > >>>> > > > >> > To Whom It May Concern:
> > > > > >>>> > > > >> >
> > > > > >>>> > > > >> > I have an LatLon 0.03x0.03 satellite
observed AOD
> > > > > >>>> > > > >> > at
> > > > > 20200519
> > > > > >>>> > > > >> > 20Z
> > > > > >>>> > > > >> > (global
> > > > > >>>> > > > >> > but most with fill_value). I want to use
> > > > > >>>> > > > >> > regrid_data_plane
> > > > > >>>> to
> > > > > >>>> > > > >> > regrid
> > > > > >>>> > > > >> > it to
> > > > > >>>> > > > >> > CMAQ Lambert Conformal Grid. But it seems
to
> > > > > >>>> > > > >> > hang
> > > > > >>>> > > > >> > during
> > > > > the
> > > > > >>>> > > > >> > Interpolation
> > > > > >>>> > > > >> > step.
> > > > > >>>> > > > >> >
> > > > > >>>> > > > >> > Can you find out what went wrong? Sometimes
it
> > > > > >>>> > > > >> > is
> > > > > >>>> > > > >> > more
> > > > > than
> > > > > >>>> an
> > > > > >>>> > > > >> > hour
> > > > > >>>> > > > >> > waiting before I kill it.
> > > > > >>>> > > > >> >
> > > > > >>>> > > > >> > runscript in
> > > > > >>>> > > > >> > venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > > > >>>> > > > >> > Chun.Huang/plot/viirs/
> > > > > >>>> > > > >> >
> > > > > >>>> > > > >> > with command > bash run.viirs_aqm_aod.sh
viirs
> > > > > >>>> > > > >> > aqm
> > > > > 20200519
> > > > > >>>> > > > >> > 20200519
> > > > > >>>> > > > >
> > > > > >>>> > > > >> > regrid.log 2>&1 &
> > > > > >>>> > > > >> >
> > > > > >>>> > > > >> > You can check the "regrid.log" in the script
> > > > > >>>> > > > >> > directory for
> > > > > >>>> > > > >> > input,
> > > > > >>>> > > > >> > model_grid, and output file location on
venus. I
> > > > > >>>> > > > >> > will keep
> > > > > >>>> it
> > > > > >>>> > > > >> > going
> > > > > >>>> > > > >> > from May 29 19:15Z.
> > > > > >>>> > > > >> >
> > > > > >>>> > > > >> > One thing I do not understand is that the
output
> > > > > >>>> > > > >> > file
> > > > > >>>> > > > >> > is
> > > > > >>>> already
> > > > > >>>> > > > >> > generated
> > > > > >>>> > > > >> > while the job is still
> > > > > >>>> > > > >> > running.
/gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > >>>> > > > >> >
Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-
> > > > > >>>> > > > >> > L3-
> > > > > >>>> > > > >> > AOD_AQM_aqm_20200519_20.nc
> > > > > >>>> > > > >> >
> > > > > >>>> > > > >> > Ho-Chun Huang
> > > > > >>>> > > > >> >
> > > > > >>>> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > > > >>>> > > > >> >
> > > > > >>>> > > > >> > 5830 University Research Ct., Rm. 2792
> > > > > >>>> > > > >> >
> > > > > >>>> > > > >> > College Park, MD 20740
> > > > > >>>> > > > >> >
> > > > > >>>> > > > >> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > >>>> > > > >> >
> > > > > >>>> > > > >> > 301-683-3958
> > > > > >>>> > > > >>
> > > > > >>>> > > > >>
> > > > > >>>> > > > >>
> > > > > >>>> > > > >>
> > > > > >>>> > > >
> > > > > >>>> > > >
> > > > > >>>> >
> > > > > >>>> >
> > > > > >>>> >
> > > > > >>>> >
> > > > > >>>>
> > > > > >>>>
> > > > >
> > > > >
> > >
> > >
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: hang on regrid_data_plane
From: Howard Soh
Time: Wed Jun 10 12:12:06 2020
The issue at github was created
(https://github.com/NCAR/MET/issues/1363).
The main saving is from reading the compressed variable. The other
saving is from building a data plane (72M calls to 120K calls). There
is no significant difference with compression level (between 1 and 9)
with the enhanced MET.
Cheers,
Howard
Cheers,
Howard
On Tue Jun 09 10:53:07 2020, ho-chun.huang at noaa.gov wrote:
> Hi, Howard:
>
> Thanks. I see, as long as there is a compression applied to the
> netcdf
> file, regrid_data_plane has to decompress each grid [for 6000
times].
> C0
> to C1 is a big reduction and I hope to be able to use compression
> level 1
> for VIIRS L3 AOD file.
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Tue, Jun 9, 2020 at 12:33 PM Howard Soh via RT
<met_help at ucar.edu>
> wrote:
>
> > The compression is a variable based. NetCDF API reads a whole
data,
> > decompresses & slices. With a quick test, it seems to be no big
> > difference
> > betwend compress level 1 and 2.
> >
> > File size (by nccopy)
> > 8.1G VIIRS-L3-AOD_AQM_20200519_200000.c0.nc (no compression)
> > 86M VIIRS-L3-AOD_AQM_20200519_200000.c1.nc (compress 1)
> > 86M VIIRS-L3-AOD_AQM_20200519_200000.c2.nc (compress 2)
> > 54M VIIRS-L3-AOD_AQM_20200519_200000.nc (compress 9)
> >
> > Wall time comparison with plot_data_plane (which reads 6000 times)
at
> > dakota (time is not accurate because the system load is not
> > consistent):
> >
> > c0: 0m44.899s
> > c1: 57m33.530s
> > c2: 57m22.974s
> >
> > Cheers,
> > Howard
> >
> > On Mon Jun 08 15:55:29 2020, ho-chun.huang at noaa.gov wrote:
> > > Hi,
> > >
> > > I thought I asked the compress level=1 at the beginning. I will
> > > double
> > > check with NESIDS again, but can you tell me now at what nc
> > > compress
> > > level
> > > that regrid_data_plane does not need to decompress the variable
for
> > > each
> > > grid. If it can resolved from the data source level, it won't
> > > waste
> > > your
> > > time for the special case.
> > >
> > > Ho-Chun Huang
> > >
> > > IMSG at NOAA/NWS/NCEP/EMC
> > >
> > > 5830 University Research Ct., Rm. 2792
> > >
> > > College Park, MD 20740
> > >
> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > >
> > > 301-683-3958
> > >
> > >
> > > On Mon, Jun 8, 2020 at 5:35 PM Howard Soh via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > > The first cause is reading the variable 6000 times (dimension
is
> > > > 10000 by
> > > > 6000). The variables are compressed with with level 9
(highest).
> > > > It
> > > > taook
> > > > more than 1 second to read & decompress a variable at dakota.
> > > > if 1 seconds, 1 * 6000 = 6000 seconds = 100 minutes
> > > > if 1.5 seconds, 1.5 * 6000 = 9000 seconds = 150 minutes.
> > > >
> > > > The first step is reading a variable one time instead of 6000
> > > > times.
> > > > But
> > > > the output did not match, so I'm working on making the same
> > > > result.
> > > > Once
> > > > this is done, I will check if there is more room to speed up
> > > > (some
> > > > steps
> > > > are executed 60M times).
> > > >
> > > > Cheers,
> > > > Howard
> > > >
> > > > On Mon Jun 08 15:05:35 2020, johnhg wrote:
> > > > > Howard and Ho-Chun,
> > > > >
> > > > > I'm wondering if we have any updates on this ticket? Howard,
> > > > > have
> > > > > you
> > > > > been
> > > > > able to find a reason for this running so slowly?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Thu, Jun 4, 2020 at 10:55 AM Ho-Chun Huang - NOAA
Affiliate
> > > > > via
> > > > > RT
> > > > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > > >
> > > > > >
> > > > > > Hi, John and Howard:
> > > > > >
> > > > > > *Ho-Chun, do you know if the geometry of the VIIRS data
> > > > > > repeats
> > > > > > day
> > > > > > after
> > > > > > day?*
> > > > > >
> > > > > >
> > > > > > There are two satellites currently providing VIIRS AOD
> > > > > > observations,
> > > > > > SUOMI-NPP and NOAA-20. Eash satellite overpasses the same
> > > > > > area
> > > > > > twice
> > > > > > per
> > > > > > day, AOD only observed during daytime overpass. The
overpass
> > > > > > for
> > > > > > local time
> > > > > > should be similar per satellite, NOAA-20 is ~ 50 mins
ahead
> > > > > > of
> > > > > > SUOMI-
> > > > > > NPP.
> > > > > >
> > > > > > Although I think daily granule pixel location per
satellite
> > > > > > may
> > > > > > shift
> > > > > > slightly day after day (still need to verify this
statement
> > > > > > with
> > > > > > NESDIS
> > > > > > personal)? Thus, for L3 gridded AOD data, *the area of
> > > > > > coverage*
> > > > > > of
> > > > > > orbital satellites should be similar *for the same hour of
> > > > > > day*.
> > > > > > Maybe +-
> > > > > > few grid boxes toward the north and west for predefined
> > > > > > search
> > > > > > area.
> > > > > > Thus,
> > > > > > if you put a predefined search box with an adequate buffer
> > > > > > zone
> > > > > > on 4
> > > > > > sides
> > > > > > for specific UTC, maybe it will work.
> > > > > >
> > > > > >
> > > > > > Ho-Chun Huang
> > > > > >
> > > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > >
> > > > > > 5830 University Research Ct., Rm. 2792
> > > > > >
> > > > > > College Park, MD 20740
> > > > > >
> > > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > >
> > > > > > 301-683-3958
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 4, 2020 at 12:00 PM Ho-Chun Huang - NOAA
> > > > > > Affiliate <
> > > > > > ho-chun.huang at noaa.gov> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Sorry for the wrong figure, attached is the direct read
> > > > > > > from
> > > > > > > NESDIS
> > > > > > > VIIRS
> > > > > > > L3 AOD file.
> > > > > > >
> > > > > > > Ho-Chun Huang
> > > > > > >
> > > > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > >
> > > > > > > 5830 University Research Ct., Rm. 2792
> > > > > > >
> > > > > > > College Park, MD 20740
> > > > > > >
> > > > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > >
> > > > > > > 301-683-3958
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 4, 2020 at 11:57 AM Ho-Chun Huang - NOAA
> > > > > > > Affiliate
> > > > > > > <
> > > > > > > ho-chun.huang at noaa.gov> wrote:
> > > > > > >
> > > > > > >> Hi,
> > > > > > >>
> > > > > > >> The regrid_data_plane already obtained the LatLon
> > > > > > >> projection
> > > > > > >> information
> > > > > > >> from input obs file
> > > > > > >>
> > > > > > >> DEBUG 1: Reading data file:
> > > > > > >>
> > > > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > > Chun.Huang/VIIRS_AOD/AOD/20200519/VIIRS-L3-
> > > > > > AOD_AQM_20200519_200000.nc
> > > > > > >> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file()
->
> > > > > > >> created
> > > > > > >> new
> > > > > > >> Met2dDataFile object of type "FileType_NcMet".
> > > > > > >> DEBUG 4:
> > > > > > >> DEBUG 4: Latitude/Longitude Grid Data:
> > > > > > >> DEBUG 4: lat_ll: -89.985
> > > > > > >> DEBUG 4: lon_ll: 179.985
> > > > > > >> DEBUG 4: delta_lat: 0.03
> > > > > > >> DEBUG 4: delta_lon: 0.03
> > > > > > >> DEBUG 4: Nlat: 6000
> > > > > > >> DEBUG 4: Nlon: 12000
> > > > > > >>
> > > > > > >> Ho-Chun Huang
> > > > > > >>
> > > > > > >> IMSG at NOAA/NWS/NCEP/EMC
> > > > > > >>
> > > > > > >> 5830 University Research Ct., Rm. 2792
> > > > > > >>
> > > > > > >> College Park, MD 20740
> > > > > > >>
> > > > > > >> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > >>
> > > > > > >> 301-683-3958
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, Jun 4, 2020 at 11:52 AM Ho-Chun Huang - NOAA
> > > > > > >> Affiliate
> > > > > > >> <
> > > > > > >> ho-chun.huang at noaa.gov> wrote:
> > > > > > >>
> > > > > > >>> Hi, Howard and John:
> > > > > > >>>
> > > > > > >>> The VIIRS L3 AOD is on fixed LatLon 0.03x0.03 grid, so
> > > > > > >>> its
> > > > > > >>> location is
> > > > > > >>> fixed everyday. Only the area of observed data will
> > > > > > >>> change
> > > > > > >>> for
> > > > > > >>> each
> > > > > > hours,
> > > > > > >>> in general moving northward and westward.
> > > > > > >>>
> > > > > > >>> This is graphic based on direct read of the input
VIIRS
> > > > > > >>> L3
> > > > > > >>> AOD
> > > > > > >>> file,
> > > > > > for
> > > > > > >>> your reference,
> > > > > > >>>
> > > > > > >>> I want to emphasize again, the input VIIRS AOD is
gridded
> > > > > > >>> and
> > > > > > >>> *NOT*
> > > > > > >>> pixel data, you do not need to recompute the geo-
location
> > > > > > >>> for
> > > > > > >>> each
> > > > > > -field
> > > > > > >>> variables.
> > > > > > >>>
> > > > > > >>> Ho-Chun Huang
> > > > > > >>>
> > > > > > >>> IMSG at NOAA/NWS/NCEP/EMC
> > > > > > >>>
> > > > > > >>> 5830 University Research Ct., Rm. 2792
> > > > > > >>>
> > > > > > >>> College Park, MD 20740
> > > > > > >>>
> > > > > > >>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > >>>
> > > > > > >>> 301-683-3958
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Thu, Jun 4, 2020 at 11:39 AM John Halley Gotway via
RT
> > > > > > >>> <
> > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > >>>
> > > > > > >>>> Howard,
> > > > > > >>>>
> > > > > > >>>> I recall that with the GOES files, you set up
> > > > > > >>>> regrid_data_plane
> > > > > > >>>> to
> > > > > > >>>> determine the grid mapping the first time it's run,
and
> > > > > > >>>> then
> > > > > > >>>> reuse
> > > > > > that
> > > > > > >>>> mapping information in subsequent calls. I really
don't
> > > > > > >>>> know
> > > > > > >>>> anything
> > > > > > >>>> about
> > > > > > >>>> the VIIRS data. Searching online, it looks like the
> > > > > > >>>> VIIRS
> > > > > > >>>> data
> > > > > > >>>> comes
> > > > > > in
> > > > > > >>>> swaths, whereas the GOES data is stationary. So
perhaps
> > > > > > >>>> the
> > > > > > >>>> approach
> > > > > > of
> > > > > > >>>> pre-computing the grid mappings won't work. I wonder
if
> > > > > > >>>> the
> > > > > > >>>> swaths are
> > > > > > >>>> repeated though.
> > > > > > >>>>
> > > > > > >>>> Ho-Chun, do you know if the geometry of the VIIRS
data
> > > > > > >>>> repeats
> > > > > > >>>> day
> > > > > > after
> > > > > > >>>> day? If we can figure out a way of computing the grid
> > > > > > >>>> mapping
> > > > > > >>>> information
> > > > > > >>>> once ahead of time and then reuse that information,
I'm
> > > > > > >>>> guessing
> > > > > > >>>> the
> > > > > > >>>> tool
> > > > > > >>>> would run much faster.
> > > > > > >>>>
> > > > > > >>>> Thanks,
> > > > > > >>>> John
> > > > > > >>>>
> > > > > > >>>> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT
> > > > > > >>>> <met_help at ucar.edu>
> > > > > > >>>> wrote:
> > > > > > >>>>
> > > > > > >>>> >
> > > > > > >>>> > <URL:
> > > > > > >>>> >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > > >>>> > >
> > > > > > >>>> >
> > > > > > >>>> > I will investigate why it takes too long.
> > > > > > >>>> >
> > > > > > >>>> > I ran regrid_data_plane for just one variable
> > > > > > >>>> > (AOD_H_Quality,
> > > > > > >>>> dimension is
> > > > > > >>>> > 6000 by 12000) and it took about 4 hours at dakota.
> > > > > > >>>> >
> > > > > > >>>> > DEBUG 1: Writing output file:
> > > > > > >>>> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
> > > > > > >>>> >
> > > > > > >>>> > real 239m2.552s
> > > > > > >>>> > user 206m42.332s
> > > > > > >>>> > sys 32m18.028s
> > > > > > >>>> >
> > > > > > >>>> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does not
have
> > > > > > >>>> > the
> > > > > > >>>> > time
> > > > > > >>>> variable
> > > > > > >>>> > "t". So point2grid does not handle this case with
> > > > > > >>>> > workaround.
> > > > > > >>>> >
> > > > > > >>>> > Cheers,
> > > > > > >>>> > Howard
> > > > > > >>>> >
> > > > > > >>>> > On Thu Jun 04 06:53:39 2020, ho-chun.huang at noaa.gov
> > > > > > >>>> > wrote:
> > > > > > >>>> > > Hi, John and Howard:
> > > > > > >>>> > >
> > > > > > >>>> > > (1) You should be able to see my MET version in
the
> > > > > > >>>> > > run
> > > > > > >>>> > > script :
> > > > > > 9.0
> > > > > > >>>> > >
> > > > > > >>>> > > (2) Background information, we used to discuss
> > > > > > >>>> > > between
> > > > > > >>>> > > NCAR
> > > > > > >>>> > > MET,
> > > > > > >>>> > > NESDIS,
> > > > > > >>>> > > and EMC about using VIIRS pixel AOD data per
granule
> > > > > > >>>> > > and
> > > > > > >>>> > > mapping
> > > > > > >>>> pixel
> > > > > > >>>> > > observed AOD into a selected model grid. It has
> > > > > > >>>> > > been
> > > > > > >>>> > > determined
> > > > > > >>>> that
> > > > > > >>>> > > the
> > > > > > >>>> > > data volume is too large for MET to handle VIIRS
> > > > > > >>>> > > granule
> > > > > > >>>> > > pixel
> > > > > > data
> > > > > > >>>> > > ingestion. EMC has asked NESDIS aerosol groups
to
> > > > > > >>>> > > provide
> > > > > > >>>> > > *L3
> > > > > > >>>> > > "gridded"
> > > > > > >>>> > > nefCDF* files for global and regional AQM model
> > > > > > >>>> > > evaluation/verification.
> > > > > > >>>> > > I am testing the regional one VIIRS-L3-AOD_*AQM*_
> > > > > > 20200519_200000.nc
> > > > > > >>>> ,
> > > > > > >>>> > > while Partha Bhattacharjee is testing VIIRS-L3-
> > > > > > >>>> > > AOD_*GEFS*_20200519_120000.nc
> > > > > > >>>> > > for global GEFS-aerosol verification [reference
> > > > > > >>>> > > helpdesk
> > > > > > >>>> > > ticket
> > > > > > >>>> > > 95358].
> > > > > > >>>> > >
> > > > > > >>>> > > (3) *I am not trying to using point2grid*, I am
> > > > > > >>>> > > trying
> > > > > > >>>> > > to
> > > > > > >>>> > > use MET
> > > > > > >>>> > > regridding utilities to map a *gridded* netCDF to
a
> > > > > > >>>> > > target
> > > > > > >>>> > > model
> > > > > > >>>> > > *gridded*
> > > > > > >>>> > > netCDF. Unless there is another better tool in
MET
> > > > > > >>>> > > packages
> > > > > > that I
> > > > > > >>>> > > am not
> > > > > > >>>> > > aware of, I assume regrid_data_plane is for
general
> > > > > > >>>> > > mapping
> > > > > > purpose.
> > > > > > >>>> > >
> > > > > > >>>> > > (4) For each hourly period, orbital satellites
> > > > > > >>>> > > overpassed
> > > > > > >>>> > > only a
> > > > > > >>>> small
> > > > > > >>>> > > area
> > > > > > >>>> > > around the globe. I have asked NESDIS aerosol to
use
> > > > > > >>>> > > fill-
> > > > > > >>>> > > values
> > > > > > for
> > > > > > >>>> > > non-observed grids and with compress level=1
hoping
> > > > > > >>>> regard-data_plane
> > > > > > >>>> > > can
> > > > > > >>>> > > skip those "-9999.f" grid quickly. I may be
wrong.
> > > > > > >>>> > > In
> > > > > > >>>> > > reality, I
> > > > > > >>>> may
> > > > > > >>>> > > only
> > > > > > >>>> > > use files from 18Z to 23Z for verification on
CONUS,
> > > > > > >>>> > > Alaska,
> > > > > > >>>> > > and
> > > > > > >>>> > > Hawaii.
> > > > > > >>>> > > The reason to keep it global coverage is to
simplify
> > > > > > >>>> > > NESDIS
> > > > > > >>>> production
> > > > > > >>>> > > processes and be flexible for future
applications.
> > > > > > >>>> > > Please
> > > > > > >>>> > > study my
> > > > > > >>>> > > problem,
> > > > > > >>>> > > if changes on the data production side [NESDIS
> > > > > > >>>> > > aerosol
> > > > > > >>>> > > group] can
> > > > > > >>>> > > speed-up
> > > > > > >>>> > > the regridding processes, please let me know.
> > > > > > >>>> > >
> > > > > > >>>> > > Thank you for your help.
> > > > > > >>>> > >
> > > > > > >>>> > > Ho-Chun Huang
> > > > > > >>>> > >
> > > > > > >>>> > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > >>>> > >
> > > > > > >>>> > > 5830 University Research Ct., Rm. 2792
> > > > > > >>>> > >
> > > > > > >>>> > > College Park, MD 20740
> > > > > > >>>> > >
> > > > > > >>>> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > >>>> > >
> > > > > > >>>> > > 301-683-3958
> > > > > > >>>> > >
> > > > > > >>>> > >
> > > > > > >>>> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley Gotway
> > > > > > >>>> > > via RT
> > > > > > >>>> > > <met_help at ucar.edu>
> > > > > > >>>> > > wrote:
> > > > > > >>>> > >
> > > > > > >>>> > > > Howard,
> > > > > > >>>> > > >
> > > > > > >>>> > > > OK, please find input data for testing here:
> > > > > > >>>> > > >
> > > > > > >>>> > > >
> > > > > > >>>> >
> > > > > > >>>>
> > > > > >
> > > >
> >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> > > > > > >>>> > > >
> > > > > > >>>> > > > When you untar this file, you'll find the
contents
> > > > > > >>>> > > > of
> > > > > > >>>> > > > Ho-
> > > > > > >>>> > > > Chun's
> > > > > > >>>> > > > directory:
> > > > > > >>>> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > > > > >>>> > > > Chun.Huang/plot/viirs
> > > > > > >>>> > > >
> > > > > > >>>> > > > In addition, I copied the input/output
directories
> > > > > > >>>> > > > from
> > > > > > >>>> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > > >>>> > > > Chun.Huang/VIIRS_AOD
> > > > > > >>>> > > > into
> > > > > > that
> > > > > > >>>> > > > viirs
> > > > > > >>>> > > > directory. Also, I included the file named
> > > > > > >>>> > > > "aqm.aot.148.grid"
> > > > > > >>>> which
> > > > > > >>>> > > > defines
> > > > > > >>>> > > > the target grid.
> > > > > > >>>> > > >
> > > > > > >>>> > > > If you're able to take a look at this data and
see
> > > > > > >>>> > > > how
> > > > > > >>>> > > > it
> > > > > > >>>> > > > runs,
> > > > > > >>>> > > > that'd be
> > > > > > >>>> > > > great! Ho-Chun found that it took over 12
hours!?
> > > > > > >>>> > > > to
> > > > > > >>>> > > > run
> > > > > > >>>> > > > on
> > > > > > WCOSS.
> > > > > > >>>> > > >
> > > > > > >>>> > > > Thanks,
> > > > > > >>>> > > > John
> > > > > > >>>> > > >
> > > > > > >>>> > > > On Wed, Jun 3, 2020 at 4:45 PM John Halley
Gotway
> > > > > > >>>> > > > <
> > > > > > >>>> johnhg at ucar.edu>
> > > > > > >>>> > > > wrote:
> > > > > > >>>> > > >
> > > > > > >>>> > > > > Howard,
> > > > > > >>>> > > > >
> > > > > > >>>> > > > > I'll go grab this data and copy it over to
> > > > > > >>>> > > > > dakota.
> > > > > > >>>> > > > > Ho-
> > > > > > >>>> > > > > Chun,
> > > > > > >>>> sorry
> > > > > > >>>> > > > > for the
> > > > > > >>>> > > > > delay in getting started with this. I'll go
grab
> > > > > > >>>> > > > > it
> > > > > > >>>> > > > > now.
> > > > > > >>>> > > > >
> > > > > > >>>> > > > > John
> > > > > > >>>> > > > >
> > > > > > >>>> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh via
RT
> > > > > > >>>> > > > > <met_help at ucar.edu>
> > > > > > >>>> > > > > wrote:
> > > > > > >>>> > > > >
> > > > > > >>>> > > > >>
> > > > > > >>>> > > > >> <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > > >>>> >
> > > > > > >>>> > > > >>
> > > > > > >>>> > > > >> I don't have access to venus.
> > > > > > >>>> > > > >> Would you upload the files (config, script,
> > > > > > >>>> > > > >> log,
> > > > > > >>>> > > > >> and
> > > > > > >>>> > > > >> data
> > > > > > >>>> files)
> > > > > > >>>> > > > >> to MET
> > > > > > >>>> > > > >> ftp server?
> > > > > > >>>> > > > >>
> > > > > > >>>> > > > >> Which MET version are you using?
> > > > > > >>>> > > > >>
> > > > > > >>>> > > > >> The GOES-AOD data was supported by
> > > > > > >>>> > > > >> regrid_data_plane
> > > > > > >>>> > > > >> (V8.x).
> > > > > > >>>> It's
> > > > > > >>>> > > > >> moved
> > > > > > >>>> > > > >> to point2grid (V9.0). The regrid_data_plane
> > > > > > >>>> > > > >> still
> > > > > > >>>> > > > >> processes
> > > > > > >>>> GOES-
> > > > > > >>>> > > > >> AOD
> > > > > > >>>> > > > data
> > > > > > >>>> > > > >> in different way (interpolated based on the
> > > > > > >>>> > > > >> neighbor
> > > > > > >>>> > > > >> data by
> > > > > > >>>> using
> > > > > > >>>> > > > >> width
> > > > > > >>>> > > > >> and shape, not by the physical location).
The
> > > > > > >>>> > > > >> output is
> > > > > > >>>> different
> > > > > > >>>> > > > >> from
> > > > > > >>>> > > > >> point2grid. VIIRS AOD is not supported by
MET,
> > > > > > >>>> > > > >> yet.
> > > > > > >>>> > > > >> There
> > > > > > >>>> might be
> > > > > > >>>> > > > >> a
> > > > > > >>>> > > > >> workaround.
> > > > > > >>>> > > > >>
> > > > > > >>>> > > > >> Cheers,
> > > > > > >>>> > > > >> Howard
> > > > > > >>>> > > > >>
> > > > > > >>>> > > > >> On Fri May 29 13:24:55 2020, ho-
> > > > > > >>>> > > > >> chun.huang at noaa.gov
> > > > > > >>>> > > > >> wrote:
> > > > > > >>>> > > > >> > To Whom It May Concern:
> > > > > > >>>> > > > >> >
> > > > > > >>>> > > > >> > I have an LatLon 0.03x0.03 satellite
observed
> > > > > > >>>> > > > >> > AOD
> > > > > > >>>> > > > >> > at
> > > > > > 20200519
> > > > > > >>>> > > > >> > 20Z
> > > > > > >>>> > > > >> > (global
> > > > > > >>>> > > > >> > but most with fill_value). I want to use
> > > > > > >>>> > > > >> > regrid_data_plane
> > > > > > >>>> to
> > > > > > >>>> > > > >> > regrid
> > > > > > >>>> > > > >> > it to
> > > > > > >>>> > > > >> > CMAQ Lambert Conformal Grid. But it seems
to
> > > > > > >>>> > > > >> > hang
> > > > > > >>>> > > > >> > during
> > > > > > the
> > > > > > >>>> > > > >> > Interpolation
> > > > > > >>>> > > > >> > step.
> > > > > > >>>> > > > >> >
> > > > > > >>>> > > > >> > Can you find out what went wrong?
Sometimes
> > > > > > >>>> > > > >> > it
> > > > > > >>>> > > > >> > is
> > > > > > >>>> > > > >> > more
> > > > > > than
> > > > > > >>>> an
> > > > > > >>>> > > > >> > hour
> > > > > > >>>> > > > >> > waiting before I kill it.
> > > > > > >>>> > > > >> >
> > > > > > >>>> > > > >> > runscript in
> > > > > > >>>> > > > >> > venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > > > > >>>> > > > >> > Chun.Huang/plot/viirs/
> > > > > > >>>> > > > >> >
> > > > > > >>>> > > > >> > with command > bash run.viirs_aqm_aod.sh
> > > > > > >>>> > > > >> > viirs
> > > > > > >>>> > > > >> > aqm
> > > > > > 20200519
> > > > > > >>>> > > > >> > 20200519
> > > > > > >>>> > > > >
> > > > > > >>>> > > > >> > regrid.log 2>&1 &
> > > > > > >>>> > > > >> >
> > > > > > >>>> > > > >> > You can check the "regrid.log" in the
script
> > > > > > >>>> > > > >> > directory for
> > > > > > >>>> > > > >> > input,
> > > > > > >>>> > > > >> > model_grid, and output file location on
> > > > > > >>>> > > > >> > venus. I
> > > > > > >>>> > > > >> > will keep
> > > > > > >>>> it
> > > > > > >>>> > > > >> > going
> > > > > > >>>> > > > >> > from May 29 19:15Z.
> > > > > > >>>> > > > >> >
> > > > > > >>>> > > > >> > One thing I do not understand is that the
> > > > > > >>>> > > > >> > output
> > > > > > >>>> > > > >> > file
> > > > > > >>>> > > > >> > is
> > > > > > >>>> already
> > > > > > >>>> > > > >> > generated
> > > > > > >>>> > > > >> > while the job is still
> > > > > > >>>> > > > >> > running.
/gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > > >>>> > > > >> >
Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-
> > > > > > >>>> > > > >> > L3-
> > > > > > >>>> > > > >> > AOD_AQM_aqm_20200519_20.nc
> > > > > > >>>> > > > >> >
> > > > > > >>>> > > > >> > Ho-Chun Huang
> > > > > > >>>> > > > >> >
> > > > > > >>>> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > >>>> > > > >> >
> > > > > > >>>> > > > >> > 5830 University Research Ct., Rm. 2792
> > > > > > >>>> > > > >> >
> > > > > > >>>> > > > >> > College Park, MD 20740
> > > > > > >>>> > > > >> >
> > > > > > >>>> > > > >> > Ho-Chun.Huang at noaa.gov
<Joe.Smith at noaa.gov>
> > > > > > >>>> > > > >> >
> > > > > > >>>> > > > >> > 301-683-3958
> > > > > > >>>> > > > >>
> > > > > > >>>> > > > >>
> > > > > > >>>> > > > >>
> > > > > > >>>> > > > >>
> > > > > > >>>> > > >
> > > > > > >>>> > > >
> > > > > > >>>> >
> > > > > > >>>> >
> > > > > > >>>> >
> > > > > > >>>> >
> > > > > > >>>>
> > > > > > >>>>
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > >
> >
> >
> >
> >
------------------------------------------------
Subject: hang on regrid_data_plane
From: Ho-Chun Huang - NOAA Affiliate
Time: Tue Jun 16 11:02:04 2020
Hi,
Does it mean a new updated regrdi_data_plane is available for testing?
will
be released in 9.1_beta??, or else? Please advise.
If there is nothing else you need to do, please let me know about the
status of testing new utilities and you can close this ticket.
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Wed, Jun 10, 2020 at 2:12 PM Howard Soh via RT <met_help at ucar.edu>
wrote:
> The issue at github was created
(https://github.com/NCAR/MET/issues/1363).
>
> The main saving is from reading the compressed variable. The other
saving
> is from building a data plane (72M calls to 120K calls). There is no
> significant difference with compression level (between 1 and 9) with
the
> enhanced MET.
>
> Cheers,
> Howard
>
> Cheers,
> Howard
> On Tue Jun 09 10:53:07 2020, ho-chun.huang at noaa.gov wrote:
> > Hi, Howard:
> >
> > Thanks. I see, as long as there is a compression applied to the
> > netcdf
> > file, regrid_data_plane has to decompress each grid [for 6000
times].
> > C0
> > to C1 is a big reduction and I hope to be able to use compression
> > level 1
> > for VIIRS L3 AOD file.
> >
> > Ho-Chun Huang
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct., Rm. 2792
> >
> > College Park, MD 20740
> >
> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >
> > 301-683-3958
> >
> >
> > On Tue, Jun 9, 2020 at 12:33 PM Howard Soh via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > The compression is a variable based. NetCDF API reads a whole
data,
> > > decompresses & slices. With a quick test, it seems to be no big
> > > difference
> > > betwend compress level 1 and 2.
> > >
> > > File size (by nccopy)
> > > 8.1G VIIRS-L3-AOD_AQM_20200519_200000.c0.nc (no compression)
> > > 86M VIIRS-L3-AOD_AQM_20200519_200000.c1.nc (compress 1)
> > > 86M VIIRS-L3-AOD_AQM_20200519_200000.c2.nc (compress 2)
> > > 54M VIIRS-L3-AOD_AQM_20200519_200000.nc (compress 9)
> > >
> > > Wall time comparison with plot_data_plane (which reads 6000
times) at
> > > dakota (time is not accurate because the system load is not
> > > consistent):
> > >
> > > c0: 0m44.899s
> > > c1: 57m33.530s
> > > c2: 57m22.974s
> > >
> > > Cheers,
> > > Howard
> > >
> > > On Mon Jun 08 15:55:29 2020, ho-chun.huang at noaa.gov wrote:
> > > > Hi,
> > > >
> > > > I thought I asked the compress level=1 at the beginning. I
will
> > > > double
> > > > check with NESIDS again, but can you tell me now at what nc
> > > > compress
> > > > level
> > > > that regrid_data_plane does not need to decompress the
variable for
> > > > each
> > > > grid. If it can resolved from the data source level, it won't
> > > > waste
> > > > your
> > > > time for the special case.
> > > >
> > > > Ho-Chun Huang
> > > >
> > > > IMSG at NOAA/NWS/NCEP/EMC
> > > >
> > > > 5830 University Research Ct., Rm. 2792
> > > >
> > > > College Park, MD 20740
> > > >
> > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > >
> > > > 301-683-3958
> > > >
> > > >
> > > > On Mon, Jun 8, 2020 at 5:35 PM Howard Soh via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > The first cause is reading the variable 6000 times
(dimension is
> > > > > 10000 by
> > > > > 6000). The variables are compressed with with level 9
(highest).
> > > > > It
> > > > > taook
> > > > > more than 1 second to read & decompress a variable at
dakota.
> > > > > if 1 seconds, 1 * 6000 = 6000 seconds = 100 minutes
> > > > > if 1.5 seconds, 1.5 * 6000 = 9000 seconds = 150 minutes.
> > > > >
> > > > > The first step is reading a variable one time instead of
6000
> > > > > times.
> > > > > But
> > > > > the output did not match, so I'm working on making the same
> > > > > result.
> > > > > Once
> > > > > this is done, I will check if there is more room to speed up
> > > > > (some
> > > > > steps
> > > > > are executed 60M times).
> > > > >
> > > > > Cheers,
> > > > > Howard
> > > > >
> > > > > On Mon Jun 08 15:05:35 2020, johnhg wrote:
> > > > > > Howard and Ho-Chun,
> > > > > >
> > > > > > I'm wondering if we have any updates on this ticket?
Howard,
> > > > > > have
> > > > > > you
> > > > > > been
> > > > > > able to find a reason for this running so slowly?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Thu, Jun 4, 2020 at 10:55 AM Ho-Chun Huang - NOAA
Affiliate
> > > > > > via
> > > > > > RT
> > > > > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > > > >
> > > > > > >
> > > > > > > Hi, John and Howard:
> > > > > > >
> > > > > > > *Ho-Chun, do you know if the geometry of the VIIRS data
> > > > > > > repeats
> > > > > > > day
> > > > > > > after
> > > > > > > day?*
> > > > > > >
> > > > > > >
> > > > > > > There are two satellites currently providing VIIRS AOD
> > > > > > > observations,
> > > > > > > SUOMI-NPP and NOAA-20. Eash satellite overpasses the
same
> > > > > > > area
> > > > > > > twice
> > > > > > > per
> > > > > > > day, AOD only observed during daytime overpass. The
overpass
> > > > > > > for
> > > > > > > local time
> > > > > > > should be similar per satellite, NOAA-20 is ~ 50 mins
ahead
> > > > > > > of
> > > > > > > SUOMI-
> > > > > > > NPP.
> > > > > > >
> > > > > > > Although I think daily granule pixel location per
satellite
> > > > > > > may
> > > > > > > shift
> > > > > > > slightly day after day (still need to verify this
statement
> > > > > > > with
> > > > > > > NESDIS
> > > > > > > personal)? Thus, for L3 gridded AOD data, *the area of
> > > > > > > coverage*
> > > > > > > of
> > > > > > > orbital satellites should be similar *for the same hour
of
> > > > > > > day*.
> > > > > > > Maybe +-
> > > > > > > few grid boxes toward the north and west for predefined
> > > > > > > search
> > > > > > > area.
> > > > > > > Thus,
> > > > > > > if you put a predefined search box with an adequate
buffer
> > > > > > > zone
> > > > > > > on 4
> > > > > > > sides
> > > > > > > for specific UTC, maybe it will work.
> > > > > > >
> > > > > > >
> > > > > > > Ho-Chun Huang
> > > > > > >
> > > > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > >
> > > > > > > 5830 University Research Ct., Rm. 2792
> > > > > > >
> > > > > > > College Park, MD 20740
> > > > > > >
> > > > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > >
> > > > > > > 301-683-3958
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 4, 2020 at 12:00 PM Ho-Chun Huang - NOAA
> > > > > > > Affiliate <
> > > > > > > ho-chun.huang at noaa.gov> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Sorry for the wrong figure, attached is the direct
read
> > > > > > > > from
> > > > > > > > NESDIS
> > > > > > > > VIIRS
> > > > > > > > L3 AOD file.
> > > > > > > >
> > > > > > > > Ho-Chun Huang
> > > > > > > >
> > > > > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > >
> > > > > > > > 5830 University Research Ct., Rm. 2792
> > > > > > > >
> > > > > > > > College Park, MD 20740
> > > > > > > >
> > > > > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > >
> > > > > > > > 301-683-3958
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jun 4, 2020 at 11:57 AM Ho-Chun Huang - NOAA
> > > > > > > > Affiliate
> > > > > > > > <
> > > > > > > > ho-chun.huang at noaa.gov> wrote:
> > > > > > > >
> > > > > > > >> Hi,
> > > > > > > >>
> > > > > > > >> The regrid_data_plane already obtained the LatLon
> > > > > > > >> projection
> > > > > > > >> information
> > > > > > > >> from input obs file
> > > > > > > >>
> > > > > > > >> DEBUG 1: Reading data file:
> > > > > > > >>
> > > > > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > > > Chun.Huang/VIIRS_AOD/AOD/20200519/VIIRS-L3-
> > > > > > > AOD_AQM_20200519_200000.nc
> > > > > > > >> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file()
->
> > > > > > > >> created
> > > > > > > >> new
> > > > > > > >> Met2dDataFile object of type "FileType_NcMet".
> > > > > > > >> DEBUG 4:
> > > > > > > >> DEBUG 4: Latitude/Longitude Grid Data:
> > > > > > > >> DEBUG 4: lat_ll: -89.985
> > > > > > > >> DEBUG 4: lon_ll: 179.985
> > > > > > > >> DEBUG 4: delta_lat: 0.03
> > > > > > > >> DEBUG 4: delta_lon: 0.03
> > > > > > > >> DEBUG 4: Nlat: 6000
> > > > > > > >> DEBUG 4: Nlon: 12000
> > > > > > > >>
> > > > > > > >> Ho-Chun Huang
> > > > > > > >>
> > > > > > > >> IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > >>
> > > > > > > >> 5830 University Research Ct., Rm. 2792
> > > > > > > >>
> > > > > > > >> College Park, MD 20740
> > > > > > > >>
> > > > > > > >> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > >>
> > > > > > > >> 301-683-3958
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Thu, Jun 4, 2020 at 11:52 AM Ho-Chun Huang - NOAA
> > > > > > > >> Affiliate
> > > > > > > >> <
> > > > > > > >> ho-chun.huang at noaa.gov> wrote:
> > > > > > > >>
> > > > > > > >>> Hi, Howard and John:
> > > > > > > >>>
> > > > > > > >>> The VIIRS L3 AOD is on fixed LatLon 0.03x0.03 grid,
so
> > > > > > > >>> its
> > > > > > > >>> location is
> > > > > > > >>> fixed everyday. Only the area of observed data will
> > > > > > > >>> change
> > > > > > > >>> for
> > > > > > > >>> each
> > > > > > > hours,
> > > > > > > >>> in general moving northward and westward.
> > > > > > > >>>
> > > > > > > >>> This is graphic based on direct read of the input
VIIRS
> > > > > > > >>> L3
> > > > > > > >>> AOD
> > > > > > > >>> file,
> > > > > > > for
> > > > > > > >>> your reference,
> > > > > > > >>>
> > > > > > > >>> I want to emphasize again, the input VIIRS AOD is
gridded
> > > > > > > >>> and
> > > > > > > >>> *NOT*
> > > > > > > >>> pixel data, you do not need to recompute the geo-
location
> > > > > > > >>> for
> > > > > > > >>> each
> > > > > > > -field
> > > > > > > >>> variables.
> > > > > > > >>>
> > > > > > > >>> Ho-Chun Huang
> > > > > > > >>>
> > > > > > > >>> IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > >>>
> > > > > > > >>> 5830 University Research Ct., Rm. 2792
> > > > > > > >>>
> > > > > > > >>> College Park, MD 20740
> > > > > > > >>>
> > > > > > > >>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > >>>
> > > > > > > >>> 301-683-3958
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Thu, Jun 4, 2020 at 11:39 AM John Halley Gotway
via RT
> > > > > > > >>> <
> > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > >>>
> > > > > > > >>>> Howard,
> > > > > > > >>>>
> > > > > > > >>>> I recall that with the GOES files, you set up
> > > > > > > >>>> regrid_data_plane
> > > > > > > >>>> to
> > > > > > > >>>> determine the grid mapping the first time it's run,
and
> > > > > > > >>>> then
> > > > > > > >>>> reuse
> > > > > > > that
> > > > > > > >>>> mapping information in subsequent calls. I really
don't
> > > > > > > >>>> know
> > > > > > > >>>> anything
> > > > > > > >>>> about
> > > > > > > >>>> the VIIRS data. Searching online, it looks like the
> > > > > > > >>>> VIIRS
> > > > > > > >>>> data
> > > > > > > >>>> comes
> > > > > > > in
> > > > > > > >>>> swaths, whereas the GOES data is stationary. So
perhaps
> > > > > > > >>>> the
> > > > > > > >>>> approach
> > > > > > > of
> > > > > > > >>>> pre-computing the grid mappings won't work. I
wonder if
> > > > > > > >>>> the
> > > > > > > >>>> swaths are
> > > > > > > >>>> repeated though.
> > > > > > > >>>>
> > > > > > > >>>> Ho-Chun, do you know if the geometry of the VIIRS
data
> > > > > > > >>>> repeats
> > > > > > > >>>> day
> > > > > > > after
> > > > > > > >>>> day? If we can figure out a way of computing the
grid
> > > > > > > >>>> mapping
> > > > > > > >>>> information
> > > > > > > >>>> once ahead of time and then reuse that information,
I'm
> > > > > > > >>>> guessing
> > > > > > > >>>> the
> > > > > > > >>>> tool
> > > > > > > >>>> would run much faster.
> > > > > > > >>>>
> > > > > > > >>>> Thanks,
> > > > > > > >>>> John
> > > > > > > >>>>
> > > > > > > >>>> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT
> > > > > > > >>>> <met_help at ucar.edu>
> > > > > > > >>>> wrote:
> > > > > > > >>>>
> > > > > > > >>>> >
> > > > > > > >>>> > <URL:
> > > > > > > >>>> >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > > > >>>> > >
> > > > > > > >>>> >
> > > > > > > >>>> > I will investigate why it takes too long.
> > > > > > > >>>> >
> > > > > > > >>>> > I ran regrid_data_plane for just one variable
> > > > > > > >>>> > (AOD_H_Quality,
> > > > > > > >>>> dimension is
> > > > > > > >>>> > 6000 by 12000) and it took about 4 hours at
dakota.
> > > > > > > >>>> >
> > > > > > > >>>> > DEBUG 1: Writing output file:
> > > > > > > >>>> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
> > > > > > > >>>> >
> > > > > > > >>>> > real 239m2.552s
> > > > > > > >>>> > user 206m42.332s
> > > > > > > >>>> > sys 32m18.028s
> > > > > > > >>>> >
> > > > > > > >>>> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does not
have
> > > > > > > >>>> > the
> > > > > > > >>>> > time
> > > > > > > >>>> variable
> > > > > > > >>>> > "t". So point2grid does not handle this case with
> > > > > > > >>>> > workaround.
> > > > > > > >>>> >
> > > > > > > >>>> > Cheers,
> > > > > > > >>>> > Howard
> > > > > > > >>>> >
> > > > > > > >>>> > On Thu Jun 04 06:53:39 2020, ho-
chun.huang at noaa.gov
> > > > > > > >>>> > wrote:
> > > > > > > >>>> > > Hi, John and Howard:
> > > > > > > >>>> > >
> > > > > > > >>>> > > (1) You should be able to see my MET version in
the
> > > > > > > >>>> > > run
> > > > > > > >>>> > > script :
> > > > > > > 9.0
> > > > > > > >>>> > >
> > > > > > > >>>> > > (2) Background information, we used to discuss
> > > > > > > >>>> > > between
> > > > > > > >>>> > > NCAR
> > > > > > > >>>> > > MET,
> > > > > > > >>>> > > NESDIS,
> > > > > > > >>>> > > and EMC about using VIIRS pixel AOD data per
granule
> > > > > > > >>>> > > and
> > > > > > > >>>> > > mapping
> > > > > > > >>>> pixel
> > > > > > > >>>> > > observed AOD into a selected model grid. It
has
> > > > > > > >>>> > > been
> > > > > > > >>>> > > determined
> > > > > > > >>>> that
> > > > > > > >>>> > > the
> > > > > > > >>>> > > data volume is too large for MET to handle
VIIRS
> > > > > > > >>>> > > granule
> > > > > > > >>>> > > pixel
> > > > > > > data
> > > > > > > >>>> > > ingestion. EMC has asked NESDIS aerosol groups
to
> > > > > > > >>>> > > provide
> > > > > > > >>>> > > *L3
> > > > > > > >>>> > > "gridded"
> > > > > > > >>>> > > nefCDF* files for global and regional AQM
model
> > > > > > > >>>> > > evaluation/verification.
> > > > > > > >>>> > > I am testing the regional one VIIRS-L3-
AOD_*AQM*_
> > > > > > > 20200519_200000.nc
> > > > > > > >>>> ,
> > > > > > > >>>> > > while Partha Bhattacharjee is testing VIIRS-L3-
> > > > > > > >>>> > > AOD_*GEFS*_20200519_120000.nc
> > > > > > > >>>> > > for global GEFS-aerosol verification [reference
> > > > > > > >>>> > > helpdesk
> > > > > > > >>>> > > ticket
> > > > > > > >>>> > > 95358].
> > > > > > > >>>> > >
> > > > > > > >>>> > > (3) *I am not trying to using point2grid*, I am
> > > > > > > >>>> > > trying
> > > > > > > >>>> > > to
> > > > > > > >>>> > > use MET
> > > > > > > >>>> > > regridding utilities to map a *gridded* netCDF
to a
> > > > > > > >>>> > > target
> > > > > > > >>>> > > model
> > > > > > > >>>> > > *gridded*
> > > > > > > >>>> > > netCDF. Unless there is another better tool
in MET
> > > > > > > >>>> > > packages
> > > > > > > that I
> > > > > > > >>>> > > am not
> > > > > > > >>>> > > aware of, I assume regrid_data_plane is for
general
> > > > > > > >>>> > > mapping
> > > > > > > purpose.
> > > > > > > >>>> > >
> > > > > > > >>>> > > (4) For each hourly period, orbital satellites
> > > > > > > >>>> > > overpassed
> > > > > > > >>>> > > only a
> > > > > > > >>>> small
> > > > > > > >>>> > > area
> > > > > > > >>>> > > around the globe. I have asked NESDIS aerosol
to use
> > > > > > > >>>> > > fill-
> > > > > > > >>>> > > values
> > > > > > > for
> > > > > > > >>>> > > non-observed grids and with compress level=1
hoping
> > > > > > > >>>> regard-data_plane
> > > > > > > >>>> > > can
> > > > > > > >>>> > > skip those "-9999.f" grid quickly. I may be
wrong.
> > > > > > > >>>> > > In
> > > > > > > >>>> > > reality, I
> > > > > > > >>>> may
> > > > > > > >>>> > > only
> > > > > > > >>>> > > use files from 18Z to 23Z for verification on
CONUS,
> > > > > > > >>>> > > Alaska,
> > > > > > > >>>> > > and
> > > > > > > >>>> > > Hawaii.
> > > > > > > >>>> > > The reason to keep it global coverage is to
simplify
> > > > > > > >>>> > > NESDIS
> > > > > > > >>>> production
> > > > > > > >>>> > > processes and be flexible for future
applications.
> > > > > > > >>>> > > Please
> > > > > > > >>>> > > study my
> > > > > > > >>>> > > problem,
> > > > > > > >>>> > > if changes on the data production side [NESDIS
> > > > > > > >>>> > > aerosol
> > > > > > > >>>> > > group] can
> > > > > > > >>>> > > speed-up
> > > > > > > >>>> > > the regridding processes, please let me know.
> > > > > > > >>>> > >
> > > > > > > >>>> > > Thank you for your help.
> > > > > > > >>>> > >
> > > > > > > >>>> > > Ho-Chun Huang
> > > > > > > >>>> > >
> > > > > > > >>>> > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > >>>> > >
> > > > > > > >>>> > > 5830 University Research Ct., Rm. 2792
> > > > > > > >>>> > >
> > > > > > > >>>> > > College Park, MD 20740
> > > > > > > >>>> > >
> > > > > > > >>>> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > >>>> > >
> > > > > > > >>>> > > 301-683-3958
> > > > > > > >>>> > >
> > > > > > > >>>> > >
> > > > > > > >>>> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley
Gotway
> > > > > > > >>>> > > via RT
> > > > > > > >>>> > > <met_help at ucar.edu>
> > > > > > > >>>> > > wrote:
> > > > > > > >>>> > >
> > > > > > > >>>> > > > Howard,
> > > > > > > >>>> > > >
> > > > > > > >>>> > > > OK, please find input data for testing here:
> > > > > > > >>>> > > >
> > > > > > > >>>> > > >
> > > > > > > >>>> >
> > > > > > > >>>>
> > > > > > >
> > > > >
> > >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> > > > > > > >>>> > > >
> > > > > > > >>>> > > > When you untar this file, you'll find the
contents
> > > > > > > >>>> > > > of
> > > > > > > >>>> > > > Ho-
> > > > > > > >>>> > > > Chun's
> > > > > > > >>>> > > > directory:
> > > > > > > >>>> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > > > > > >>>> > > > Chun.Huang/plot/viirs
> > > > > > > >>>> > > >
> > > > > > > >>>> > > > In addition, I copied the input/output
directories
> > > > > > > >>>> > > > from
> > > > > > > >>>> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > > > >>>> > > > Chun.Huang/VIIRS_AOD
> > > > > > > >>>> > > > into
> > > > > > > that
> > > > > > > >>>> > > > viirs
> > > > > > > >>>> > > > directory. Also, I included the file named
> > > > > > > >>>> > > > "aqm.aot.148.grid"
> > > > > > > >>>> which
> > > > > > > >>>> > > > defines
> > > > > > > >>>> > > > the target grid.
> > > > > > > >>>> > > >
> > > > > > > >>>> > > > If you're able to take a look at this data
and see
> > > > > > > >>>> > > > how
> > > > > > > >>>> > > > it
> > > > > > > >>>> > > > runs,
> > > > > > > >>>> > > > that'd be
> > > > > > > >>>> > > > great! Ho-Chun found that it took over 12
hours!?
> > > > > > > >>>> > > > to
> > > > > > > >>>> > > > run
> > > > > > > >>>> > > > on
> > > > > > > WCOSS.
> > > > > > > >>>> > > >
> > > > > > > >>>> > > > Thanks,
> > > > > > > >>>> > > > John
> > > > > > > >>>> > > >
> > > > > > > >>>> > > > On Wed, Jun 3, 2020 at 4:45 PM John Halley
Gotway
> > > > > > > >>>> > > > <
> > > > > > > >>>> johnhg at ucar.edu>
> > > > > > > >>>> > > > wrote:
> > > > > > > >>>> > > >
> > > > > > > >>>> > > > > Howard,
> > > > > > > >>>> > > > >
> > > > > > > >>>> > > > > I'll go grab this data and copy it over to
> > > > > > > >>>> > > > > dakota.
> > > > > > > >>>> > > > > Ho-
> > > > > > > >>>> > > > > Chun,
> > > > > > > >>>> sorry
> > > > > > > >>>> > > > > for the
> > > > > > > >>>> > > > > delay in getting started with this. I'll go
grab
> > > > > > > >>>> > > > > it
> > > > > > > >>>> > > > > now.
> > > > > > > >>>> > > > >
> > > > > > > >>>> > > > > John
> > > > > > > >>>> > > > >
> > > > > > > >>>> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh
via RT
> > > > > > > >>>> > > > > <met_help at ucar.edu>
> > > > > > > >>>> > > > > wrote:
> > > > > > > >>>> > > > >
> > > > > > > >>>> > > > >>
> > > > > > > >>>> > > > >> <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > > > >>>> >
> > > > > > > >>>> > > > >>
> > > > > > > >>>> > > > >> I don't have access to venus.
> > > > > > > >>>> > > > >> Would you upload the files (config,
script,
> > > > > > > >>>> > > > >> log,
> > > > > > > >>>> > > > >> and
> > > > > > > >>>> > > > >> data
> > > > > > > >>>> files)
> > > > > > > >>>> > > > >> to MET
> > > > > > > >>>> > > > >> ftp server?
> > > > > > > >>>> > > > >>
> > > > > > > >>>> > > > >> Which MET version are you using?
> > > > > > > >>>> > > > >>
> > > > > > > >>>> > > > >> The GOES-AOD data was supported by
> > > > > > > >>>> > > > >> regrid_data_plane
> > > > > > > >>>> > > > >> (V8.x).
> > > > > > > >>>> It's
> > > > > > > >>>> > > > >> moved
> > > > > > > >>>> > > > >> to point2grid (V9.0). The
regrid_data_plane
> > > > > > > >>>> > > > >> still
> > > > > > > >>>> > > > >> processes
> > > > > > > >>>> GOES-
> > > > > > > >>>> > > > >> AOD
> > > > > > > >>>> > > > data
> > > > > > > >>>> > > > >> in different way (interpolated based on
the
> > > > > > > >>>> > > > >> neighbor
> > > > > > > >>>> > > > >> data by
> > > > > > > >>>> using
> > > > > > > >>>> > > > >> width
> > > > > > > >>>> > > > >> and shape, not by the physical location).
The
> > > > > > > >>>> > > > >> output is
> > > > > > > >>>> different
> > > > > > > >>>> > > > >> from
> > > > > > > >>>> > > > >> point2grid. VIIRS AOD is not supported by
MET,
> > > > > > > >>>> > > > >> yet.
> > > > > > > >>>> > > > >> There
> > > > > > > >>>> might be
> > > > > > > >>>> > > > >> a
> > > > > > > >>>> > > > >> workaround.
> > > > > > > >>>> > > > >>
> > > > > > > >>>> > > > >> Cheers,
> > > > > > > >>>> > > > >> Howard
> > > > > > > >>>> > > > >>
> > > > > > > >>>> > > > >> On Fri May 29 13:24:55 2020, ho-
> > > > > > > >>>> > > > >> chun.huang at noaa.gov
> > > > > > > >>>> > > > >> wrote:
> > > > > > > >>>> > > > >> > To Whom It May Concern:
> > > > > > > >>>> > > > >> >
> > > > > > > >>>> > > > >> > I have an LatLon 0.03x0.03 satellite
observed
> > > > > > > >>>> > > > >> > AOD
> > > > > > > >>>> > > > >> > at
> > > > > > > 20200519
> > > > > > > >>>> > > > >> > 20Z
> > > > > > > >>>> > > > >> > (global
> > > > > > > >>>> > > > >> > but most with fill_value). I want to
use
> > > > > > > >>>> > > > >> > regrid_data_plane
> > > > > > > >>>> to
> > > > > > > >>>> > > > >> > regrid
> > > > > > > >>>> > > > >> > it to
> > > > > > > >>>> > > > >> > CMAQ Lambert Conformal Grid. But it
seems to
> > > > > > > >>>> > > > >> > hang
> > > > > > > >>>> > > > >> > during
> > > > > > > the
> > > > > > > >>>> > > > >> > Interpolation
> > > > > > > >>>> > > > >> > step.
> > > > > > > >>>> > > > >> >
> > > > > > > >>>> > > > >> > Can you find out what went wrong?
Sometimes
> > > > > > > >>>> > > > >> > it
> > > > > > > >>>> > > > >> > is
> > > > > > > >>>> > > > >> > more
> > > > > > > than
> > > > > > > >>>> an
> > > > > > > >>>> > > > >> > hour
> > > > > > > >>>> > > > >> > waiting before I kill it.
> > > > > > > >>>> > > > >> >
> > > > > > > >>>> > > > >> > runscript in
> > > > > > > >>>> > > > >> > venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > > > > > >>>> > > > >> > Chun.Huang/plot/viirs/
> > > > > > > >>>> > > > >> >
> > > > > > > >>>> > > > >> > with command > bash run.viirs_aqm_aod.sh
> > > > > > > >>>> > > > >> > viirs
> > > > > > > >>>> > > > >> > aqm
> > > > > > > 20200519
> > > > > > > >>>> > > > >> > 20200519
> > > > > > > >>>> > > > >
> > > > > > > >>>> > > > >> > regrid.log 2>&1 &
> > > > > > > >>>> > > > >> >
> > > > > > > >>>> > > > >> > You can check the "regrid.log" in the
script
> > > > > > > >>>> > > > >> > directory for
> > > > > > > >>>> > > > >> > input,
> > > > > > > >>>> > > > >> > model_grid, and output file location on
> > > > > > > >>>> > > > >> > venus. I
> > > > > > > >>>> > > > >> > will keep
> > > > > > > >>>> it
> > > > > > > >>>> > > > >> > going
> > > > > > > >>>> > > > >> > from May 29 19:15Z.
> > > > > > > >>>> > > > >> >
> > > > > > > >>>> > > > >> > One thing I do not understand is that
the
> > > > > > > >>>> > > > >> > output
> > > > > > > >>>> > > > >> > file
> > > > > > > >>>> > > > >> > is
> > > > > > > >>>> already
> > > > > > > >>>> > > > >> > generated
> > > > > > > >>>> > > > >> > while the job is still
> > > > > > > >>>> > > > >> > running.
/gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > > > >>>> > > > >> >
Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-
> > > > > > > >>>> > > > >> > L3-
> > > > > > > >>>> > > > >> > AOD_AQM_aqm_20200519_20.nc
> > > > > > > >>>> > > > >> >
> > > > > > > >>>> > > > >> > Ho-Chun Huang
> > > > > > > >>>> > > > >> >
> > > > > > > >>>> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > >>>> > > > >> >
> > > > > > > >>>> > > > >> > 5830 University Research Ct., Rm. 2792
> > > > > > > >>>> > > > >> >
> > > > > > > >>>> > > > >> > College Park, MD 20740
> > > > > > > >>>> > > > >> >
> > > > > > > >>>> > > > >> > Ho-Chun.Huang at noaa.gov
<Joe.Smith at noaa.gov>
> > > > > > > >>>> > > > >> >
> > > > > > > >>>> > > > >> > 301-683-3958
> > > > > > > >>>> > > > >>
> > > > > > > >>>> > > > >>
> > > > > > > >>>> > > > >>
> > > > > > > >>>> > > > >>
> > > > > > > >>>> > > >
> > > > > > > >>>> > > >
> > > > > > > >>>> >
> > > > > > > >>>> >
> > > > > > > >>>> >
> > > > > > > >>>> >
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: hang on regrid_data_plane
From: Howard Soh
Time: Tue Jun 16 12:56:33 2020
It will be available with next beta release.
John tested this at kiowa
(kiowa:/d1/projects/MET/MET_pull_requests/met-9.1_beta2/feature_1363)
Cheers,
Howard
On Tue Jun 16 11:02:04 2020, ho-chun.huang at noaa.gov wrote:
> Hi,
>
> Does it mean a new updated regrdi_data_plane is available for
testing?
> will
> be released in 9.1_beta??, or else? Please advise.
>
> If there is nothing else you need to do, please let me know about
the
> status of testing new utilities and you can close this ticket.
>
> Ho-Chun Huang
>
> IMSG at NOAA/NWS/NCEP/EMC
>
> 5830 University Research Ct., Rm. 2792
>
> College Park, MD 20740
>
> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
>
> 301-683-3958
>
>
> On Wed, Jun 10, 2020 at 2:12 PM Howard Soh via RT
<met_help at ucar.edu>
> wrote:
>
> > The issue at github was created
> > (https://github.com/NCAR/MET/issues/1363).
> >
> > The main saving is from reading the compressed variable. The other
> > saving
> > is from building a data plane (72M calls to 120K calls). There is
no
> > significant difference with compression level (between 1 and 9)
with
> > the
> > enhanced MET.
> >
> > Cheers,
> > Howard
> >
> > Cheers,
> > Howard
> > On Tue Jun 09 10:53:07 2020, ho-chun.huang at noaa.gov wrote:
> > > Hi, Howard:
> > >
> > > Thanks. I see, as long as there is a compression applied to the
> > > netcdf
> > > file, regrid_data_plane has to decompress each grid [for 6000
> > > times].
> > > C0
> > > to C1 is a big reduction and I hope to be able to use
compression
> > > level 1
> > > for VIIRS L3 AOD file.
> > >
> > > Ho-Chun Huang
> > >
> > > IMSG at NOAA/NWS/NCEP/EMC
> > >
> > > 5830 University Research Ct., Rm. 2792
> > >
> > > College Park, MD 20740
> > >
> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > >
> > > 301-683-3958
> > >
> > >
> > > On Tue, Jun 9, 2020 at 12:33 PM Howard Soh via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > > The compression is a variable based. NetCDF API reads a whole
> > > > data,
> > > > decompresses & slices. With a quick test, it seems to be no
big
> > > > difference
> > > > betwend compress level 1 and 2.
> > > >
> > > > File size (by nccopy)
> > > > 8.1G VIIRS-L3-AOD_AQM_20200519_200000.c0.nc (no compression)
> > > > 86M VIIRS-L3-AOD_AQM_20200519_200000.c1.nc (compress 1)
> > > > 86M VIIRS-L3-AOD_AQM_20200519_200000.c2.nc (compress 2)
> > > > 54M VIIRS-L3-AOD_AQM_20200519_200000.nc (compress 9)
> > > >
> > > > Wall time comparison with plot_data_plane (which reads 6000
> > > > times) at
> > > > dakota (time is not accurate because the system load is not
> > > > consistent):
> > > >
> > > > c0: 0m44.899s
> > > > c1: 57m33.530s
> > > > c2: 57m22.974s
> > > >
> > > > Cheers,
> > > > Howard
> > > >
> > > > On Mon Jun 08 15:55:29 2020, ho-chun.huang at noaa.gov wrote:
> > > > > Hi,
> > > > >
> > > > > I thought I asked the compress level=1 at the beginning. I
> > > > > will
> > > > > double
> > > > > check with NESIDS again, but can you tell me now at what nc
> > > > > compress
> > > > > level
> > > > > that regrid_data_plane does not need to decompress the
variable
> > > > > for
> > > > > each
> > > > > grid. If it can resolved from the data source level, it
won't
> > > > > waste
> > > > > your
> > > > > time for the special case.
> > > > >
> > > > > Ho-Chun Huang
> > > > >
> > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > >
> > > > > 5830 University Research Ct., Rm. 2792
> > > > >
> > > > > College Park, MD 20740
> > > > >
> > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > >
> > > > > 301-683-3958
> > > > >
> > > > >
> > > > > On Mon, Jun 8, 2020 at 5:35 PM Howard Soh via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > The first cause is reading the variable 6000 times
(dimension
> > > > > > is
> > > > > > 10000 by
> > > > > > 6000). The variables are compressed with with level 9
> > > > > > (highest).
> > > > > > It
> > > > > > taook
> > > > > > more than 1 second to read & decompress a variable at
dakota.
> > > > > > if 1 seconds, 1 * 6000 = 6000 seconds = 100 minutes
> > > > > > if 1.5 seconds, 1.5 * 6000 = 9000 seconds = 150 minutes.
> > > > > >
> > > > > > The first step is reading a variable one time instead of
6000
> > > > > > times.
> > > > > > But
> > > > > > the output did not match, so I'm working on making the
same
> > > > > > result.
> > > > > > Once
> > > > > > this is done, I will check if there is more room to speed
up
> > > > > > (some
> > > > > > steps
> > > > > > are executed 60M times).
> > > > > >
> > > > > > Cheers,
> > > > > > Howard
> > > > > >
> > > > > > On Mon Jun 08 15:05:35 2020, johnhg wrote:
> > > > > > > Howard and Ho-Chun,
> > > > > > >
> > > > > > > I'm wondering if we have any updates on this ticket?
> > > > > > > Howard,
> > > > > > > have
> > > > > > > you
> > > > > > > been
> > > > > > > able to find a reason for this running so slowly?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Thu, Jun 4, 2020 at 10:55 AM Ho-Chun Huang - NOAA
> > > > > > > Affiliate
> > > > > > > via
> > > > > > > RT
> > > > > > > <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi, John and Howard:
> > > > > > > >
> > > > > > > > *Ho-Chun, do you know if the geometry of the VIIRS
data
> > > > > > > > repeats
> > > > > > > > day
> > > > > > > > after
> > > > > > > > day?*
> > > > > > > >
> > > > > > > >
> > > > > > > > There are two satellites currently providing VIIRS AOD
> > > > > > > > observations,
> > > > > > > > SUOMI-NPP and NOAA-20. Eash satellite overpasses the
> > > > > > > > same
> > > > > > > > area
> > > > > > > > twice
> > > > > > > > per
> > > > > > > > day, AOD only observed during daytime overpass. The
> > > > > > > > overpass
> > > > > > > > for
> > > > > > > > local time
> > > > > > > > should be similar per satellite, NOAA-20 is ~ 50 mins
> > > > > > > > ahead
> > > > > > > > of
> > > > > > > > SUOMI-
> > > > > > > > NPP.
> > > > > > > >
> > > > > > > > Although I think daily granule pixel location per
> > > > > > > > satellite
> > > > > > > > may
> > > > > > > > shift
> > > > > > > > slightly day after day (still need to verify this
> > > > > > > > statement
> > > > > > > > with
> > > > > > > > NESDIS
> > > > > > > > personal)? Thus, for L3 gridded AOD data, *the area
of
> > > > > > > > coverage*
> > > > > > > > of
> > > > > > > > orbital satellites should be similar *for the same
hour
> > > > > > > > of
> > > > > > > > day*.
> > > > > > > > Maybe +-
> > > > > > > > few grid boxes toward the north and west for
predefined
> > > > > > > > search
> > > > > > > > area.
> > > > > > > > Thus,
> > > > > > > > if you put a predefined search box with an adequate
> > > > > > > > buffer
> > > > > > > > zone
> > > > > > > > on 4
> > > > > > > > sides
> > > > > > > > for specific UTC, maybe it will work.
> > > > > > > >
> > > > > > > >
> > > > > > > > Ho-Chun Huang
> > > > > > > >
> > > > > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > >
> > > > > > > > 5830 University Research Ct., Rm. 2792
> > > > > > > >
> > > > > > > > College Park, MD 20740
> > > > > > > >
> > > > > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > >
> > > > > > > > 301-683-3958
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jun 4, 2020 at 12:00 PM Ho-Chun Huang - NOAA
> > > > > > > > Affiliate <
> > > > > > > > ho-chun.huang at noaa.gov> wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > Sorry for the wrong figure, attached is the direct
read
> > > > > > > > > from
> > > > > > > > > NESDIS
> > > > > > > > > VIIRS
> > > > > > > > > L3 AOD file.
> > > > > > > > >
> > > > > > > > > Ho-Chun Huang
> > > > > > > > >
> > > > > > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > > >
> > > > > > > > > 5830 University Research Ct., Rm. 2792
> > > > > > > > >
> > > > > > > > > College Park, MD 20740
> > > > > > > > >
> > > > > > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > > >
> > > > > > > > > 301-683-3958
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jun 4, 2020 at 11:57 AM Ho-Chun Huang - NOAA
> > > > > > > > > Affiliate
> > > > > > > > > <
> > > > > > > > > ho-chun.huang at noaa.gov> wrote:
> > > > > > > > >
> > > > > > > > >> Hi,
> > > > > > > > >>
> > > > > > > > >> The regrid_data_plane already obtained the LatLon
> > > > > > > > >> projection
> > > > > > > > >> information
> > > > > > > > >> from input obs file
> > > > > > > > >>
> > > > > > > > >> DEBUG 1: Reading data file:
> > > > > > > > >>
> > > > > > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > > > > Chun.Huang/VIIRS_AOD/AOD/20200519/VIIRS-L3-
> > > > > > > > AOD_AQM_20200519_200000.nc
> > > > > > > > >> DEBUG 4:
Met2dDataFileFactory::new_met_2d_data_file()
> > > > > > > > >> ->
> > > > > > > > >> created
> > > > > > > > >> new
> > > > > > > > >> Met2dDataFile object of type "FileType_NcMet".
> > > > > > > > >> DEBUG 4:
> > > > > > > > >> DEBUG 4: Latitude/Longitude Grid Data:
> > > > > > > > >> DEBUG 4: lat_ll: -89.985
> > > > > > > > >> DEBUG 4: lon_ll: 179.985
> > > > > > > > >> DEBUG 4: delta_lat: 0.03
> > > > > > > > >> DEBUG 4: delta_lon: 0.03
> > > > > > > > >> DEBUG 4: Nlat: 6000
> > > > > > > > >> DEBUG 4: Nlon: 12000
> > > > > > > > >>
> > > > > > > > >> Ho-Chun Huang
> > > > > > > > >>
> > > > > > > > >> IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > > >>
> > > > > > > > >> 5830 University Research Ct., Rm. 2792
> > > > > > > > >>
> > > > > > > > >> College Park, MD 20740
> > > > > > > > >>
> > > > > > > > >> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > > >>
> > > > > > > > >> 301-683-3958
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Thu, Jun 4, 2020 at 11:52 AM Ho-Chun Huang -
NOAA
> > > > > > > > >> Affiliate
> > > > > > > > >> <
> > > > > > > > >> ho-chun.huang at noaa.gov> wrote:
> > > > > > > > >>
> > > > > > > > >>> Hi, Howard and John:
> > > > > > > > >>>
> > > > > > > > >>> The VIIRS L3 AOD is on fixed LatLon 0.03x0.03
grid,
> > > > > > > > >>> so
> > > > > > > > >>> its
> > > > > > > > >>> location is
> > > > > > > > >>> fixed everyday. Only the area of observed data
will
> > > > > > > > >>> change
> > > > > > > > >>> for
> > > > > > > > >>> each
> > > > > > > > hours,
> > > > > > > > >>> in general moving northward and westward.
> > > > > > > > >>>
> > > > > > > > >>> This is graphic based on direct read of the input
> > > > > > > > >>> VIIRS
> > > > > > > > >>> L3
> > > > > > > > >>> AOD
> > > > > > > > >>> file,
> > > > > > > > for
> > > > > > > > >>> your reference,
> > > > > > > > >>>
> > > > > > > > >>> I want to emphasize again, the input VIIRS AOD is
> > > > > > > > >>> gridded
> > > > > > > > >>> and
> > > > > > > > >>> *NOT*
> > > > > > > > >>> pixel data, you do not need to recompute the geo-
> > > > > > > > >>> location
> > > > > > > > >>> for
> > > > > > > > >>> each
> > > > > > > > -field
> > > > > > > > >>> variables.
> > > > > > > > >>>
> > > > > > > > >>> Ho-Chun Huang
> > > > > > > > >>>
> > > > > > > > >>> IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > > >>>
> > > > > > > > >>> 5830 University Research Ct., Rm. 2792
> > > > > > > > >>>
> > > > > > > > >>> College Park, MD 20740
> > > > > > > > >>>
> > > > > > > > >>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > > >>>
> > > > > > > > >>> 301-683-3958
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Thu, Jun 4, 2020 at 11:39 AM John Halley Gotway
> > > > > > > > >>> via RT
> > > > > > > > >>> <
> > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Howard,
> > > > > > > > >>>>
> > > > > > > > >>>> I recall that with the GOES files, you set up
> > > > > > > > >>>> regrid_data_plane
> > > > > > > > >>>> to
> > > > > > > > >>>> determine the grid mapping the first time it's
run,
> > > > > > > > >>>> and
> > > > > > > > >>>> then
> > > > > > > > >>>> reuse
> > > > > > > > that
> > > > > > > > >>>> mapping information in subsequent calls. I really
> > > > > > > > >>>> don't
> > > > > > > > >>>> know
> > > > > > > > >>>> anything
> > > > > > > > >>>> about
> > > > > > > > >>>> the VIIRS data. Searching online, it looks like
the
> > > > > > > > >>>> VIIRS
> > > > > > > > >>>> data
> > > > > > > > >>>> comes
> > > > > > > > in
> > > > > > > > >>>> swaths, whereas the GOES data is stationary. So
> > > > > > > > >>>> perhaps
> > > > > > > > >>>> the
> > > > > > > > >>>> approach
> > > > > > > > of
> > > > > > > > >>>> pre-computing the grid mappings won't work. I
wonder
> > > > > > > > >>>> if
> > > > > > > > >>>> the
> > > > > > > > >>>> swaths are
> > > > > > > > >>>> repeated though.
> > > > > > > > >>>>
> > > > > > > > >>>> Ho-Chun, do you know if the geometry of the VIIRS
> > > > > > > > >>>> data
> > > > > > > > >>>> repeats
> > > > > > > > >>>> day
> > > > > > > > after
> > > > > > > > >>>> day? If we can figure out a way of computing the
> > > > > > > > >>>> grid
> > > > > > > > >>>> mapping
> > > > > > > > >>>> information
> > > > > > > > >>>> once ahead of time and then reuse that
information,
> > > > > > > > >>>> I'm
> > > > > > > > >>>> guessing
> > > > > > > > >>>> the
> > > > > > > > >>>> tool
> > > > > > > > >>>> would run much faster.
> > > > > > > > >>>>
> > > > > > > > >>>> Thanks,
> > > > > > > > >>>> John
> > > > > > > > >>>>
> > > > > > > > >>>> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via RT
> > > > > > > > >>>> <met_help at ucar.edu>
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>
> > > > > > > > >>>> >
> > > > > > > > >>>> > <URL:
> > > > > > > > >>>> >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > > > > >>>> > >
> > > > > > > > >>>> >
> > > > > > > > >>>> > I will investigate why it takes too long.
> > > > > > > > >>>> >
> > > > > > > > >>>> > I ran regrid_data_plane for just one variable
> > > > > > > > >>>> > (AOD_H_Quality,
> > > > > > > > >>>> dimension is
> > > > > > > > >>>> > 6000 by 12000) and it took about 4 hours at
> > > > > > > > >>>> > dakota.
> > > > > > > > >>>> >
> > > > > > > > >>>> > DEBUG 1: Writing output file:
> > > > > > > > >>>> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
> > > > > > > > >>>> >
> > > > > > > > >>>> > real 239m2.552s
> > > > > > > > >>>> > user 206m42.332s
> > > > > > > > >>>> > sys 32m18.028s
> > > > > > > > >>>> >
> > > > > > > > >>>> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does
not
> > > > > > > > >>>> > have
> > > > > > > > >>>> > the
> > > > > > > > >>>> > time
> > > > > > > > >>>> variable
> > > > > > > > >>>> > "t". So point2grid does not handle this case
with
> > > > > > > > >>>> > workaround.
> > > > > > > > >>>> >
> > > > > > > > >>>> > Cheers,
> > > > > > > > >>>> > Howard
> > > > > > > > >>>> >
> > > > > > > > >>>> > On Thu Jun 04 06:53:39 2020, ho-
> > > > > > > > >>>> > chun.huang at noaa.gov
> > > > > > > > >>>> > wrote:
> > > > > > > > >>>> > > Hi, John and Howard:
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > (1) You should be able to see my MET version
in
> > > > > > > > >>>> > > the
> > > > > > > > >>>> > > run
> > > > > > > > >>>> > > script :
> > > > > > > > 9.0
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > (2) Background information, we used to
discuss
> > > > > > > > >>>> > > between
> > > > > > > > >>>> > > NCAR
> > > > > > > > >>>> > > MET,
> > > > > > > > >>>> > > NESDIS,
> > > > > > > > >>>> > > and EMC about using VIIRS pixel AOD data per
> > > > > > > > >>>> > > granule
> > > > > > > > >>>> > > and
> > > > > > > > >>>> > > mapping
> > > > > > > > >>>> pixel
> > > > > > > > >>>> > > observed AOD into a selected model grid. It
> > > > > > > > >>>> > > has
> > > > > > > > >>>> > > been
> > > > > > > > >>>> > > determined
> > > > > > > > >>>> that
> > > > > > > > >>>> > > the
> > > > > > > > >>>> > > data volume is too large for MET to handle
VIIRS
> > > > > > > > >>>> > > granule
> > > > > > > > >>>> > > pixel
> > > > > > > > data
> > > > > > > > >>>> > > ingestion. EMC has asked NESDIS aerosol
groups
> > > > > > > > >>>> > > to
> > > > > > > > >>>> > > provide
> > > > > > > > >>>> > > *L3
> > > > > > > > >>>> > > "gridded"
> > > > > > > > >>>> > > nefCDF* files for global and regional AQM
model
> > > > > > > > >>>> > > evaluation/verification.
> > > > > > > > >>>> > > I am testing the regional one VIIRS-L3-
> > > > > > > > >>>> > > AOD_*AQM*_
> > > > > > > > 20200519_200000.nc
> > > > > > > > >>>> ,
> > > > > > > > >>>> > > while Partha Bhattacharjee is testing VIIRS-
L3-
> > > > > > > > >>>> > > AOD_*GEFS*_20200519_120000.nc
> > > > > > > > >>>> > > for global GEFS-aerosol verification
[reference
> > > > > > > > >>>> > > helpdesk
> > > > > > > > >>>> > > ticket
> > > > > > > > >>>> > > 95358].
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > (3) *I am not trying to using point2grid*, I
am
> > > > > > > > >>>> > > trying
> > > > > > > > >>>> > > to
> > > > > > > > >>>> > > use MET
> > > > > > > > >>>> > > regridding utilities to map a *gridded*
netCDF
> > > > > > > > >>>> > > to a
> > > > > > > > >>>> > > target
> > > > > > > > >>>> > > model
> > > > > > > > >>>> > > *gridded*
> > > > > > > > >>>> > > netCDF. Unless there is another better tool
in
> > > > > > > > >>>> > > MET
> > > > > > > > >>>> > > packages
> > > > > > > > that I
> > > > > > > > >>>> > > am not
> > > > > > > > >>>> > > aware of, I assume regrid_data_plane is for
> > > > > > > > >>>> > > general
> > > > > > > > >>>> > > mapping
> > > > > > > > purpose.
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > (4) For each hourly period, orbital
satellites
> > > > > > > > >>>> > > overpassed
> > > > > > > > >>>> > > only a
> > > > > > > > >>>> small
> > > > > > > > >>>> > > area
> > > > > > > > >>>> > > around the globe. I have asked NESDIS aerosol
to
> > > > > > > > >>>> > > use
> > > > > > > > >>>> > > fill-
> > > > > > > > >>>> > > values
> > > > > > > > for
> > > > > > > > >>>> > > non-observed grids and with compress level=1
> > > > > > > > >>>> > > hoping
> > > > > > > > >>>> regard-data_plane
> > > > > > > > >>>> > > can
> > > > > > > > >>>> > > skip those "-9999.f" grid quickly. I may be
> > > > > > > > >>>> > > wrong.
> > > > > > > > >>>> > > In
> > > > > > > > >>>> > > reality, I
> > > > > > > > >>>> may
> > > > > > > > >>>> > > only
> > > > > > > > >>>> > > use files from 18Z to 23Z for verification on
> > > > > > > > >>>> > > CONUS,
> > > > > > > > >>>> > > Alaska,
> > > > > > > > >>>> > > and
> > > > > > > > >>>> > > Hawaii.
> > > > > > > > >>>> > > The reason to keep it global coverage is to
> > > > > > > > >>>> > > simplify
> > > > > > > > >>>> > > NESDIS
> > > > > > > > >>>> production
> > > > > > > > >>>> > > processes and be flexible for future
> > > > > > > > >>>> > > applications.
> > > > > > > > >>>> > > Please
> > > > > > > > >>>> > > study my
> > > > > > > > >>>> > > problem,
> > > > > > > > >>>> > > if changes on the data production side
[NESDIS
> > > > > > > > >>>> > > aerosol
> > > > > > > > >>>> > > group] can
> > > > > > > > >>>> > > speed-up
> > > > > > > > >>>> > > the regridding processes, please let me know.
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > Thank you for your help.
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > Ho-Chun Huang
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > 5830 University Research Ct., Rm. 2792
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > College Park, MD 20740
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > 301-683-3958
> > > > > > > > >>>> > >
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley
> > > > > > > > >>>> > > Gotway
> > > > > > > > >>>> > > via RT
> > > > > > > > >>>> > > <met_help at ucar.edu>
> > > > > > > > >>>> > > wrote:
> > > > > > > > >>>> > >
> > > > > > > > >>>> > > > Howard,
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > > > OK, please find input data for testing
here:
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > > >
> > > > > > > > >>>> >
> > > > > > > > >>>>
> > > > > > > >
> > > > > >
> > > >
> >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > > > When you untar this file, you'll find the
> > > > > > > > >>>> > > > contents
> > > > > > > > >>>> > > > of
> > > > > > > > >>>> > > > Ho-
> > > > > > > > >>>> > > > Chun's
> > > > > > > > >>>> > > > directory:
> > > > > > > > >>>> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > > > > > > >>>> > > > Chun.Huang/plot/viirs
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > > > In addition, I copied the input/output
> > > > > > > > >>>> > > > directories
> > > > > > > > >>>> > > > from
> > > > > > > > >>>> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > > > > >>>> > > > Chun.Huang/VIIRS_AOD
> > > > > > > > >>>> > > > into
> > > > > > > > that
> > > > > > > > >>>> > > > viirs
> > > > > > > > >>>> > > > directory. Also, I included the file named
> > > > > > > > >>>> > > > "aqm.aot.148.grid"
> > > > > > > > >>>> which
> > > > > > > > >>>> > > > defines
> > > > > > > > >>>> > > > the target grid.
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > > > If you're able to take a look at this data
and
> > > > > > > > >>>> > > > see
> > > > > > > > >>>> > > > how
> > > > > > > > >>>> > > > it
> > > > > > > > >>>> > > > runs,
> > > > > > > > >>>> > > > that'd be
> > > > > > > > >>>> > > > great! Ho-Chun found that it took over 12
> > > > > > > > >>>> > > > hours!?
> > > > > > > > >>>> > > > to
> > > > > > > > >>>> > > > run
> > > > > > > > >>>> > > > on
> > > > > > > > WCOSS.
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > > > Thanks,
> > > > > > > > >>>> > > > John
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > > > On Wed, Jun 3, 2020 at 4:45 PM John Halley
> > > > > > > > >>>> > > > Gotway
> > > > > > > > >>>> > > > <
> > > > > > > > >>>> johnhg at ucar.edu>
> > > > > > > > >>>> > > > wrote:
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > > > > Howard,
> > > > > > > > >>>> > > > >
> > > > > > > > >>>> > > > > I'll go grab this data and copy it over
to
> > > > > > > > >>>> > > > > dakota.
> > > > > > > > >>>> > > > > Ho-
> > > > > > > > >>>> > > > > Chun,
> > > > > > > > >>>> sorry
> > > > > > > > >>>> > > > > for the
> > > > > > > > >>>> > > > > delay in getting started with this. I'll
go
> > > > > > > > >>>> > > > > grab
> > > > > > > > >>>> > > > > it
> > > > > > > > >>>> > > > > now.
> > > > > > > > >>>> > > > >
> > > > > > > > >>>> > > > > John
> > > > > > > > >>>> > > > >
> > > > > > > > >>>> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard Soh
> > > > > > > > >>>> > > > > via RT
> > > > > > > > >>>> > > > > <met_help at ucar.edu>
> > > > > > > > >>>> > > > > wrote:
> > > > > > > > >>>> > > > >
> > > > > > > > >>>> > > > >>
> > > > > > > > >>>> > > > >> <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > > > > >>>> >
> > > > > > > > >>>> > > > >>
> > > > > > > > >>>> > > > >> I don't have access to venus.
> > > > > > > > >>>> > > > >> Would you upload the files (config,
script,
> > > > > > > > >>>> > > > >> log,
> > > > > > > > >>>> > > > >> and
> > > > > > > > >>>> > > > >> data
> > > > > > > > >>>> files)
> > > > > > > > >>>> > > > >> to MET
> > > > > > > > >>>> > > > >> ftp server?
> > > > > > > > >>>> > > > >>
> > > > > > > > >>>> > > > >> Which MET version are you using?
> > > > > > > > >>>> > > > >>
> > > > > > > > >>>> > > > >> The GOES-AOD data was supported by
> > > > > > > > >>>> > > > >> regrid_data_plane
> > > > > > > > >>>> > > > >> (V8.x).
> > > > > > > > >>>> It's
> > > > > > > > >>>> > > > >> moved
> > > > > > > > >>>> > > > >> to point2grid (V9.0). The
regrid_data_plane
> > > > > > > > >>>> > > > >> still
> > > > > > > > >>>> > > > >> processes
> > > > > > > > >>>> GOES-
> > > > > > > > >>>> > > > >> AOD
> > > > > > > > >>>> > > > data
> > > > > > > > >>>> > > > >> in different way (interpolated based on
the
> > > > > > > > >>>> > > > >> neighbor
> > > > > > > > >>>> > > > >> data by
> > > > > > > > >>>> using
> > > > > > > > >>>> > > > >> width
> > > > > > > > >>>> > > > >> and shape, not by the physical
location).
> > > > > > > > >>>> > > > >> The
> > > > > > > > >>>> > > > >> output is
> > > > > > > > >>>> different
> > > > > > > > >>>> > > > >> from
> > > > > > > > >>>> > > > >> point2grid. VIIRS AOD is not supported
by
> > > > > > > > >>>> > > > >> MET,
> > > > > > > > >>>> > > > >> yet.
> > > > > > > > >>>> > > > >> There
> > > > > > > > >>>> might be
> > > > > > > > >>>> > > > >> a
> > > > > > > > >>>> > > > >> workaround.
> > > > > > > > >>>> > > > >>
> > > > > > > > >>>> > > > >> Cheers,
> > > > > > > > >>>> > > > >> Howard
> > > > > > > > >>>> > > > >>
> > > > > > > > >>>> > > > >> On Fri May 29 13:24:55 2020, ho-
> > > > > > > > >>>> > > > >> chun.huang at noaa.gov
> > > > > > > > >>>> > > > >> wrote:
> > > > > > > > >>>> > > > >> > To Whom It May Concern:
> > > > > > > > >>>> > > > >> >
> > > > > > > > >>>> > > > >> > I have an LatLon 0.03x0.03 satellite
> > > > > > > > >>>> > > > >> > observed
> > > > > > > > >>>> > > > >> > AOD
> > > > > > > > >>>> > > > >> > at
> > > > > > > > 20200519
> > > > > > > > >>>> > > > >> > 20Z
> > > > > > > > >>>> > > > >> > (global
> > > > > > > > >>>> > > > >> > but most with fill_value). I want to
use
> > > > > > > > >>>> > > > >> > regrid_data_plane
> > > > > > > > >>>> to
> > > > > > > > >>>> > > > >> > regrid
> > > > > > > > >>>> > > > >> > it to
> > > > > > > > >>>> > > > >> > CMAQ Lambert Conformal Grid. But it
> > > > > > > > >>>> > > > >> > seems to
> > > > > > > > >>>> > > > >> > hang
> > > > > > > > >>>> > > > >> > during
> > > > > > > > the
> > > > > > > > >>>> > > > >> > Interpolation
> > > > > > > > >>>> > > > >> > step.
> > > > > > > > >>>> > > > >> >
> > > > > > > > >>>> > > > >> > Can you find out what went wrong?
> > > > > > > > >>>> > > > >> > Sometimes
> > > > > > > > >>>> > > > >> > it
> > > > > > > > >>>> > > > >> > is
> > > > > > > > >>>> > > > >> > more
> > > > > > > > than
> > > > > > > > >>>> an
> > > > > > > > >>>> > > > >> > hour
> > > > > > > > >>>> > > > >> > waiting before I kill it.
> > > > > > > > >>>> > > > >> >
> > > > > > > > >>>> > > > >> > runscript in
> > > > > > > > >>>> > > > >> >
venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > > > > > > >>>> > > > >> > Chun.Huang/plot/viirs/
> > > > > > > > >>>> > > > >> >
> > > > > > > > >>>> > > > >> > with command > bash
run.viirs_aqm_aod.sh
> > > > > > > > >>>> > > > >> > viirs
> > > > > > > > >>>> > > > >> > aqm
> > > > > > > > 20200519
> > > > > > > > >>>> > > > >> > 20200519
> > > > > > > > >>>> > > > >
> > > > > > > > >>>> > > > >> > regrid.log 2>&1 &
> > > > > > > > >>>> > > > >> >
> > > > > > > > >>>> > > > >> > You can check the "regrid.log" in the
> > > > > > > > >>>> > > > >> > script
> > > > > > > > >>>> > > > >> > directory for
> > > > > > > > >>>> > > > >> > input,
> > > > > > > > >>>> > > > >> > model_grid, and output file location
on
> > > > > > > > >>>> > > > >> > venus. I
> > > > > > > > >>>> > > > >> > will keep
> > > > > > > > >>>> it
> > > > > > > > >>>> > > > >> > going
> > > > > > > > >>>> > > > >> > from May 29 19:15Z.
> > > > > > > > >>>> > > > >> >
> > > > > > > > >>>> > > > >> > One thing I do not understand is that
the
> > > > > > > > >>>> > > > >> > output
> > > > > > > > >>>> > > > >> > file
> > > > > > > > >>>> > > > >> > is
> > > > > > > > >>>> already
> > > > > > > > >>>> > > > >> > generated
> > > > > > > > >>>> > > > >> > while the job is still
> > > > > > > > >>>> > > > >> > running.
> > > > > > > > >>>> > > > >> > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > > > > >>>> > > > >> >
Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-
> > > > > > > > >>>> > > > >> > L3-
> > > > > > > > >>>> > > > >> > AOD_AQM_aqm_20200519_20.nc
> > > > > > > > >>>> > > > >> >
> > > > > > > > >>>> > > > >> > Ho-Chun Huang
> > > > > > > > >>>> > > > >> >
> > > > > > > > >>>> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > > >>>> > > > >> >
> > > > > > > > >>>> > > > >> > 5830 University Research Ct., Rm. 2792
> > > > > > > > >>>> > > > >> >
> > > > > > > > >>>> > > > >> > College Park, MD 20740
> > > > > > > > >>>> > > > >> >
> > > > > > > > >>>> > > > >> > Ho-Chun.Huang at noaa.gov
> > > > > > > > >>>> > > > >> > <Joe.Smith at noaa.gov>
> > > > > > > > >>>> > > > >> >
> > > > > > > > >>>> > > > >> > 301-683-3958
> > > > > > > > >>>> > > > >>
> > > > > > > > >>>> > > > >>
> > > > > > > > >>>> > > > >>
> > > > > > > > >>>> > > > >>
> > > > > > > > >>>> > > >
> > > > > > > > >>>> > > >
> > > > > > > > >>>> >
> > > > > > > > >>>> >
> > > > > > > > >>>> >
> > > > > > > > >>>> >
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > >
> >
> >
> >
> >
------------------------------------------------
Subject: hang on regrid_data_plane
From: Ho-Chun Huang - NOAA Affiliate
Time: Tue Jun 16 13:43:27 2020
Thanks Howard. You can close the ticket.
Ho-Chun Huang
IMSG at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2792
College Park, MD 20740
Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
301-683-3958
On Tue, Jun 16, 2020 at 2:56 PM Howard Soh via RT <met_help at ucar.edu>
wrote:
> It will be available with next beta release.
>
> John tested this at kiowa
> (kiowa:/d1/projects/MET/MET_pull_requests/met-
9.1_beta2/feature_1363)
>
> Cheers,
> Howard
>
> On Tue Jun 16 11:02:04 2020, ho-chun.huang at noaa.gov wrote:
> > Hi,
> >
> > Does it mean a new updated regrdi_data_plane is available for
testing?
> > will
> > be released in 9.1_beta??, or else? Please advise.
> >
> > If there is nothing else you need to do, please let me know about
the
> > status of testing new utilities and you can close this ticket.
> >
> > Ho-Chun Huang
> >
> > IMSG at NOAA/NWS/NCEP/EMC
> >
> > 5830 University Research Ct., Rm. 2792
> >
> > College Park, MD 20740
> >
> > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> >
> > 301-683-3958
> >
> >
> > On Wed, Jun 10, 2020 at 2:12 PM Howard Soh via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > The issue at github was created
> > > (https://github.com/NCAR/MET/issues/1363).
> > >
> > > The main saving is from reading the compressed variable. The
other
> > > saving
> > > is from building a data plane (72M calls to 120K calls). There
is no
> > > significant difference with compression level (between 1 and 9)
with
> > > the
> > > enhanced MET.
> > >
> > > Cheers,
> > > Howard
> > >
> > > Cheers,
> > > Howard
> > > On Tue Jun 09 10:53:07 2020, ho-chun.huang at noaa.gov wrote:
> > > > Hi, Howard:
> > > >
> > > > Thanks. I see, as long as there is a compression applied to
the
> > > > netcdf
> > > > file, regrid_data_plane has to decompress each grid [for 6000
> > > > times].
> > > > C0
> > > > to C1 is a big reduction and I hope to be able to use
compression
> > > > level 1
> > > > for VIIRS L3 AOD file.
> > > >
> > > > Ho-Chun Huang
> > > >
> > > > IMSG at NOAA/NWS/NCEP/EMC
> > > >
> > > > 5830 University Research Ct., Rm. 2792
> > > >
> > > > College Park, MD 20740
> > > >
> > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > >
> > > > 301-683-3958
> > > >
> > > >
> > > > On Tue, Jun 9, 2020 at 12:33 PM Howard Soh via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > The compression is a variable based. NetCDF API reads a
whole
> > > > > data,
> > > > > decompresses & slices. With a quick test, it seems to be no
big
> > > > > difference
> > > > > betwend compress level 1 and 2.
> > > > >
> > > > > File size (by nccopy)
> > > > > 8.1G VIIRS-L3-AOD_AQM_20200519_200000.c0.nc (no compression)
> > > > > 86M VIIRS-L3-AOD_AQM_20200519_200000.c1.nc (compress 1)
> > > > > 86M VIIRS-L3-AOD_AQM_20200519_200000.c2.nc (compress 2)
> > > > > 54M VIIRS-L3-AOD_AQM_20200519_200000.nc (compress 9)
> > > > >
> > > > > Wall time comparison with plot_data_plane (which reads 6000
> > > > > times) at
> > > > > dakota (time is not accurate because the system load is not
> > > > > consistent):
> > > > >
> > > > > c0: 0m44.899s
> > > > > c1: 57m33.530s
> > > > > c2: 57m22.974s
> > > > >
> > > > > Cheers,
> > > > > Howard
> > > > >
> > > > > On Mon Jun 08 15:55:29 2020, ho-chun.huang at noaa.gov wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I thought I asked the compress level=1 at the beginning.
I
> > > > > > will
> > > > > > double
> > > > > > check with NESIDS again, but can you tell me now at what
nc
> > > > > > compress
> > > > > > level
> > > > > > that regrid_data_plane does not need to decompress the
variable
> > > > > > for
> > > > > > each
> > > > > > grid. If it can resolved from the data source level, it
won't
> > > > > > waste
> > > > > > your
> > > > > > time for the special case.
> > > > > >
> > > > > > Ho-Chun Huang
> > > > > >
> > > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > >
> > > > > > 5830 University Research Ct., Rm. 2792
> > > > > >
> > > > > > College Park, MD 20740
> > > > > >
> > > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > >
> > > > > > 301-683-3958
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 8, 2020 at 5:35 PM Howard Soh via RT
> > > > > > <met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > The first cause is reading the variable 6000 times
(dimension
> > > > > > > is
> > > > > > > 10000 by
> > > > > > > 6000). The variables are compressed with with level 9
> > > > > > > (highest).
> > > > > > > It
> > > > > > > taook
> > > > > > > more than 1 second to read & decompress a variable at
dakota.
> > > > > > > if 1 seconds, 1 * 6000 = 6000 seconds = 100 minutes
> > > > > > > if 1.5 seconds, 1.5 * 6000 = 9000 seconds = 150 minutes.
> > > > > > >
> > > > > > > The first step is reading a variable one time instead of
6000
> > > > > > > times.
> > > > > > > But
> > > > > > > the output did not match, so I'm working on making the
same
> > > > > > > result.
> > > > > > > Once
> > > > > > > this is done, I will check if there is more room to
speed up
> > > > > > > (some
> > > > > > > steps
> > > > > > > are executed 60M times).
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Howard
> > > > > > >
> > > > > > > On Mon Jun 08 15:05:35 2020, johnhg wrote:
> > > > > > > > Howard and Ho-Chun,
> > > > > > > >
> > > > > > > > I'm wondering if we have any updates on this ticket?
> > > > > > > > Howard,
> > > > > > > > have
> > > > > > > > you
> > > > > > > > been
> > > > > > > > able to find a reason for this running so slowly?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Thu, Jun 4, 2020 at 10:55 AM Ho-Chun Huang - NOAA
> > > > > > > > Affiliate
> > > > > > > > via
> > > > > > > > RT
> > > > > > > > <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi, John and Howard:
> > > > > > > > >
> > > > > > > > > *Ho-Chun, do you know if the geometry of the VIIRS
data
> > > > > > > > > repeats
> > > > > > > > > day
> > > > > > > > > after
> > > > > > > > > day?*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > There are two satellites currently providing VIIRS
AOD
> > > > > > > > > observations,
> > > > > > > > > SUOMI-NPP and NOAA-20. Eash satellite overpasses
the
> > > > > > > > > same
> > > > > > > > > area
> > > > > > > > > twice
> > > > > > > > > per
> > > > > > > > > day, AOD only observed during daytime overpass. The
> > > > > > > > > overpass
> > > > > > > > > for
> > > > > > > > > local time
> > > > > > > > > should be similar per satellite, NOAA-20 is ~ 50
mins
> > > > > > > > > ahead
> > > > > > > > > of
> > > > > > > > > SUOMI-
> > > > > > > > > NPP.
> > > > > > > > >
> > > > > > > > > Although I think daily granule pixel location per
> > > > > > > > > satellite
> > > > > > > > > may
> > > > > > > > > shift
> > > > > > > > > slightly day after day (still need to verify this
> > > > > > > > > statement
> > > > > > > > > with
> > > > > > > > > NESDIS
> > > > > > > > > personal)? Thus, for L3 gridded AOD data, *the area
of
> > > > > > > > > coverage*
> > > > > > > > > of
> > > > > > > > > orbital satellites should be similar *for the same
hour
> > > > > > > > > of
> > > > > > > > > day*.
> > > > > > > > > Maybe +-
> > > > > > > > > few grid boxes toward the north and west for
predefined
> > > > > > > > > search
> > > > > > > > > area.
> > > > > > > > > Thus,
> > > > > > > > > if you put a predefined search box with an adequate
> > > > > > > > > buffer
> > > > > > > > > zone
> > > > > > > > > on 4
> > > > > > > > > sides
> > > > > > > > > for specific UTC, maybe it will work.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Ho-Chun Huang
> > > > > > > > >
> > > > > > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > > >
> > > > > > > > > 5830 University Research Ct., Rm. 2792
> > > > > > > > >
> > > > > > > > > College Park, MD 20740
> > > > > > > > >
> > > > > > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > > >
> > > > > > > > > 301-683-3958
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jun 4, 2020 at 12:00 PM Ho-Chun Huang - NOAA
> > > > > > > > > Affiliate <
> > > > > > > > > ho-chun.huang at noaa.gov> wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > Sorry for the wrong figure, attached is the direct
read
> > > > > > > > > > from
> > > > > > > > > > NESDIS
> > > > > > > > > > VIIRS
> > > > > > > > > > L3 AOD file.
> > > > > > > > > >
> > > > > > > > > > Ho-Chun Huang
> > > > > > > > > >
> > > > > > > > > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > > > >
> > > > > > > > > > 5830 University Research Ct., Rm. 2792
> > > > > > > > > >
> > > > > > > > > > College Park, MD 20740
> > > > > > > > > >
> > > > > > > > > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > > > >
> > > > > > > > > > 301-683-3958
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Jun 4, 2020 at 11:57 AM Ho-Chun Huang -
NOAA
> > > > > > > > > > Affiliate
> > > > > > > > > > <
> > > > > > > > > > ho-chun.huang at noaa.gov> wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi,
> > > > > > > > > >>
> > > > > > > > > >> The regrid_data_plane already obtained the LatLon
> > > > > > > > > >> projection
> > > > > > > > > >> information
> > > > > > > > > >> from input obs file
> > > > > > > > > >>
> > > > > > > > > >> DEBUG 1: Reading data file:
> > > > > > > > > >>
> > > > > > > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > > > > > Chun.Huang/VIIRS_AOD/AOD/20200519/VIIRS-L3-
> > > > > > > > > AOD_AQM_20200519_200000.nc
> > > > > > > > > >> DEBUG 4:
Met2dDataFileFactory::new_met_2d_data_file()
> > > > > > > > > >> ->
> > > > > > > > > >> created
> > > > > > > > > >> new
> > > > > > > > > >> Met2dDataFile object of type "FileType_NcMet".
> > > > > > > > > >> DEBUG 4:
> > > > > > > > > >> DEBUG 4: Latitude/Longitude Grid Data:
> > > > > > > > > >> DEBUG 4: lat_ll: -89.985
> > > > > > > > > >> DEBUG 4: lon_ll: 179.985
> > > > > > > > > >> DEBUG 4: delta_lat: 0.03
> > > > > > > > > >> DEBUG 4: delta_lon: 0.03
> > > > > > > > > >> DEBUG 4: Nlat: 6000
> > > > > > > > > >> DEBUG 4: Nlon: 12000
> > > > > > > > > >>
> > > > > > > > > >> Ho-Chun Huang
> > > > > > > > > >>
> > > > > > > > > >> IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > > > >>
> > > > > > > > > >> 5830 University Research Ct., Rm. 2792
> > > > > > > > > >>
> > > > > > > > > >> College Park, MD 20740
> > > > > > > > > >>
> > > > > > > > > >> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > > > >>
> > > > > > > > > >> 301-683-3958
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Thu, Jun 4, 2020 at 11:52 AM Ho-Chun Huang -
NOAA
> > > > > > > > > >> Affiliate
> > > > > > > > > >> <
> > > > > > > > > >> ho-chun.huang at noaa.gov> wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Hi, Howard and John:
> > > > > > > > > >>>
> > > > > > > > > >>> The VIIRS L3 AOD is on fixed LatLon 0.03x0.03
grid,
> > > > > > > > > >>> so
> > > > > > > > > >>> its
> > > > > > > > > >>> location is
> > > > > > > > > >>> fixed everyday. Only the area of observed data
will
> > > > > > > > > >>> change
> > > > > > > > > >>> for
> > > > > > > > > >>> each
> > > > > > > > > hours,
> > > > > > > > > >>> in general moving northward and westward.
> > > > > > > > > >>>
> > > > > > > > > >>> This is graphic based on direct read of the
input
> > > > > > > > > >>> VIIRS
> > > > > > > > > >>> L3
> > > > > > > > > >>> AOD
> > > > > > > > > >>> file,
> > > > > > > > > for
> > > > > > > > > >>> your reference,
> > > > > > > > > >>>
> > > > > > > > > >>> I want to emphasize again, the input VIIRS AOD
is
> > > > > > > > > >>> gridded
> > > > > > > > > >>> and
> > > > > > > > > >>> *NOT*
> > > > > > > > > >>> pixel data, you do not need to recompute the
geo-
> > > > > > > > > >>> location
> > > > > > > > > >>> for
> > > > > > > > > >>> each
> > > > > > > > > -field
> > > > > > > > > >>> variables.
> > > > > > > > > >>>
> > > > > > > > > >>> Ho-Chun Huang
> > > > > > > > > >>>
> > > > > > > > > >>> IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > > > >>>
> > > > > > > > > >>> 5830 University Research Ct., Rm. 2792
> > > > > > > > > >>>
> > > > > > > > > >>> College Park, MD 20740
> > > > > > > > > >>>
> > > > > > > > > >>> Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > > > >>>
> > > > > > > > > >>> 301-683-3958
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Thu, Jun 4, 2020 at 11:39 AM John Halley
Gotway
> > > > > > > > > >>> via RT
> > > > > > > > > >>> <
> > > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> Howard,
> > > > > > > > > >>>>
> > > > > > > > > >>>> I recall that with the GOES files, you set up
> > > > > > > > > >>>> regrid_data_plane
> > > > > > > > > >>>> to
> > > > > > > > > >>>> determine the grid mapping the first time it's
run,
> > > > > > > > > >>>> and
> > > > > > > > > >>>> then
> > > > > > > > > >>>> reuse
> > > > > > > > > that
> > > > > > > > > >>>> mapping information in subsequent calls. I
really
> > > > > > > > > >>>> don't
> > > > > > > > > >>>> know
> > > > > > > > > >>>> anything
> > > > > > > > > >>>> about
> > > > > > > > > >>>> the VIIRS data. Searching online, it looks like
the
> > > > > > > > > >>>> VIIRS
> > > > > > > > > >>>> data
> > > > > > > > > >>>> comes
> > > > > > > > > in
> > > > > > > > > >>>> swaths, whereas the GOES data is stationary. So
> > > > > > > > > >>>> perhaps
> > > > > > > > > >>>> the
> > > > > > > > > >>>> approach
> > > > > > > > > of
> > > > > > > > > >>>> pre-computing the grid mappings won't work. I
wonder
> > > > > > > > > >>>> if
> > > > > > > > > >>>> the
> > > > > > > > > >>>> swaths are
> > > > > > > > > >>>> repeated though.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Ho-Chun, do you know if the geometry of the
VIIRS
> > > > > > > > > >>>> data
> > > > > > > > > >>>> repeats
> > > > > > > > > >>>> day
> > > > > > > > > after
> > > > > > > > > >>>> day? If we can figure out a way of computing
the
> > > > > > > > > >>>> grid
> > > > > > > > > >>>> mapping
> > > > > > > > > >>>> information
> > > > > > > > > >>>> once ahead of time and then reuse that
information,
> > > > > > > > > >>>> I'm
> > > > > > > > > >>>> guessing
> > > > > > > > > >>>> the
> > > > > > > > > >>>> tool
> > > > > > > > > >>>> would run much faster.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Thanks,
> > > > > > > > > >>>> John
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Thu, Jun 4, 2020 at 8:48 AM Howard Soh via
RT
> > > > > > > > > >>>> <met_help at ucar.edu>
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>
> > > > > > > > > >>>> >
> > > > > > > > > >>>> > <URL:
> > > > > > > > > >>>> >
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> >
> > > > > > > > > >>>> > I will investigate why it takes too long.
> > > > > > > > > >>>> >
> > > > > > > > > >>>> > I ran regrid_data_plane for just one variable
> > > > > > > > > >>>> > (AOD_H_Quality,
> > > > > > > > > >>>> dimension is
> > > > > > > > > >>>> > 6000 by 12000) and it took about 4 hours at
> > > > > > > > > >>>> > dakota.
> > > > > > > > > >>>> >
> > > > > > > > > >>>> > DEBUG 1: Writing output file:
> > > > > > > > > >>>> VIIRS-L3-AOD_AQM_aqm_20200519_20.regrided.nc
> > > > > > > > > >>>> >
> > > > > > > > > >>>> > real 239m2.552s
> > > > > > > > > >>>> > user 206m42.332s
> > > > > > > > > >>>> > sys 32m18.028s
> > > > > > > > > >>>> >
> > > > > > > > > >>>> > The VIIRS-L3-AOD_AQM_20200519_200000.nc does
not
> > > > > > > > > >>>> > have
> > > > > > > > > >>>> > the
> > > > > > > > > >>>> > time
> > > > > > > > > >>>> variable
> > > > > > > > > >>>> > "t". So point2grid does not handle this case
with
> > > > > > > > > >>>> > workaround.
> > > > > > > > > >>>> >
> > > > > > > > > >>>> > Cheers,
> > > > > > > > > >>>> > Howard
> > > > > > > > > >>>> >
> > > > > > > > > >>>> > On Thu Jun 04 06:53:39 2020, ho-
> > > > > > > > > >>>> > chun.huang at noaa.gov
> > > > > > > > > >>>> > wrote:
> > > > > > > > > >>>> > > Hi, John and Howard:
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > (1) You should be able to see my MET
version in
> > > > > > > > > >>>> > > the
> > > > > > > > > >>>> > > run
> > > > > > > > > >>>> > > script :
> > > > > > > > > 9.0
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > (2) Background information, we used to
discuss
> > > > > > > > > >>>> > > between
> > > > > > > > > >>>> > > NCAR
> > > > > > > > > >>>> > > MET,
> > > > > > > > > >>>> > > NESDIS,
> > > > > > > > > >>>> > > and EMC about using VIIRS pixel AOD data
per
> > > > > > > > > >>>> > > granule
> > > > > > > > > >>>> > > and
> > > > > > > > > >>>> > > mapping
> > > > > > > > > >>>> pixel
> > > > > > > > > >>>> > > observed AOD into a selected model grid.
It
> > > > > > > > > >>>> > > has
> > > > > > > > > >>>> > > been
> > > > > > > > > >>>> > > determined
> > > > > > > > > >>>> that
> > > > > > > > > >>>> > > the
> > > > > > > > > >>>> > > data volume is too large for MET to handle
VIIRS
> > > > > > > > > >>>> > > granule
> > > > > > > > > >>>> > > pixel
> > > > > > > > > data
> > > > > > > > > >>>> > > ingestion. EMC has asked NESDIS aerosol
groups
> > > > > > > > > >>>> > > to
> > > > > > > > > >>>> > > provide
> > > > > > > > > >>>> > > *L3
> > > > > > > > > >>>> > > "gridded"
> > > > > > > > > >>>> > > nefCDF* files for global and regional AQM
model
> > > > > > > > > >>>> > > evaluation/verification.
> > > > > > > > > >>>> > > I am testing the regional one VIIRS-L3-
> > > > > > > > > >>>> > > AOD_*AQM*_
> > > > > > > > > 20200519_200000.nc
> > > > > > > > > >>>> ,
> > > > > > > > > >>>> > > while Partha Bhattacharjee is testing
VIIRS-L3-
> > > > > > > > > >>>> > > AOD_*GEFS*_20200519_120000.nc
> > > > > > > > > >>>> > > for global GEFS-aerosol verification
[reference
> > > > > > > > > >>>> > > helpdesk
> > > > > > > > > >>>> > > ticket
> > > > > > > > > >>>> > > 95358].
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > (3) *I am not trying to using point2grid*,
I am
> > > > > > > > > >>>> > > trying
> > > > > > > > > >>>> > > to
> > > > > > > > > >>>> > > use MET
> > > > > > > > > >>>> > > regridding utilities to map a *gridded*
netCDF
> > > > > > > > > >>>> > > to a
> > > > > > > > > >>>> > > target
> > > > > > > > > >>>> > > model
> > > > > > > > > >>>> > > *gridded*
> > > > > > > > > >>>> > > netCDF. Unless there is another better
tool in
> > > > > > > > > >>>> > > MET
> > > > > > > > > >>>> > > packages
> > > > > > > > > that I
> > > > > > > > > >>>> > > am not
> > > > > > > > > >>>> > > aware of, I assume regrid_data_plane is for
> > > > > > > > > >>>> > > general
> > > > > > > > > >>>> > > mapping
> > > > > > > > > purpose.
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > (4) For each hourly period, orbital
satellites
> > > > > > > > > >>>> > > overpassed
> > > > > > > > > >>>> > > only a
> > > > > > > > > >>>> small
> > > > > > > > > >>>> > > area
> > > > > > > > > >>>> > > around the globe. I have asked NESDIS
aerosol to
> > > > > > > > > >>>> > > use
> > > > > > > > > >>>> > > fill-
> > > > > > > > > >>>> > > values
> > > > > > > > > for
> > > > > > > > > >>>> > > non-observed grids and with compress
level=1
> > > > > > > > > >>>> > > hoping
> > > > > > > > > >>>> regard-data_plane
> > > > > > > > > >>>> > > can
> > > > > > > > > >>>> > > skip those "-9999.f" grid quickly. I may
be
> > > > > > > > > >>>> > > wrong.
> > > > > > > > > >>>> > > In
> > > > > > > > > >>>> > > reality, I
> > > > > > > > > >>>> may
> > > > > > > > > >>>> > > only
> > > > > > > > > >>>> > > use files from 18Z to 23Z for verification
on
> > > > > > > > > >>>> > > CONUS,
> > > > > > > > > >>>> > > Alaska,
> > > > > > > > > >>>> > > and
> > > > > > > > > >>>> > > Hawaii.
> > > > > > > > > >>>> > > The reason to keep it global coverage is to
> > > > > > > > > >>>> > > simplify
> > > > > > > > > >>>> > > NESDIS
> > > > > > > > > >>>> production
> > > > > > > > > >>>> > > processes and be flexible for future
> > > > > > > > > >>>> > > applications.
> > > > > > > > > >>>> > > Please
> > > > > > > > > >>>> > > study my
> > > > > > > > > >>>> > > problem,
> > > > > > > > > >>>> > > if changes on the data production side
[NESDIS
> > > > > > > > > >>>> > > aerosol
> > > > > > > > > >>>> > > group] can
> > > > > > > > > >>>> > > speed-up
> > > > > > > > > >>>> > > the regridding processes, please let me
know.
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > Thank you for your help.
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > Ho-Chun Huang
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > 5830 University Research Ct., Rm. 2792
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > College Park, MD 20740
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > Ho-Chun.Huang at noaa.gov <Joe.Smith at noaa.gov>
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > 301-683-3958
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > On Wed, Jun 3, 2020 at 7:05 PM John Halley
> > > > > > > > > >>>> > > Gotway
> > > > > > > > > >>>> > > via RT
> > > > > > > > > >>>> > > <met_help at ucar.edu>
> > > > > > > > > >>>> > > wrote:
> > > > > > > > > >>>> > >
> > > > > > > > > >>>> > > > Howard,
> > > > > > > > > >>>> > > >
> > > > > > > > > >>>> > > > OK, please find input data for testing
here:
> > > > > > > > > >>>> > > >
> > > > > > > > > >>>> > > >
> > > > > > > > > >>>> >
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/huang_data_20200603/viirs.tar.gz
> > > > > > > > > >>>> > > >
> > > > > > > > > >>>> > > > When you untar this file, you'll find the
> > > > > > > > > >>>> > > > contents
> > > > > > > > > >>>> > > > of
> > > > > > > > > >>>> > > > Ho-
> > > > > > > > > >>>> > > > Chun's
> > > > > > > > > >>>> > > > directory:
> > > > > > > > > >>>> > > > venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > > > > > > > >>>> > > > Chun.Huang/plot/viirs
> > > > > > > > > >>>> > > >
> > > > > > > > > >>>> > > > In addition, I copied the input/output
> > > > > > > > > >>>> > > > directories
> > > > > > > > > >>>> > > > from
> > > > > > > > > >>>> > > > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > > > > > >>>> > > > Chun.Huang/VIIRS_AOD
> > > > > > > > > >>>> > > > into
> > > > > > > > > that
> > > > > > > > > >>>> > > > viirs
> > > > > > > > > >>>> > > > directory. Also, I included the file
named
> > > > > > > > > >>>> > > > "aqm.aot.148.grid"
> > > > > > > > > >>>> which
> > > > > > > > > >>>> > > > defines
> > > > > > > > > >>>> > > > the target grid.
> > > > > > > > > >>>> > > >
> > > > > > > > > >>>> > > > If you're able to take a look at this
data and
> > > > > > > > > >>>> > > > see
> > > > > > > > > >>>> > > > how
> > > > > > > > > >>>> > > > it
> > > > > > > > > >>>> > > > runs,
> > > > > > > > > >>>> > > > that'd be
> > > > > > > > > >>>> > > > great! Ho-Chun found that it took over 12
> > > > > > > > > >>>> > > > hours!?
> > > > > > > > > >>>> > > > to
> > > > > > > > > >>>> > > > run
> > > > > > > > > >>>> > > > on
> > > > > > > > > WCOSS.
> > > > > > > > > >>>> > > >
> > > > > > > > > >>>> > > > Thanks,
> > > > > > > > > >>>> > > > John
> > > > > > > > > >>>> > > >
> > > > > > > > > >>>> > > > On Wed, Jun 3, 2020 at 4:45 PM John
Halley
> > > > > > > > > >>>> > > > Gotway
> > > > > > > > > >>>> > > > <
> > > > > > > > > >>>> johnhg at ucar.edu>
> > > > > > > > > >>>> > > > wrote:
> > > > > > > > > >>>> > > >
> > > > > > > > > >>>> > > > > Howard,
> > > > > > > > > >>>> > > > >
> > > > > > > > > >>>> > > > > I'll go grab this data and copy it over
to
> > > > > > > > > >>>> > > > > dakota.
> > > > > > > > > >>>> > > > > Ho-
> > > > > > > > > >>>> > > > > Chun,
> > > > > > > > > >>>> sorry
> > > > > > > > > >>>> > > > > for the
> > > > > > > > > >>>> > > > > delay in getting started with this.
I'll go
> > > > > > > > > >>>> > > > > grab
> > > > > > > > > >>>> > > > > it
> > > > > > > > > >>>> > > > > now.
> > > > > > > > > >>>> > > > >
> > > > > > > > > >>>> > > > > John
> > > > > > > > > >>>> > > > >
> > > > > > > > > >>>> > > > > On Wed, Jun 3, 2020 at 3:41 PM Howard
Soh
> > > > > > > > > >>>> > > > > via RT
> > > > > > > > > >>>> > > > > <met_help at ucar.edu>
> > > > > > > > > >>>> > > > > wrote:
> > > > > > > > > >>>> > > > >
> > > > > > > > > >>>> > > > >>
> > > > > > > > > >>>> > > > >> <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95411
> > > > > > > > > >>>> >
> > > > > > > > > >>>> > > > >>
> > > > > > > > > >>>> > > > >> I don't have access to venus.
> > > > > > > > > >>>> > > > >> Would you upload the files (config,
script,
> > > > > > > > > >>>> > > > >> log,
> > > > > > > > > >>>> > > > >> and
> > > > > > > > > >>>> > > > >> data
> > > > > > > > > >>>> files)
> > > > > > > > > >>>> > > > >> to MET
> > > > > > > > > >>>> > > > >> ftp server?
> > > > > > > > > >>>> > > > >>
> > > > > > > > > >>>> > > > >> Which MET version are you using?
> > > > > > > > > >>>> > > > >>
> > > > > > > > > >>>> > > > >> The GOES-AOD data was supported by
> > > > > > > > > >>>> > > > >> regrid_data_plane
> > > > > > > > > >>>> > > > >> (V8.x).
> > > > > > > > > >>>> It's
> > > > > > > > > >>>> > > > >> moved
> > > > > > > > > >>>> > > > >> to point2grid (V9.0). The
regrid_data_plane
> > > > > > > > > >>>> > > > >> still
> > > > > > > > > >>>> > > > >> processes
> > > > > > > > > >>>> GOES-
> > > > > > > > > >>>> > > > >> AOD
> > > > > > > > > >>>> > > > data
> > > > > > > > > >>>> > > > >> in different way (interpolated based
on the
> > > > > > > > > >>>> > > > >> neighbor
> > > > > > > > > >>>> > > > >> data by
> > > > > > > > > >>>> using
> > > > > > > > > >>>> > > > >> width
> > > > > > > > > >>>> > > > >> and shape, not by the physical
location).
> > > > > > > > > >>>> > > > >> The
> > > > > > > > > >>>> > > > >> output is
> > > > > > > > > >>>> different
> > > > > > > > > >>>> > > > >> from
> > > > > > > > > >>>> > > > >> point2grid. VIIRS AOD is not supported
by
> > > > > > > > > >>>> > > > >> MET,
> > > > > > > > > >>>> > > > >> yet.
> > > > > > > > > >>>> > > > >> There
> > > > > > > > > >>>> might be
> > > > > > > > > >>>> > > > >> a
> > > > > > > > > >>>> > > > >> workaround.
> > > > > > > > > >>>> > > > >>
> > > > > > > > > >>>> > > > >> Cheers,
> > > > > > > > > >>>> > > > >> Howard
> > > > > > > > > >>>> > > > >>
> > > > > > > > > >>>> > > > >> On Fri May 29 13:24:55 2020, ho-
> > > > > > > > > >>>> > > > >> chun.huang at noaa.gov
> > > > > > > > > >>>> > > > >> wrote:
> > > > > > > > > >>>> > > > >> > To Whom It May Concern:
> > > > > > > > > >>>> > > > >> >
> > > > > > > > > >>>> > > > >> > I have an LatLon 0.03x0.03 satellite
> > > > > > > > > >>>> > > > >> > observed
> > > > > > > > > >>>> > > > >> > AOD
> > > > > > > > > >>>> > > > >> > at
> > > > > > > > > 20200519
> > > > > > > > > >>>> > > > >> > 20Z
> > > > > > > > > >>>> > > > >> > (global
> > > > > > > > > >>>> > > > >> > but most with fill_value). I want
to use
> > > > > > > > > >>>> > > > >> > regrid_data_plane
> > > > > > > > > >>>> to
> > > > > > > > > >>>> > > > >> > regrid
> > > > > > > > > >>>> > > > >> > it to
> > > > > > > > > >>>> > > > >> > CMAQ Lambert Conformal Grid. But it
> > > > > > > > > >>>> > > > >> > seems to
> > > > > > > > > >>>> > > > >> > hang
> > > > > > > > > >>>> > > > >> > during
> > > > > > > > > the
> > > > > > > > > >>>> > > > >> > Interpolation
> > > > > > > > > >>>> > > > >> > step.
> > > > > > > > > >>>> > > > >> >
> > > > > > > > > >>>> > > > >> > Can you find out what went wrong?
> > > > > > > > > >>>> > > > >> > Sometimes
> > > > > > > > > >>>> > > > >> > it
> > > > > > > > > >>>> > > > >> > is
> > > > > > > > > >>>> > > > >> > more
> > > > > > > > > than
> > > > > > > > > >>>> an
> > > > > > > > > >>>> > > > >> > hour
> > > > > > > > > >>>> > > > >> > waiting before I kill it.
> > > > > > > > > >>>> > > > >> >
> > > > > > > > > >>>> > > > >> > runscript in
> > > > > > > > > >>>> > > > >> >
venus:/gpfs/dell2/emc/modeling/save/Ho-
> > > > > > > > > >>>> > > > >> > Chun.Huang/plot/viirs/
> > > > > > > > > >>>> > > > >> >
> > > > > > > > > >>>> > > > >> > with command > bash
run.viirs_aqm_aod.sh
> > > > > > > > > >>>> > > > >> > viirs
> > > > > > > > > >>>> > > > >> > aqm
> > > > > > > > > 20200519
> > > > > > > > > >>>> > > > >> > 20200519
> > > > > > > > > >>>> > > > >
> > > > > > > > > >>>> > > > >> > regrid.log 2>&1 &
> > > > > > > > > >>>> > > > >> >
> > > > > > > > > >>>> > > > >> > You can check the "regrid.log" in
the
> > > > > > > > > >>>> > > > >> > script
> > > > > > > > > >>>> > > > >> > directory for
> > > > > > > > > >>>> > > > >> > input,
> > > > > > > > > >>>> > > > >> > model_grid, and output file location
on
> > > > > > > > > >>>> > > > >> > venus. I
> > > > > > > > > >>>> > > > >> > will keep
> > > > > > > > > >>>> it
> > > > > > > > > >>>> > > > >> > going
> > > > > > > > > >>>> > > > >> > from May 29 19:15Z.
> > > > > > > > > >>>> > > > >> >
> > > > > > > > > >>>> > > > >> > One thing I do not understand is
that the
> > > > > > > > > >>>> > > > >> > output
> > > > > > > > > >>>> > > > >> > file
> > > > > > > > > >>>> > > > >> > is
> > > > > > > > > >>>> already
> > > > > > > > > >>>> > > > >> > generated
> > > > > > > > > >>>> > > > >> > while the job is still
> > > > > > > > > >>>> > > > >> > running.
> > > > > > > > > >>>> > > > >> > /gpfs/dell2/emc/modeling/noscrub/Ho-
> > > > > > > > > >>>> > > > >> >
> Chun.Huang/VIIRS_AOD/REGRID/aqm.20200519/VIIRS-
> > > > > > > > > >>>> > > > >> > L3-
> > > > > > > > > >>>> > > > >> > AOD_AQM_aqm_20200519_20.nc
> > > > > > > > > >>>> > > > >> >
> > > > > > > > > >>>> > > > >> > Ho-Chun Huang
> > > > > > > > > >>>> > > > >> >
> > > > > > > > > >>>> > > > >> > IMSG at NOAA/NWS/NCEP/EMC
> > > > > > > > > >>>> > > > >> >
> > > > > > > > > >>>> > > > >> > 5830 University Research Ct., Rm.
2792
> > > > > > > > > >>>> > > > >> >
> > > > > > > > > >>>> > > > >> > College Park, MD 20740
> > > > > > > > > >>>> > > > >> >
> > > > > > > > > >>>> > > > >> > Ho-Chun.Huang at noaa.gov
> > > > > > > > > >>>> > > > >> > <Joe.Smith at noaa.gov>
> > > > > > > > > >>>> > > > >> >
> > > > > > > > > >>>> > > > >> > 301-683-3958
> > > > > > > > > >>>> > > > >>
> > > > > > > > > >>>> > > > >>
> > > > > > > > > >>>> > > > >>
> > > > > > > > > >>>> > > > >>
> > > > > > > > > >>>> > > >
> > > > > > > > > >>>> > > >
> > > > > > > > > >>>> >
> > > > > > > > > >>>> >
> > > > > > > > > >>>> >
> > > > > > > > > >>>> >
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list