[Met_help] [rt.rap.ucar.edu #97459] History for NetCDF Satellite Data
Howard Soh via RT
met_help at ucar.edu
Wed Dec 2 19:26:43 MST 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
I used to have problems reading in some satellite data before and I ended
to have to convert them to "txt' file and then start from there. Now the
data creator tried to provide new data in netcdf format so we can read them
using MET directly. I tried an example using "plot_data_plane", here is
what I got:
cd /gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/Download/Raikoke/New
$ plot_data_plane
VOLCAT_HIMAWARI-8_FLDK_s2020296_173000_v300250_VCB_w167_FLDK_b2020295_204000_g001.nc
test.ps 'name="ash_mass_loading";level="(*,*)";'
DEBUG 1: Opening data file:
VOLCAT_HIMAWARI-8_FLDK_s2020296_173000_v300250_VCB_w167_FLDK_b2020295_204000_g001.nc
terminate called after throwing an instance of
'netCDF::exceptions::NcInvalidCoords'
what(): NetCDF: Index exceeds dimension bound
file: ncVar.cpp line:1598
Aborted (core dumped)
Frankly I still don't know what information is needed to fill in to the
header file. Would you help to take a look and let us know what is missing?
So I can get back to the data producer. Thank you.
Binyu
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: NetCDF Satellite Data
From: John Halley Gotway
Time: Mon Nov 16 16:14:06 2020
Binyu,
I'm adding my response here and will then reassign this ticket to
Howard Soh to see if he has any more ideas.
I downloaded your file to our anonymous ftp site:
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/wang_data_20201116/VOLCAT_HIMAWARI-
8_FLDK_s2020296_173000_v300250_VCB_w167_FLDK_b2020295_204000_g001.nc
I'm able to replicate the same error you're getting:
terminate called after throwing an instance of
'netCDF::exceptions::NcInvalidCoords'
After some testing I see that this is caused by the time variable
having 2 dimensions: int time(y, x) ;
When reading 2-D gridded data, the code expects the time variable to
be 1 dimensional. A file may contain multiple timesteps and a
coordinate variable defines what those timesteps are. To sidestep this
issue, I just removed the "time(y, x)" variable entirely. But then MET
isn't able to read the projection info:
ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out
projection from information in netCDF file.
Looking closely at the latitude and longitude variables, I see that
this really is not gridded data at all. Instead, it's a dense set of
pixels, but it doesn't appear that the lat/lon's are evenly spaced or
follow any sort of projection (e.g. Lambert Conformal, Mercator,
PolarStereographic, and so on).
Since this is a dense collection of individual points, you'll still
need to run the Point2Grid tool to place them onto a real grid. And
that's the question to Howard. Is there anyway that Binyu could format
this NetCDF file that would enable it to be read easily by the
Point2Grid tool?
Thanks,
John
------------------------------------------------
Subject: NetCDF Satellite Data
From: Howard Soh
Time: Mon Nov 16 21:15:13 2020
The point2grid handles a CF complaint NetCDF which has
1) 2D variables with lat/lon dimensions and the time variable with 1
value OR
2) 3D variables with time/lat/lon dimensions and the time variable
with 1 dimension.
The point2grid should be enhanced to support the 2-dimensional time
variable (if the enhancement is approved).
The sample NetCDF has two time offsets: 1603387817 and 1603387833
("seconds since 1970-01-01 00:00:00"). One possible way without
changing MET applications is generating two files (one file with time
offset 1603387817 and the other file with time offset 1603387817).
netcdf1:
int time(1);
time = 1603387817
fill the missing value for the time 1603387833
netcdf2:
int time(1);
time = 1603387833
fill the missing value for the time 1603387817
Cheers,
Howard
------------------------------------------------
Subject: NetCDF Satellite Data
From: binyu.wang at noaa.gov
Time: Tue Nov 17 07:11:48 2020
Thank you. Howard. I will forward your email to the person who created
the
data, and I may get back to you later.
Binyu
On Mon, Nov 16, 2020 at 11:15 PM Howard Soh via RT <met_help at ucar.edu>
wrote:
> The point2grid handles a CF complaint NetCDF which has
> 1) 2D variables with lat/lon dimensions and the time variable with 1
value
> OR
> 2) 3D variables with time/lat/lon dimensions and the time variable
with 1
> dimension.
>
> The point2grid should be enhanced to support the 2-dimensional time
> variable (if the enhancement is approved).
>
> The sample NetCDF has two time offsets: 1603387817 and 1603387833
> ("seconds since 1970-01-01 00:00:00"). One possible way without
changing
> MET applications is generating two files (one file with time offset
> 1603387817 and the other file with time offset 1603387817).
>
> netcdf1:
> int time(1);
> time = 1603387817
> fill the missing value for the time 1603387833
>
> netcdf2:
> int time(1);
> time = 1603387833
> fill the missing value for the time 1603387817
>
> Cheers,
> Howard
>
>
------------------------------------------------
Subject: NetCDF Satellite Data
From: John Halley Gotway
Time: Tue Nov 17 10:44:37 2020
Howard,
Do you know if having a 2D time variable is consistent with the
Climate
Forecast convention?
I'd prefer to make MET compatible with existing standards (like the CF
convention) and ask data providers to write to that standard. And I'd
like
to avoid writing custom code to handle custom data formats.
As long as storing time in this way is not inconsistent with the
CF-convention, please write up a GitHub issue to describe this work
and
assign it to yourself for met-10.0.
If there's a more CF-compliant way of structuring this data, please
let
Binyu know.
Thanks for looking into this!
John
On Mon, Nov 16, 2020 at 9:15 PM Howard Soh via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97459 >
>
> The point2grid handles a CF complaint NetCDF which has
> 1) 2D variables with lat/lon dimensions and the time variable with 1
value
> OR
> 2) 3D variables with time/lat/lon dimensions and the time variable
with 1
> dimension.
>
> The point2grid should be enhanced to support the 2-dimensional time
> variable (if the enhancement is approved).
>
> The sample NetCDF has two time offsets: 1603387817 and 1603387833
> ("seconds since 1970-01-01 00:00:00"). One possible way without
changing
> MET applications is generating two files (one file with time offset
> 1603387817 and the other file with time offset 1603387817).
>
> netcdf1:
> int time(1);
> time = 1603387817
> fill the missing value for the time 1603387833
>
> netcdf2:
> int time(1);
> time = 1603387833
> fill the missing value for the time 1603387817
>
> Cheers,
> Howard
>
>
------------------------------------------------
Subject: NetCDF Satellite Data
From: Howard Soh
Time: Tue Nov 17 11:50:34 2020
The CF does not specify the dimension of the coordinate variables
(lat/long/time). Many NetCDF files have the 2D lat/long variables to
support other projections. The 2D time variable is not common but it
might be useful to save the storage for sparse point observation data.
Cheers,
Howard
------------------------------------------------
Subject: NetCDF Satellite Data
From: John Halley Gotway
Time: Tue Nov 17 12:14:10 2020
Howard,
OK, if you think it makes sense, please go ahead and write up an issue
to
describe the change. And then work on it whenever it fits well into
your
plans. Hopefully soon enough to be included in the met-10.0 release.
Thanks,
John
On Tue, Nov 17, 2020 at 11:50 AM Howard Soh via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97459 >
>
> The CF does not specify the dimension of the coordinate variables
> (lat/long/time). Many NetCDF files have the 2D lat/long variables to
> support other projections. The 2D time variable is not common but it
might
> be useful to save the storage for sparse point observation data.
>
> Cheers,
> Howard
>
------------------------------------------------
Subject: NetCDF Satellite Data
From: Howard Soh
Time: Fri Nov 20 09:55:30 2020
The github issue was created.
https://github.com/dtcenter/MET/issues/1580
If no more issues, I would like to close this ticket.
Cheers,
Howard
------------------------------------------------
Subject: NetCDF Satellite Data
From: binyu.wang at noaa.gov
Time: Fri Nov 20 13:25:08 2020
Hello Howard,
Thank you for offering to help. I want to make sure I fully understand
the
solution: so you will find a solution to read the 2-D time variables.;
with
the new solution, my data can be read by MET/METplus directly without
any
other extra steps. Is that correct? You put "1-2 days" in the githup,
does
that mean you can have the problem solved very soon? I am asking this
because I am not sure if I should talk with the data provider and let
them
regenerate new data.
Thank you.
On Fri, Nov 20, 2020 at 11:55 AM Howard Soh via RT <met_help at ucar.edu>
wrote:
> The github issue was created.
> https://github.com/dtcenter/MET/issues/1580
>
> If no more issues, I would like to close this ticket.
>
> Cheers,
> Howard
>
------------------------------------------------
Subject: NetCDF Satellite Data
From: Howard Soh
Time: Tue Nov 24 11:39:09 2020
Yes, the github issue 1580 is that point2grid supports the sample data
you provided.
The MET's CF NetCDF library works for the gridded and the point data
both. The risk of passing this data to the other MET tools is not
considered on the time estimation.
Cheers,
Howard
------------------------------------------------
Subject: NetCDF Satellite Data
From: binyu.wang at noaa.gov
Time: Tue Nov 24 12:18:19 2020
That is great if MET/METplus could read the sample data directly so no
extra work will be needed to pre-process that. We want the data
provider to
use the same format in the future so we could use the same script/code
for
our routine work. Please let me know if there is any issue so we can
communicate with the provider for any change.
Thank you.
Binyu
On Tue, Nov 24, 2020 at 1:39 PM Howard Soh via RT <met_help at ucar.edu>
wrote:
> Yes, the github issue 1580 is that point2grid supports the sample
data you
> provided.
>
> The MET's CF NetCDF library works for the gridded and the point data
both.
> The risk of passing this data to the other MET tools is not
considered on
> the time estimation.
>
> Cheers,
> Howard
>
------------------------------------------------
More information about the Met_help
mailing list