[ncl-talk] GOES16 netCDF files
Tim Risner
trisner at awis.com
Sat Jan 20 04:23:07 MST 2018
No worries Rick, I appreciate the extra sets of eyes on the issue.
GOES16 data is stored in publicly accessible s3 buckets in the Amazon
cloud, with time relative file names.
It's a bit difficult to scour, then pull the data, but given our move to
the Amazon cloud, I've been able to build a pretty simple data pull.
Here are 2 links to 2 separate files that are GOES16 images from this
morning.
1st is the CONUS file that is used in the original working ncl code:
https://s3.amazonaws.com/noaa-goes16/ABI-L2-MCMIPC/2018/020/11/OR_ABI-L2-MCMIPC-M3_G16_s20180201102199_e20180201104572_c20180201105079.nc
Hit the link and the netcdf file will download.
The above link, paired with the attached NCL code, originally created by
you guys and tweaked by me, gets you the attached image. (needs more work
and cleaning up, but a workable image for sure.)
2nd, here is the "Full Disk" file link of nearly the same time.
The file is supposed to contain the same variables and info, just cover
more area.
https://s3.amazonaws.com/noaa-goes16/ABI-L2-MCMIPF/2018/020/11/OR_ABI-L2-MCMIPF-M3_G16_s20180201100391_e20180201111164_c20180201111254.nc
This file is popping the
fatal:NclMalloc Failed:[errno=12]
Segmentation fault
error.
I'm thinking the fix would be to only use those lat/lon values from the
"FULL DISK" file that we need. These values are known and set in the .nc
file.
In the attached example as:
res at mpMinLatF = 1.0
res at mpMaxLatF = 37.0
res at mpMinLonF = -87.5
res at mpMaxLonF = -22.5
After a few days of trying, I'm at a loss as to how to use these boundaries
in the goesYX.ncl file when pulling the data and converting to lat/lon.
*Tim Risner*
*Operational Meteorologist*
*(888) 798-9955 ext 1006*
<http://www.awis.com>[image: https://www.facebook.com/AWISWeatherServices/]
<https://www.facebook.com/AWISWeatherServices/>
<https://twitter.com/awisweather>
On Fri, Jan 19, 2018 at 4:52 PM, Rick Brownrigg <brownrig at ucar.edu> wrote:
> Hi Tim,
>
> Sorry to bug you, I've had a heck of time finding datasets for this
> satellite (presumably because its still in calibration/testing mode) -- I
> can find all manner of information and preprocessed imagery, but actual
> netcdf files -- very frustrating. Can I ask where you are getting your data
> from?
>
> Rick
>
> On Fri, Jan 19, 2018 at 3:16 PM, Tim Risner <trisner at awis.com> wrote:
>
>> I’m attempting to use the same code that is previously mentioned to map
>> regions that are outside the CONUS, which requires the “Full Disk” set of
>> files, versus the “CONUS” files that are working.
>> Variables and files are the same, of course the area is bigger, therefore
>> the variables are much larger.
>>
>> I deleted all variables in the code once they were no longer needed and
>> have been mulling the code over for a few days now, but can’t seem to find
>> a solution to clean the memory sufficiently.
>>
>> One solution would be to only use the portion of the file I need (i.e.
>> only the western Atlantic ocean), but I can’t seem to only strip that
>> section of the file/variable.
>>
>>
>>
>> On Jan 19, 2018, at 16:04, Rick Brownrigg <brownrig at ucar.edu> wrote:
>>
>> Hi Tim,
>>
>> Those are big variables, but not that big. Are you trying to read in
>> large numbers of them?
>>
>> Rick
>>
>>
>> On Fri, Jan 19, 2018 at 1:54 PM, Tim Risner <trisner at awis.com> wrote:
>>
>>> Works great Mary, and you can easily pull the sat wavelength you need
>>> from it.
>>> One issue I did run into was if you use the “Full Disk” files rather
>>> then the “CONUS” files used in the example, you get
>>> what I expect to be a memory issue?
>>>
>>> Error to screen
>>> *******************
>>> “fatal: NclMalloc Failed:[errno=12] Segmentation fault” error.
>>>
>>>
>>> info
>>> ******
>>> Variable: phi
>>> Type: double
>>> Total Size: 235358208 bytes
>>> 29419776 values
>>> Number of Dimensions: 2
>>> Dimensions and sizes: [5424] x [5424]
>>> Coordinates:
>>> Number Of Attributes: 1
>>> _FillValue : -999
>>>
>>> Variable: lambda
>>> Type: double
>>> Total Size: 235358208 bytes
>>> 29419776 values
>>> Number of Dimensions: 2
>>> Dimensions and sizes: [5424] x [5424]
>>> Coordinates:
>>> Number Of Attributes: 1
>>> _FillValue : -999
>>>
>>> Variable: data
>>> Type: float
>>> Total Size: 117679104 bytes
>>> 29419776 values
>>> Number of Dimensions: 2
>>> Dimensions and sizes: [y | 5424] x [x | 5424]
>>> Coordinates:
>>> y: [0..5423]
>>> x: [0..5423]
>>> Number Of Attributes: 16
>>> lon2d : <ARRAY of 29419776 elements>
>>> lat2d : <ARRAY of 29419776 elements>
>>> long_name : ABI Cloud and Moisture Imagery reflectance factor
>>> standard_name : toa_lambertian_equivalent_albe
>>> do_multiplied_by_cosine_solar_zenith_angle
>>> _Unsigned : true
>>> sensor_band_bit_depth : 12
>>> downsampling_method : average
>>> units : 1
>>> resolution : y: 0.000056 rad x: 0.000056 rad
>>> coordinates : band_id_C02 band_wavelength_C02 t y x
>>> grid_mapping : goes_imager_projection
>>> cell_methods : t: point area: sum (interval: 0.000014 rad)
>>> ancillary_variables : DQF_C02
>>> _FillValue_original : -1
>>> _FillValue : -1
>>> valid_range : ( 0, 1.299999 )
>>>
>>>
>>>
>>>
>>>
>>> On Jan 11, 2018, at 6:07 PM, Mary Haley <haley at ucar.edu> wrote:
>>>
>>> Hi Tim,
>>>
>>> My apologies for getting back to you so late. We got side-tracked with
>>> another harder GOES16 issue (regridding a large GOES16 variable) and wanted
>>> to finish that up to make sure we were doing things correctly.
>>>
>>> Rick and Bill (members of the NCL development team) spent some time
>>> looking into documentation for GOES16 and figured out how to calculate the
>>> lat / lon values, given the Y / X coordinates and map projection
>>> information on the file. No sooner than this was done that the satellite
>>> was repositioned, and we had to tweak the calculations a little.
>>>
>>> I've attached the NCL scripts that Rick created, and the plot it
>>> generates. This script should work on the new GOES16 data as well.
>>>
>>> You need to run:
>>>
>>> ncl plot_goes16.ncl
>>>
>>> The CMI_16 variable was arbitrarily chosen, but we did chose 16 over the
>>> other numbers because it was more interesting.
>>>
>>> I understand that there are different channels, and a better way to plot
>>> this data might be to overlay the different channels with different
>>> opacities set. I need to look into how another user did this, and I'll
>>> post back here if I ever get around to it.
>>>
>>> --Mary
>>>
>>>
>>>
>>>
>>> On Wed, Dec 13, 2017 at 1:02 PM, Tim Risner <trisner at awis.com> wrote:
>>>
>>>> thanks Rick, others.
>>>> version is 6.4.0
>>>> running on Linux Amazon EC2 instance. x86_64 x86_64 x86_64 GNU/Linux
>>>> testing GOES16 file attached.
>>>> I obtained such file from the Amazon AWS bucket system that is now
>>>> publicly accessible by the gov't/Amazon.
>>>> Information on file lists it as a GOES16 netcdf file.
>>>>
>>>> I assumed I could simply replace the file in the example for GOES 11
>>>> example here: http://www.ncl.ucar.edu/Applications/Scripts/goes_1.ncl
>>>> and all would work if the files and datasets were exchangeable.
>>>> Assumptions are likely my problem.
>>>>
>>>> Error output:
>>>>
>>>> fatal:["Execute.c":132]:variable (data) is not in file
>>>> (/var/NCL/INPUT/GOES16.nc)
>>>> fatal:["Execute.c":6381]:variable (data) is not in file (f)
>>>> fatal:["Execute.c":8640]:Execute: Error occurred at or near line 4 in
>>>> file /var/NCL/CODE/testgoes.nc
>>>>
>>>>
>>>>
>>>>
>>>> *Tim Risner*
>>>> *Operational Meteorologist*
>>>> *(888) 798-9955 ext 1006 <(888)%20798-9955>*
>>>>
>>>> <http://www.awis.com/>[image:
>>>> https://www.facebook.com/AWISWeatherServices/]
>>>> <https://www.facebook.com/AWISWeatherServices/>
>>>> <https://twitter.com/awisweather>
>>>>
>>>> On Wed, Dec 13, 2017 at 9:02 AM, Rick Brownrigg <brownrig at ucar.edu>
>>>> wrote:
>>>>
>>>>> Hi Tim,
>>>>>
>>>>> Do you have a dataset you could share? I download some data from this
>>>>> site:
>>>>>
>>>>> https://www.goes-r.gov
>>>>>
>>>>> but not knowing what I'm really looking for, I could not ascertain
>>>>> that it was specifically GOES16. NCL was able to read it.
>>>>>
>>>>> I should also ask what version of NCL are you working with?
>>>>>
>>>>> If your dataset is too large to email, you can post it to our FTP site:
>>>>>
>>>>> http://ncl.ucar.edu/ftp_files.shtml
>>>>>
>>>>> Rick
>>>>>
>>>>> On Wed, Dec 13, 2017 at 7:35 AM, Tim Risner <trisner at awis.com> wrote:
>>>>>
>>>>>> Hello All,
>>>>>> I am curious as to what the GOES16 netCDF plan is for NCL.
>>>>>> I've attempted to ingest the new GOES16 netCDF into generic GOES code
>>>>>> found on the examples pages, but am getting a variables not found error
>>>>>> when attempting to produce an image. Variable not found is "data". I assume
>>>>>> this is due to the differences between the GOES16 netCDF files and the
>>>>>> older GOES products? or is it just operator error and GOES16 netcdf files
>>>>>> should work?
>>>>>>
>>>>>> Thanks!!!
>>>>>>
>>>>>>
>>>>>> *Tim Risner*
>>>>>> *Operational Meteorologist*
>>>>>> *(888) 798-9955 ext 1006 <(888)%20798-9955>*
>>>>>>
>>>>>> <http://www.awis.com/>[image:
>>>>>> https://www.facebook.com/AWISWeatherServices/]
>>>>>> <https://www.facebook.com/AWISWeatherServices/>
>>>>>> <https://twitter.com/awisweather>
>>>>>>
>>>>>> _______________________________________________
>>>>>> ncl-talk mailing list
>>>>>> ncl-talk at ucar.edu
>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> ncl-talk at ucar.edu
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>>
>>> <goes16.png><plot_goes16.ncl><goesYXtoLatLon.ncl>
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180120/ca5d9d75/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: intgoes16.png
Type: image/png
Size: 109671 bytes
Desc: not available
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180120/ca5d9d75/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GOES16.westernatlantic.VIS.nc
Type: application/x-netcdf
Size: 3139 bytes
Desc: not available
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180120/ca5d9d75/attachment.nc>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: goesYX.ncl
Type: application/octet-stream
Size: 3032 bytes
Desc: not available
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180120/ca5d9d75/attachment.obj>
More information about the ncl-talk
mailing list