[ncl-talk] GOES16 netCDF files
Philippe Papin
ppapin at albany.edu
Wed Aug 8 13:38:28 MDT 2018
Hi Tim & NCL-talk,
Just wondering if this issue with GOES-16 data was ever resolved. I was
having similar memory issues today when trying to load up and plot GOES-16
channel 2 visible imagery (full disk).
The issue does not appear to be with loading the data originally within an
ncl script, but rather when you call the goesYXtoLatLon.ncl function (as
seen in in example 6 here: https://www.ncl.ucar.edu/Applications/goes.shtml),
in which the memory blows up to unsustainable proportions in order to
create the 2d lat/lon coordinates for the GOES-16 array.
I was finally able to get things working, but had to end up going into
goesYXtoLatLon.ncl and deleting a lot of variables right away after they
were used. Having too many [21696] x [21696] arrays in the function was
causing a major memory crunch, so this is likely where the script failed if
you ran out of allocated memory on the system you were running the ncl
script on.
Ultimately, the best solution might be to only run goesYXtoLatLon.ncl once
and save its output into a seperate netCDF file, then rerun the master
script with this new netCDF file of lat/lon information. That will probably
save time & memory.
Cheers,
-Philippe
On Sat, Jan 20, 2018 at 3:23 AM Tim Risner via ncl-talk <ncl-talk at ucar.edu>
wrote:
> No worries Rick, I appreciate the extra sets of eyes on the issue.
>
> GOES16 data is stored in publicly accessible s3 buckets in the Amazon
> cloud, with time relative file names.
> It's a bit difficult to scour, then pull the data, but given our move to
> the Amazon cloud, I've been able to build a pretty simple data pull.
>
> Here are 2 links to 2 separate files that are GOES16 images from this
> morning.
> 1st is the CONUS file that is used in the original working ncl code:
>
> https://s3.amazonaws.com/noaa-goes16/ABI-L2-MCMIPC/2018/020/11/OR_ABI-L2-MCMIPC-M3_G16_s20180201102199_e20180201104572_c20180201105079.nc
> Hit the link and the netcdf file will download.
> The above link, paired with the attached NCL code, originally created by
> you guys and tweaked by me, gets you the attached image. (needs more work
> and cleaning up, but a workable image for sure.)
>
> 2nd, here is the "Full Disk" file link of nearly the same time.
> The file is supposed to contain the same variables and info, just cover
> more area.
>
> https://s3.amazonaws.com/noaa-goes16/ABI-L2-MCMIPF/2018/020/11/OR_ABI-L2-MCMIPF-M3_G16_s20180201100391_e20180201111164_c20180201111254.nc
> This file is popping the
> fatal:NclMalloc Failed:[errno=12]
> Segmentation fault
> error.
>
>
> I'm thinking the fix would be to only use those lat/lon values from the
> "FULL DISK" file that we need. These values are known and set in the .nc
> file.
> In the attached example as:
> res at mpMinLatF = 1.0
> res at mpMaxLatF = 37.0
> res at mpMinLonF = -87.5
> res at mpMaxLonF = -22.5
> After a few days of trying, I'm at a loss as to how to use these
> boundaries in the goesYX.ncl file when pulling the data and converting to
> lat/lon.
>
>
>
>
>
>
> *Tim Risner*
> *Operational Meteorologist*
> *(888) 798-9955 ext 1006*
>
> <http://www.awis.com>[image:
> https://www.facebook.com/AWISWeatherServices/]
> <https://www.facebook.com/AWISWeatherServices/>
> <https://twitter.com/awisweather>
>
> On Fri, Jan 19, 2018 at 4:52 PM, Rick Brownrigg <brownrig at ucar.edu> wrote:
>
>> Hi Tim,
>>
>> Sorry to bug you, I've had a heck of time finding datasets for this
>> satellite (presumably because its still in calibration/testing mode) -- I
>> can find all manner of information and preprocessed imagery, but actual
>> netcdf files -- very frustrating. Can I ask where you are getting your data
>> from?
>>
>> Rick
>>
>> On Fri, Jan 19, 2018 at 3:16 PM, Tim Risner <trisner at awis.com> wrote:
>>
>>> I’m attempting to use the same code that is previously mentioned to map
>>> regions that are outside the CONUS, which requires the “Full Disk” set of
>>> files, versus the “CONUS” files that are working.
>>> Variables and files are the same, of course the area is bigger,
>>> therefore the variables are much larger.
>>>
>>> I deleted all variables in the code once they were no longer needed and
>>> have been mulling the code over for a few days now, but can’t seem to find
>>> a solution to clean the memory sufficiently.
>>>
>>> One solution would be to only use the portion of the file I need (i.e.
>>> only the western Atlantic ocean), but I can’t seem to only strip that
>>> section of the file/variable.
>>>
>>>
>>>
>>> On Jan 19, 2018, at 16:04, Rick Brownrigg <brownrig at ucar.edu> wrote:
>>>
>>> Hi Tim,
>>>
>>> Those are big variables, but not that big. Are you trying to read in
>>> large numbers of them?
>>>
>>> Rick
>>>
>>>
>>> On Fri, Jan 19, 2018 at 1:54 PM, Tim Risner <trisner at awis.com> wrote:
>>>
>>>> Works great Mary, and you can easily pull the sat wavelength you need
>>>> from it.
>>>> One issue I did run into was if you use the “Full Disk” files rather
>>>> then the “CONUS” files used in the example, you get
>>>> what I expect to be a memory issue?
>>>>
>>>> Error to screen
>>>> *******************
>>>> “fatal: NclMalloc Failed:[errno=12] Segmentation fault” error.
>>>>
>>>>
>>>> info
>>>> ******
>>>> Variable: phi
>>>> Type: double
>>>> Total Size: 235358208 bytes
>>>> 29419776 values
>>>> Number of Dimensions: 2
>>>> Dimensions and sizes: [5424] x [5424]
>>>> Coordinates:
>>>> Number Of Attributes: 1
>>>> _FillValue : -999
>>>>
>>>> Variable: lambda
>>>> Type: double
>>>> Total Size: 235358208 bytes
>>>> 29419776 values
>>>> Number of Dimensions: 2
>>>> Dimensions and sizes: [5424] x [5424]
>>>> Coordinates:
>>>> Number Of Attributes: 1
>>>> _FillValue : -999
>>>>
>>>> Variable: data
>>>> Type: float
>>>> Total Size: 117679104 bytes
>>>> 29419776 values
>>>> Number of Dimensions: 2
>>>> Dimensions and sizes: [y | 5424] x [x | 5424]
>>>> Coordinates:
>>>> y: [0..5423]
>>>> x: [0..5423]
>>>> Number Of Attributes: 16
>>>> lon2d : <ARRAY of 29419776 elements>
>>>> lat2d : <ARRAY of 29419776 elements>
>>>> long_name : ABI Cloud and Moisture Imagery reflectance factor
>>>> standard_name :
>>>> toa_lambertian_equivalent_albedo_multiplied_by_cosine_solar_zenith_angle
>>>> _Unsigned : true
>>>> sensor_band_bit_depth : 12
>>>> downsampling_method : average
>>>> units : 1
>>>> resolution : y: 0.000056 rad x: 0.000056 rad
>>>> coordinates : band_id_C02 band_wavelength_C02 t y x
>>>> grid_mapping : goes_imager_projection
>>>> cell_methods : t: point area: sum (interval: 0.000014 rad)
>>>> ancillary_variables : DQF_C02
>>>> _FillValue_original : -1
>>>> _FillValue : -1
>>>> valid_range : ( 0, 1.299999 )
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Jan 11, 2018, at 6:07 PM, Mary Haley <haley at ucar.edu> wrote:
>>>>
>>>> Hi Tim,
>>>>
>>>> My apologies for getting back to you so late. We got side-tracked with
>>>> another harder GOES16 issue (regridding a large GOES16 variable) and wanted
>>>> to finish that up to make sure we were doing things correctly.
>>>>
>>>> Rick and Bill (members of the NCL development team) spent some time
>>>> looking into documentation for GOES16 and figured out how to calculate the
>>>> lat / lon values, given the Y / X coordinates and map projection
>>>> information on the file. No sooner than this was done that the satellite
>>>> was repositioned, and we had to tweak the calculations a little.
>>>>
>>>> I've attached the NCL scripts that Rick created, and the plot it
>>>> generates. This script should work on the new GOES16 data as well.
>>>>
>>>> You need to run:
>>>>
>>>> ncl plot_goes16.ncl
>>>>
>>>> The CMI_16 variable was arbitrarily chosen, but we did chose 16 over
>>>> the other numbers because it was more interesting.
>>>>
>>>> I understand that there are different channels, and a better way to
>>>> plot this data might be to overlay the different channels with different
>>>> opacities set. I need to look into how another user did this, and I'll
>>>> post back here if I ever get around to it.
>>>>
>>>> --Mary
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Dec 13, 2017 at 1:02 PM, Tim Risner <trisner at awis.com> wrote:
>>>>
>>>>> thanks Rick, others.
>>>>> version is 6.4.0
>>>>> running on Linux Amazon EC2 instance. x86_64 x86_64 x86_64 GNU/Linux
>>>>> testing GOES16 file attached.
>>>>> I obtained such file from the Amazon AWS bucket system that is now
>>>>> publicly accessible by the gov't/Amazon.
>>>>> Information on file lists it as a GOES16 netcdf file.
>>>>>
>>>>> I assumed I could simply replace the file in the example for GOES 11
>>>>> example here: http://www.ncl.ucar.edu/Applications/Scripts/goes_1.ncl
>>>>> and all would work if the files and datasets were exchangeable.
>>>>> Assumptions are likely my problem.
>>>>>
>>>>> Error output:
>>>>>
>>>>> fatal:["Execute.c":132]:variable (data) is not in file
>>>>> (/var/NCL/INPUT/GOES16.nc)
>>>>> fatal:["Execute.c":6381]:variable (data) is not in file (f)
>>>>> fatal:["Execute.c":8640]:Execute: Error occurred at or near line 4 in
>>>>> file /var/NCL/CODE/testgoes.nc
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Tim Risner*
>>>>> *Operational Meteorologist*
>>>>> *(888) 798-9955 ext 1006 <(888)%20798-9955>*
>>>>>
>>>>> <http://www.awis.com/>[image:
>>>>> https://www.facebook.com/AWISWeatherServices/]
>>>>> <https://www.facebook.com/AWISWeatherServices/>
>>>>> <https://twitter.com/awisweather>
>>>>>
>>>>> On Wed, Dec 13, 2017 at 9:02 AM, Rick Brownrigg <brownrig at ucar.edu>
>>>>> wrote:
>>>>>
>>>>>> Hi Tim,
>>>>>>
>>>>>> Do you have a dataset you could share? I download some data from
>>>>>> this site:
>>>>>>
>>>>>> https://www.goes-r.gov
>>>>>>
>>>>>> but not knowing what I'm really looking for, I could not ascertain
>>>>>> that it was specifically GOES16. NCL was able to read it.
>>>>>>
>>>>>> I should also ask what version of NCL are you working with?
>>>>>>
>>>>>> If your dataset is too large to email, you can post it to our FTP
>>>>>> site:
>>>>>>
>>>>>> http://ncl.ucar.edu/ftp_files.shtml
>>>>>>
>>>>>> Rick
>>>>>>
>>>>>> On Wed, Dec 13, 2017 at 7:35 AM, Tim Risner <trisner at awis.com> wrote:
>>>>>>
>>>>>>> Hello All,
>>>>>>> I am curious as to what the GOES16 netCDF plan is for NCL.
>>>>>>> I've attempted to ingest the new GOES16 netCDF into generic GOES
>>>>>>> code found on the examples pages, but am getting a variables not found
>>>>>>> error when attempting to produce an image. Variable not found is "data". I
>>>>>>> assume this is due to the differences between the GOES16 netCDF files and
>>>>>>> the older GOES products? or is it just operator error and GOES16 netcdf
>>>>>>> files should work?
>>>>>>>
>>>>>>> Thanks!!!
>>>>>>>
>>>>>>>
>>>>>>> *Tim Risner*
>>>>>>> *Operational Meteorologist*
>>>>>>> *(888) 798-9955 ext 1006 <(888)%20798-9955>*
>>>>>>>
>>>>>>> <http://www.awis.com/>[image:
>>>>>>> https://www.facebook.com/AWISWeatherServices/]
>>>>>>> <https://www.facebook.com/AWISWeatherServices/>
>>>>>>> <https://twitter.com/awisweather>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ncl-talk mailing list
>>>>>>> ncl-talk at ucar.edu
>>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> ncl-talk at ucar.edu
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>
>>>>>
>>>> <goes16.png><plot_goes16.ncl><goesYXtoLatLon.ncl>
>>>>
>>>>
>>>>
>>>
>>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
--
-------------------------------------
*Philippe P. Papin, Ph.D.*
NRC Research Associate Postdoc
Global Modeling Division
Naval Research Laboratory - Monterey, CA
http://www.atmos.albany.edu/student/ppapin/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180808/f1003be1/attachment.html>
More information about the ncl-talk
mailing list