[ncl-talk] GOES16 netCDF files
Tim Risner
trisner at awis.com
Thu Aug 9 04:59:28 MDT 2018
Philippe,
The solution on my end was two-fold.
I also went through the code and removed variables after use, which is
something I like to do anyways.
Second, the machine I was using had 4GB of memory. I boosted it to 8GB.
Processing after these two changes has had no problems.
*Tim Risner*
*Operational Meteorologist*
*(888) 798-9955 ext 1006*
<http://www.awis.com>[image: https://www.facebook.com/AWISWeatherServices/]
<https://www.facebook.com/AWISWeatherServices/>
<https://twitter.com/awisweather>
On Wed, Aug 8, 2018 at 2:38 PM, Philippe Papin <ppapin at albany.edu> wrote:
> Hi Tim & NCL-talk,
>
> Just wondering if this issue with GOES-16 data was ever resolved. I was
> having similar memory issues today when trying to load up and plot GOES-16
> channel 2 visible imagery (full disk).
>
> The issue does not appear to be with loading the data originally within an
> ncl script, but rather when you call the goesYXtoLatLon.ncl function (as
> seen in in example 6 here: https://www.ncl.ucar.
> edu/Applications/goes.shtml), in which the memory blows up to
> unsustainable proportions in order to create the 2d lat/lon coordinates for
> the GOES-16 array.
>
> I was finally able to get things working, but had to end up going into
> goesYXtoLatLon.ncl and deleting a lot of variables right away after they
> were used. Having too many [21696] x [21696] arrays in the function was
> causing a major memory crunch, so this is likely where the script failed if
> you ran out of allocated memory on the system you were running the ncl
> script on.
>
> Ultimately, the best solution might be to only run goesYXtoLatLon.ncl
> once and save its output into a seperate netCDF file, then rerun the master
> script with this new netCDF file of lat/lon information. That will probably
> save time & memory.
>
> Cheers,
> -Philippe
>
>
>
>
> On Sat, Jan 20, 2018 at 3:23 AM Tim Risner via ncl-talk <ncl-talk at ucar.edu>
> wrote:
>
>> No worries Rick, I appreciate the extra sets of eyes on the issue.
>>
>> GOES16 data is stored in publicly accessible s3 buckets in the Amazon
>> cloud, with time relative file names.
>> It's a bit difficult to scour, then pull the data, but given our move to
>> the Amazon cloud, I've been able to build a pretty simple data pull.
>>
>> Here are 2 links to 2 separate files that are GOES16 images from this
>> morning.
>> 1st is the CONUS file that is used in the original working ncl code:
>> https://s3.amazonaws.com/noaa-goes16/ABI-L2-MCMIPC/2018/020/
>> 11/OR_ABI-L2-MCMIPC-M3_G16_s20180201102199_e20180201104572_
>> c20180201105079.nc
>> Hit the link and the netcdf file will download.
>> The above link, paired with the attached NCL code, originally created by
>> you guys and tweaked by me, gets you the attached image. (needs more work
>> and cleaning up, but a workable image for sure.)
>>
>> 2nd, here is the "Full Disk" file link of nearly the same time.
>> The file is supposed to contain the same variables and info, just cover
>> more area.
>> https://s3.amazonaws.com/noaa-goes16/ABI-L2-MCMIPF/2018/020/
>> 11/OR_ABI-L2-MCMIPF-M3_G16_s20180201100391_e20180201111164_
>> c20180201111254.nc
>> This file is popping the
>> fatal:NclMalloc Failed:[errno=12]
>> Segmentation fault
>> error.
>>
>>
>> I'm thinking the fix would be to only use those lat/lon values from the
>> "FULL DISK" file that we need. These values are known and set in the .nc
>> file.
>> In the attached example as:
>> res at mpMinLatF = 1.0
>> res at mpMaxLatF = 37.0
>> res at mpMinLonF = -87.5
>> res at mpMaxLonF = -22.5
>> After a few days of trying, I'm at a loss as to how to use these
>> boundaries in the goesYX.ncl file when pulling the data and converting to
>> lat/lon.
>>
>>
>>
>>
>>
>>
>> *Tim Risner*
>> *Operational Meteorologist*
>> *(888) 798-9955 ext 1006*
>>
>> <http://www.awis.com>[image:
>> https://www.facebook.com/AWISWeatherServices/]
>> <https://www.facebook.com/AWISWeatherServices/>
>> <https://twitter.com/awisweather>
>>
>> On Fri, Jan 19, 2018 at 4:52 PM, Rick Brownrigg <brownrig at ucar.edu>
>> wrote:
>>
>>> Hi Tim,
>>>
>>> Sorry to bug you, I've had a heck of time finding datasets for this
>>> satellite (presumably because its still in calibration/testing mode) -- I
>>> can find all manner of information and preprocessed imagery, but actual
>>> netcdf files -- very frustrating. Can I ask where you are getting your data
>>> from?
>>>
>>> Rick
>>>
>>> On Fri, Jan 19, 2018 at 3:16 PM, Tim Risner <trisner at awis.com> wrote:
>>>
>>>> I’m attempting to use the same code that is previously mentioned to map
>>>> regions that are outside the CONUS, which requires the “Full Disk” set of
>>>> files, versus the “CONUS” files that are working.
>>>> Variables and files are the same, of course the area is bigger,
>>>> therefore the variables are much larger.
>>>>
>>>> I deleted all variables in the code once they were no longer needed and
>>>> have been mulling the code over for a few days now, but can’t seem to find
>>>> a solution to clean the memory sufficiently.
>>>>
>>>> One solution would be to only use the portion of the file I need (i.e.
>>>> only the western Atlantic ocean), but I can’t seem to only strip that
>>>> section of the file/variable.
>>>>
>>>>
>>>>
>>>> On Jan 19, 2018, at 16:04, Rick Brownrigg <brownrig at ucar.edu> wrote:
>>>>
>>>> Hi Tim,
>>>>
>>>> Those are big variables, but not that big. Are you trying to read in
>>>> large numbers of them?
>>>>
>>>> Rick
>>>>
>>>>
>>>> On Fri, Jan 19, 2018 at 1:54 PM, Tim Risner <trisner at awis.com> wrote:
>>>>
>>>>> Works great Mary, and you can easily pull the sat wavelength you need
>>>>> from it.
>>>>> One issue I did run into was if you use the “Full Disk” files rather
>>>>> then the “CONUS” files used in the example, you get
>>>>> what I expect to be a memory issue?
>>>>>
>>>>> Error to screen
>>>>> *******************
>>>>> “fatal: NclMalloc Failed:[errno=12] Segmentation fault” error.
>>>>>
>>>>>
>>>>> info
>>>>> ******
>>>>> Variable: phi
>>>>> Type: double
>>>>> Total Size: 235358208 bytes
>>>>> 29419776 values
>>>>> Number of Dimensions: 2
>>>>> Dimensions and sizes: [5424] x [5424]
>>>>> Coordinates:
>>>>> Number Of Attributes: 1
>>>>> _FillValue : -999
>>>>>
>>>>> Variable: lambda
>>>>> Type: double
>>>>> Total Size: 235358208 bytes
>>>>> 29419776 values
>>>>> Number of Dimensions: 2
>>>>> Dimensions and sizes: [5424] x [5424]
>>>>> Coordinates:
>>>>> Number Of Attributes: 1
>>>>> _FillValue : -999
>>>>>
>>>>> Variable: data
>>>>> Type: float
>>>>> Total Size: 117679104 bytes
>>>>> 29419776 values
>>>>> Number of Dimensions: 2
>>>>> Dimensions and sizes: [y | 5424] x [x | 5424]
>>>>> Coordinates:
>>>>> y: [0..5423]
>>>>> x: [0..5423]
>>>>> Number Of Attributes: 16
>>>>> lon2d : <ARRAY of 29419776 elements>
>>>>> lat2d : <ARRAY of 29419776 elements>
>>>>> long_name : ABI Cloud and Moisture Imagery reflectance factor
>>>>> standard_name : toa_lambertian_equivalent_
>>>>> albedo_multiplied_by_cosine_solar_zenith_angle
>>>>> _Unsigned : true
>>>>> sensor_band_bit_depth : 12
>>>>> downsampling_method : average
>>>>> units : 1
>>>>> resolution : y: 0.000056 rad x: 0.000056 rad
>>>>> coordinates : band_id_C02 band_wavelength_C02 t y x
>>>>> grid_mapping : goes_imager_projection
>>>>> cell_methods : t: point area: sum (interval: 0.000014 rad)
>>>>> ancillary_variables : DQF_C02
>>>>> _FillValue_original : -1
>>>>> _FillValue : -1
>>>>> valid_range : ( 0, 1.299999 )
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Jan 11, 2018, at 6:07 PM, Mary Haley <haley at ucar.edu> wrote:
>>>>>
>>>>> Hi Tim,
>>>>>
>>>>> My apologies for getting back to you so late. We got side-tracked with
>>>>> another harder GOES16 issue (regridding a large GOES16 variable) and wanted
>>>>> to finish that up to make sure we were doing things correctly.
>>>>>
>>>>> Rick and Bill (members of the NCL development team) spent some time
>>>>> looking into documentation for GOES16 and figured out how to calculate the
>>>>> lat / lon values, given the Y / X coordinates and map projection
>>>>> information on the file. No sooner than this was done that the satellite
>>>>> was repositioned, and we had to tweak the calculations a little.
>>>>>
>>>>> I've attached the NCL scripts that Rick created, and the plot it
>>>>> generates. This script should work on the new GOES16 data as well.
>>>>>
>>>>> You need to run:
>>>>>
>>>>> ncl plot_goes16.ncl
>>>>>
>>>>> The CMI_16 variable was arbitrarily chosen, but we did chose 16 over
>>>>> the other numbers because it was more interesting.
>>>>>
>>>>> I understand that there are different channels, and a better way to
>>>>> plot this data might be to overlay the different channels with different
>>>>> opacities set. I need to look into how another user did this, and I'll
>>>>> post back here if I ever get around to it.
>>>>>
>>>>> --Mary
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Dec 13, 2017 at 1:02 PM, Tim Risner <trisner at awis.com> wrote:
>>>>>
>>>>>> thanks Rick, others.
>>>>>> version is 6.4.0
>>>>>> running on Linux Amazon EC2 instance. x86_64 x86_64 x86_64 GNU/Linux
>>>>>> testing GOES16 file attached.
>>>>>> I obtained such file from the Amazon AWS bucket system that is now
>>>>>> publicly accessible by the gov't/Amazon.
>>>>>> Information on file lists it as a GOES16 netcdf file.
>>>>>>
>>>>>> I assumed I could simply replace the file in the example for GOES 11
>>>>>> example here: http://www.ncl.ucar.edu/Applications/Scripts/goes_1.ncl
>>>>>> and all would work if the files and datasets were exchangeable.
>>>>>> Assumptions are likely my problem.
>>>>>>
>>>>>> Error output:
>>>>>>
>>>>>> fatal:["Execute.c":132]:variable (data) is not in file
>>>>>> (/var/NCL/INPUT/GOES16.nc)
>>>>>> fatal:["Execute.c":6381]:variable (data) is not in file (f)
>>>>>> fatal:["Execute.c":8640]:Execute: Error occurred at or near line 4
>>>>>> in file /var/NCL/CODE/testgoes.nc
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Tim Risner*
>>>>>> *Operational Meteorologist*
>>>>>> *(888) 798-9955 ext 1006 <(888)%20798-9955>*
>>>>>>
>>>>>> <http://www.awis.com/>[image:
>>>>>> https://www.facebook.com/AWISWeatherServices/]
>>>>>> <https://www.facebook.com/AWISWeatherServices/>
>>>>>> <https://twitter.com/awisweather>
>>>>>>
>>>>>> On Wed, Dec 13, 2017 at 9:02 AM, Rick Brownrigg <brownrig at ucar.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Tim,
>>>>>>>
>>>>>>> Do you have a dataset you could share? I download some data from
>>>>>>> this site:
>>>>>>>
>>>>>>> https://www.goes-r.gov
>>>>>>>
>>>>>>> but not knowing what I'm really looking for, I could not ascertain
>>>>>>> that it was specifically GOES16. NCL was able to read it.
>>>>>>>
>>>>>>> I should also ask what version of NCL are you working with?
>>>>>>>
>>>>>>> If your dataset is too large to email, you can post it to our FTP
>>>>>>> site:
>>>>>>>
>>>>>>> http://ncl.ucar.edu/ftp_files.shtml
>>>>>>>
>>>>>>> Rick
>>>>>>>
>>>>>>> On Wed, Dec 13, 2017 at 7:35 AM, Tim Risner <trisner at awis.com> wrot
>>>>>>> e:
>>>>>>>
>>>>>>>> Hello All,
>>>>>>>> I am curious as to what the GOES16 netCDF plan is for NCL.
>>>>>>>> I've attempted to ingest the new GOES16 netCDF into generic GOES
>>>>>>>> code found on the examples pages, but am getting a variables not found
>>>>>>>> error when attempting to produce an image. Variable not found is "data". I
>>>>>>>> assume this is due to the differences between the GOES16 netCDF files and
>>>>>>>> the older GOES products? or is it just operator error and GOES16 netcdf
>>>>>>>> files should work?
>>>>>>>>
>>>>>>>> Thanks!!!
>>>>>>>>
>>>>>>>>
>>>>>>>> *Tim Risner*
>>>>>>>> *Operational Meteorologist*
>>>>>>>> *(888) 798-9955 ext 1006 <(888)%20798-9955>*
>>>>>>>>
>>>>>>>> <http://www.awis.com/>[image:
>>>>>>>> https://www.facebook.com/AWISWeatherServices/]
>>>>>>>> <https://www.facebook.com/AWISWeatherServices/>
>>>>>>>> <https://twitter.com/awisweather>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> ncl-talk mailing list
>>>>>>>> ncl-talk at ucar.edu
>>>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> ncl-talk mailing list
>>>>>> ncl-talk at ucar.edu
>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>
>>>>>>
>>>>> <goes16.png><plot_goes16.ncl><goesYXtoLatLon.ncl>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
>
> --
> -------------------------------------
> *Philippe P. Papin, Ph.D.*
> NRC Research Associate Postdoc
> Global Modeling Division
> Naval Research Laboratory - Monterey, CA
> http://www.atmos.albany.edu/student/ppapin/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180809/4dff1dc5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1.png
Type: image/png
Size: 957087 bytes
Desc: not available
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180809/4dff1dc5/attachment-0001.png>
More information about the ncl-talk
mailing list