[ncl-talk] iteration over files
Mary Haley
haley at ucar.edu
Thu Feb 26 10:48:02 MST 2015
Paul,
Thanks so much for providing a file. I was able to produce the same results
as you, and I have reported it to the developers who will look into it.
I'm hoping we can get a fix in by the next release of NCL, which should be
in March.
--Mary
On Thu, Feb 26, 2015 at 3:42 AM, Steeven Paul Yerraguntla <
steevenpaul at gmail.com> wrote:
> Dear NCL users,
> Please see the attachments for the the file.
>
> regards
>
>
> On Wed, Feb 25, 2015 at 8:28 PM, Mary Haley <haley at ucar.edu> wrote:
>
>> Paul,
>>
>> The output from print(gp) says that "gp" has 0 bytes. This means that NCL
>> is failing when trying to read this variable:
>>
>> Variable: gp
>> Type: none
>> Total Size: 0 bytes
>> 1 values
>> Number of Dimensions: 0
>> Dimensions and sizes:
>> Coordinates:
>> Number Of Attributes: 59
>>
>> I believe we have fixed this bug in the upcoming release of NCL. Would
>> you be able to provide us with the file so we can test it?
>>
>> --Mary
>>
>>
>>
>> On Tue, Feb 24, 2015 at 2:21 AM, Steeven Paul Yerraguntla <
>> steevenpaul at gmail.com> wrote:
>>
>>> Dear NCL users
>>> Thanks to Griffin for good suggestions and i could convert the hdf to
>>> nc with the script provided by Dennis long ago and the problems is the
>>> iteration is done only to the first file in the loop and the error
>>> occuring with the message "segmentation fault (core dumped)". I run
>>> the script again using the command ncl -x filename.ncl and the output
>>> as follows. Please let me know how can i get the iteration completed.
>>>
>>>
>>> ***********************************************************************************
>>> Copyright (C) 1995-2012 - All Rights Reserved
>>> University Corporation for Atmospheric Research
>>> NCAR Command Language Version 6.1.0
>>> The use of this software is governed by a License Agreement.
>>> See http://www.ncl.ucar.edu/ for more details.
>>> + ;************************************************
>>> + ; Allways examine file contents:
>>> + ; %> ncl_filedump foo.h5 | less
>>> + ;************************************************
>>> Loading file "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>>> +
>>> Loading file "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>>> +
>>> Loading file "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>>> +
>>> + ;************************************************
>>> + begin
>>> + diri = "./"
>>> + diroxx = "./"
>>> +
>>> + fili = systemfunc("cd "+diri+" ; ls *.h5")
>>> + nfil = dimsizes(fili)
>>> + print("nfil="+nfil)
>>> +
>>> + do nf=0,nfil-1
>>> + print("===========================")
>>> + f = addfile(diri+fili(nf), "r")
>>> + Latitude = f->/OLR/OLR_Dataset.Latitude
>>> + Longitude = f->/OLR/OLR_Dataset.Longitude
>>> +
>>> + print("---")
>>> + printVarSummary(Latitude) ; [DIM_000 |
>>> 250601]
>>> + print("Latitude: min="+min(Latitude)+" max="+max(Latitude))
>>> + print("---")
>>> + printVarSummary(Longitude) ; [DIM_000 |
>>> 250601]
>>> + print("Longitude: min="+min(Longitude)+" max="+max(Longitude))
>>> + print("---")
>>> + ;print(Latitude+" "+Longitude)
>>> + print("---")
>>> +
>>> + gp = f->GP_PARAM_INFO ; read group
>>> + print(gp)
>>> + print("---")
>>> +
>>> + latS = gp at ValidBottomLat
>>> + latN = gp at ValidTopLat
>>> + lonL = gp at ValidLeftLon
>>> + lonR = gp at ValidRightLon
>>> +
>>> + dlat = gp at LatInterval
>>> + nlat = toint((latN-latS)/dlat)+1
>>> + lat = fspan(latS, latN, nlat)
>>> + lat!0 = "lat"
>>> + lat at units = "degrees_north"
>>> +
>>> + dlon = gp at LonInterval
>>> + mlon = toint((lonR-lonL)/dlon)+1
>>> + lon = fspan(lonL, lonR, mlon)
>>> + lon!0 = "lon"
>>> + lon at units = "degrees_east"
>>> +
>>> + ;OLR = f->/OLR/OLR_Dataset.OLR
>>> + ;printVarSummary(OLR) ; [DIM_000 | 250601]
>>> +
>>> + olr = onedtond( f->/OLR/OLR_Dataset.OLR, (/nlat,mlon/) )
>>> + olr!0 = "lat"
>>> + olr!1 = "lon"
>>> + olr&lat = lat
>>> + olr&lon = lon
>>> + ;;olr at long_name = gp at GP_PARAM_NAME
>>> + ;;olr at units = gp at OLR_Unit
>>> + olr at long_name = "OLR"
>>> + olr at units = "W/m^2"
>>> +
>>> + print("typeof(gp at MissingValueInProduct
>>> ="+typeof(gp at MissingValueInProduct))
>>> ; type double
>>> +
>>> + olr at _FillValue= tofloat(gp at MissingValueInProduct) ; match olr type
>>> +
>>> + printVarSummary(olr)
>>> + print("olr: min="+min(olr)+" max="+max(olr))
>>> + print("---")
>>> +
>>> + ;************************************************
>>> + ; Examine distribution
>>> + ;************************************************
>>> +
>>> + opt = True
>>> + opt at PrintStat = True
>>> + stat = stat_dispersion(olr, opt )
>>> +
>>> + print("---")
>>> + fbase = str_get_field(fili(nf),1,".")
>>> + print("fbase="+fbase)
>>> +
>>> + diro = "./"
>>> + filo = fbase+".nc"
>>> + ptho = diro+filo
>>> +
>>> + system("/bin/rm -f "+ptho) ; remove any pre-existing file
>>> + ncdf = addfile(ptho ,"c") ; open output netCDF file
>>> +
>>> + ; create global attributes
>>> of the file
>>> + fAtt = True ; assign file attributes
>>> + fAtt at title = "NCL: H5 to netCDF"
>>> + fAtt at source = "MOSDAC: Meteorological & Oceanographic
>>> Satellite Data Archival Center (India)"
>>> + fAtt at source_file = fili(nf)
>>> + fAtt at Conventions = "None"
>>> + fAtt at creation_date = systemfunc ("date")
>>> + fileattdef( ncdf, fAtt ) ; copy file attributes
>>> +
>>> + ncdf->OLR = olr
>>> + end do
>>> + end
>>> (0) nfil=31
>>> (0) ===========================
>>> (0) ---
>>>
>>> Variable: Latitude
>>> Type: float
>>> Total Size: 1002404 bytes
>>> 250601 values
>>> Number of Dimensions: 1
>>> Dimensions and sizes: [DIM_000 | 250601]
>>> Coordinates:
>>> Number Of Attributes: 0
>>> (0) Latitude: min=-60 max=60
>>> (0) ---
>>>
>>> Variable: Longitude
>>> Type: float
>>> Total Size: 1002404 bytes
>>> 250601 values
>>> Number of Dimensions: 1
>>> Dimensions and sizes: [DIM_000 | 250601]
>>> Coordinates:
>>> Number Of Attributes: 0
>>> (0) Longitude: min=10 max=140
>>> (0) ---
>>> (0) ---
>>>
>>>
>>> Variable: gp
>>> Type: none
>>> Total Size: 0 bytes
>>> 1 values
>>> Number of Dimensions: 0
>>> Dimensions and sizes:
>>> Coordinates:
>>> Number Of Attributes: 59
>>> BinningInput_01 : K1VHR_05OCT2013_0000
>>> .
>>> .
>>> .
>>> BinningInput_44 : K1VHR_05OCT2013_2330
>>> BinningTime : 2330R_05OCT2013_2330
>>> GP_PARAM_DESCRIPTION : Daily_05OCT2013_2330
>>> GP_PARAM_NAME : Outgoing Longwave Radiation (OLR)
>>> Input_Channels : TIR
>>> LatInterval : 0.25
>>> Latitude_Unit : Degrees
>>> LonInterval : 0.25
>>> Longitude_Unit : Degrees
>>> MissingValueInProduct : 999
>>> NoOfBinningInputs : 44
>>> OLR_Unit : Watts/sq. met.
>>> ValidBottomLat : -60
>>> ValidLeftLon : 10
>>> ValidRightLon : 140
>>> ValidTopLat : 60
>>> (0) ---
>>> (0) typeof(gp at MissingValueInProduct=double
>>>
>>> Variable: olr
>>> Type: float
>>> Total Size: 1002404 bytes
>>> 250601 values
>>> Number of Dimensions: 2
>>> Dimensions and sizes: [lat | 481] x [lon | 521]
>>> Coordinates:
>>> lat: [ -60.. 60]
>>> lon: [ 10.. 140]
>>> Number Of Attributes: 3
>>> _FillValue : 999
>>> units : W/m^2
>>> long_name : OLR
>>> (0) olr: min=110.786 max=343.597
>>> (0) ---
>>> (0)
>>> (0) ===> Robust Dispersion Statistics: OLR <===
>>> (0) [0] Mean=254.263
>>> .
>>> .
>>> .
>>> (0) [29] Kurtosis=-0.392987
>>> (0)
>>> (0) ---
>>> (0) fbase=K1VHR_05OCT2013_DAILY_L03_OLR
>>> (0) ===========================
>>> (0) ---
>>>
>>> Variable: Latitude
>>> Type: float
>>> Total Size: 1002404 bytes
>>> 250601 values
>>> Number of Dimensions: 1
>>> Dimensions and sizes: [DIM_000 | 250601]
>>> Coordinates:
>>> Number Of Attributes: 0
>>> (0) Latitude: min=-60 max=60
>>> (0) ---
>>>
>>> Variable: Longitude
>>> Type: float
>>> Total Size: 1002404 bytes
>>> 250601 values
>>> Number of Dimensions: 1
>>> Dimensions and sizes: [DIM_000 | 250601]
>>> Coordinates:
>>> Number Of Attributes: 0
>>> (0) Longitude: min=10 max=140
>>> (0) ---
>>> (0) ---
>>> Segmentation fault (core dumped)
>>>
>>> **************************************************************************
>>>
>>> regards
>>> Paul.
>>>
>>> On 2/24/15, Kyle Griffin <ksgriffin2 at wisc.edu> wrote:
>>> > Look at what the error message is telling you. In this case, it says
>>> > "variable OLR1 is undefined" - which is your problem. OLR1 does not
>>> exist
>>> > anywhere else in your script! It appears you are trying to write it
>>> out to
>>> > the output file correctly, but there is no data to output because you
>>> never
>>> > read that variable in!
>>> >
>>> > Some info on reading (and writing, but that's not relevant for you at
>>> the
>>> > moment) HDF5 files is available here:
>>> >
>>> > http://www.ncl.ucar.edu/Applications/hdf5.shtml
>>> >
>>> > It is not the most organized of pages, but you can get the general
>>> gist of
>>> > things if you have a more complicated HDF5 file with groups. Otherwise,
>>> > note that the typical notation for reading in a variable from file(s)
>>> is
>>> > like this, using your code from above:
>>> >
>>> > f = addfile(diri+fili(nf), "r") ;same as above
>>> > OLR1 = f->OLR_var_name ;change to match the variable name desired
>>> from
>>> > your file
>>> >
>>> > You may need/want other coordinate variables from the file as well. I
>>> don't
>>> > have much experience with HDF5 myself, but if you have some issues,
>>> you can
>>> > look into resolving them with some tips from the general file I/O page
>>> as
>>> > well:
>>> >
>>> > http://www.ncl.ucar.edu/Applications/list_io.shtml
>>> >
>>> > Please respond back to the list if you have more questions!
>>> >
>>> >
>>> > Kyle
>>> >
>>> > ----------------------------------------
>>> > Kyle S. Griffin
>>> > Department of Atmospheric and Oceanic Sciences
>>> > University of Wisconsin - Madison
>>> > Room 1421
>>> > 1225 W Dayton St, Madison, WI 53706
>>> > Email: ksgriffin2 at wisc.edu
>>> >
>>> > On Mon, Feb 23, 2015 at 10:56 PM, Steeven Paul Yerraguntla <
>>> > steevenpaul at gmail.com> wrote:
>>> >
>>> >> Dear NCL users,
>>> >> I wanted to convert number of OLR hdf(.h5) files to netcdf
>>> >> files
>>> >> and to get it done i used the following script and facing error at the
>>> >> line
>>> >> marked in the following script. I could not make out how to solve it.
>>> >> Please help me in solving out it. Thanks in advance.
>>> >>
>>> >> =================================================================
>>> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>>> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>>> >> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>>> >> ;************************************************
>>> >> begin
>>> >> diri = "./"
>>> >> diro = "./"
>>> >>
>>> >> fili = systemfunc("cd "+diri+" ; ls *.h5")
>>> >> nfil = dimsizes(fili)
>>> >> print("nfil="+nfil)
>>> >>
>>> >> do nf=0,nfil-1
>>> >> print("===========================")
>>> >> f = addfile(diri+fili(nf), "r")
>>> >>
>>> >> print("---")
>>> >> fbase = str_get_field(fili(nf),1,".")
>>> >> print("fbase="+fbase)
>>> >>
>>> >> diro = "./"
>>> >> filo = fbase+".nc"
>>> >> ptho = diro+filo
>>> >>
>>> >> system("/bin/rm -f "+ptho) ; remove any pre-existing file
>>> >> ncdf = addfile(ptho ,"c") ; open output netCDF file
>>> >>
>>> >> ; create global attributes
>>> of
>>> >> the file
>>> >> fAtt = True ; assign file attributes
>>> >> fAtt at title = "NCL: H5 to netCDF"
>>> >> fAtt at source = "MOSDAC: Meteorological & Oceanographic
>>> >> Satellite Data Archival Center (India)"
>>> >> fAtt at source_file = fili(nf)
>>> >> fAtt at Conventions = "None"
>>> >> fAtt at creation_date = systemfunc ("date")
>>> >> fileattdef( ncdf, fAtt ) ; copy file attributes
>>> >>
>>> >> *ncdf->OLR = OLR1 <============== This is where error is
>>> >> occuring*
>>> >> end do
>>> >> end
>>> >> =====================================
>>> >> *Error message as follows:*
>>> >>
>>> >> (0) nfil=31
>>> >> (0) ===========================
>>> >> (0) ---
>>> >> (0) fbase=K1VHR_05OCT2013_DAILY_L03_OLR
>>> >> fatal:Variable (OLR1) is undefined
>>> >> fatal:["Execute.c":7743]:Execute: Error occurred at or near line 41 in
>>> >> file nfiles_conversion_h5_to_nc.ncl
>>> >>
>>> >> _______________________________________________
>>> >> ncl-talk mailing list
>>> >> List instructions, subscriber options, unsubscribe:
>>> >> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>> >>
>>> >>
>>> >
>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>> *
>>> ***
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150226/8bde22fb/attachment.html
More information about the ncl-talk
mailing list