[ncl-talk] Problems converting hdf to netcdf using NCL

Dennis Shea shea at ucar.edu
Wed May 27 09:07:25 MDT 2015


I think you have to do some debugging.

===

Look at "time":   time: [20143050010..2.426116152246728e-315]
The last time is obviously wrong! How did this happen?
It did not happen with the 5 sample files ... why with all the files?

Each iteration thru the loop .... print the current 'time' value.

Also ... after the loop is complete ...

print(time)   ; print all values

This issue must be cleared up!

If 'time' is messed up ... what about other variables?

-------

In the code that reads/plots the netCDF ... did you do any debugging??

    diri  = "./"
    fili  = "MODIS_L2.Swath.nc"
    f     = addfile(diri+fili, "r")

    print(f)

    AOT = f->AOT
    printVarSummary(AOT)
    printMinMax(AOT,0)

   lat2d = f->LAT2D
   printVarSummary(LAT2D)
   printMinMax(lat2d,0)

   lon2d = f->LON2D
   printVarSummary(lon2d)
   printMinMax(lon2d,0)

Do these not match the original created values

--------
Minor: the dimension name "Cell_ACross_Swath"  should be
"Cell_Across_Swath"


On Wed, May 27, 2015 at 8:34 AM, BLIUJUS, STEVEN D CTR USAF AFMC
AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:

> Dennis the script works however for some reason the data is not getting
> put into the NetCDF file correctly. I have ~120 files I'm going through,
> and when the script prints the dimensions, the following are stated:
>
> Variable: time
> Type: double
> Total Size: 2080 bytes
>             260 values
> Number of Dimensions: 1
> Dimensions and sizes:   [time | 260]
> Coordinates:
>             time: [20143050010..2.426116152246728e-315]
> Number Of Attributes: 1
>   long_name :   current time as YYYYDDDHHMM
>
> Variable: data
> Type: float
> Total Size: 28641600 bytes
>             7160400 values
> Number of Dimensions: 3
> Dimensions and sizes:   [time | 260] x [Cell_Along_Swath | 204] x
> [Cell_ACross_Swath | 135]
> Coordinates:
>             time: [20143050010..2.426116152246728e-315]
> Number Of Attributes: 2
>   long_name :    AOT at 0.55 micron for both ocean and land
>   _FillValue :  -9999
> (0)     data: min=-0.05   max=4.277
>
> Variable: lat2d
> Type: float
> Total Size: 28641600 bytes
>             7160400 values
> Number of Dimensions: 3
> Dimensions and sizes:   [time | 260] x [Cell_Along_Swath | 204] x
> [Cell_ACross_Swath | 135]
> Coordinates:
>             time: [20143050010..2.426116152246728e-315]
> Number Of Attributes: 3
>   units :       degrees_north
>   long_name :   Geodetic Latitude
>   _FillValue :  -999
> (0)     lat2d: min=-89.894   max=77.2616
>
> Variable: lon2d
> Type: float
> Total Size: 28641600 bytes
>             7160400 values
> Number of Dimensions: 3
> Dimensions and sizes:   [time | 260] x [Cell_Along_Swath | 204] x
> [Cell_ACross_Swath | 135]
> Coordinates:
>             time: [20143050010..2.426116152246728e-315]
> Number Of Attributes: 3
>   units :       degrees_east
>   long_name :   Geodetic Longitude
>   _FillValue :  -999
> (0)     lon2d: min=-179.999   max=179.977
> (0)     -----
>
>
> The thing that stands out is when the NetCDF file is created, the
> Longitude in the file only goes from 85.1406 - 172.263 and not -179.999 -
> 179.977 like the varSummary states. There is an error that gets thrown
> during the run:
>
> fatal:Dimension size mismatch on subscript #1, left-hand and right-hand
> side dimensions do not match
> fatal:["Execute.c":8128]:Execute: Error occurred at or near line 71 in
> file modis.ncl
>
> I have attached the image created from your script, compared to the image
> I get from the other script. I also reattached your script as I modified
> some directory things and added the libraries which shifted some of the
> lines. It should be noted that the information present from your script
> does match the data location on the image I created from a different
> script, it's just missing everything else.
>
>
> Steve
>
> -----Original Message-----
> From: Dennis Shea [mailto:shea at ucar.edu]
> Sent: Tuesday, May 26, 2015 5:43 PM
> To: BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL
> Cc: ncl-talk at ucar.edu
> Subject: Re: [ncl-talk] Problems converting hdf to netcdf using NCL
>
> I just looked at the 5 sample files you sent ....
>
> The dimension sizes are *not* the same size across all files
>
>
> The 1st 4files are:
>
>
> MOD04_L2.A2014305.0000.006.2015077213911.pscs_000500937810.hdf
>   dimensions:
>       Cell_Along_Swath = 203       <=====
>       Cell_Across_Swath = 135
>
> The last file is:
>
> MOD04_L2.A2014305.0500.006.2015077215106.pscs_000500937810
>    dimensions:
>       Cell_Along_Swath = 204     <====
>       Cell_Across_Swath= 135
>
>
> That can be 'programmed around'.
>
>
> See attached
>
>
>
>
> On Tue, May 26, 2015 at 6:34 AM, BLIUJUS, STEVEN D CTR USAF AFMC
> AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:
>
>
>         Dennis,
>
>         #2 and #3: Cell_Along_Swath, and Cell_Across_Swath are exactly the
> same for every file as far as actual dimension size, the variables are not
> though.
>         #4: I am trying to create a mosaic of all the files and that is
> why I am wanting to generate one NetCDF file. So when I run my script the
> image above is what comes out. This image is what I want to create in a
> NetCDF file (Globe.PNG).
>         #5: I did this because I was following a script online that I
> found where I needed to give the main variable(Optical Depth) the
> latitude/longitude dimensions. It used (::5) which what I just followed.
> When I try plotting using just (:,:), the plot take an extremely long time
> to plot(though there is much more data prevalent)...If it is one file it
> doesn’t take long, but when you add them all in the mosaic form, it takes a
> much more substantial amount of time to generate the image.
>         #6: I'm not sure exactly what you mean by this as there is not
> time field in the file.
>         #7: See 5 attached MODIS files
>         #8: I am trying to create a NetCDF file for a coworker to use for
> his verification software. He only reads in NetCDF files or text files
> (though he would prefer NetCDF files)
>
>         I also attached the script (with all previous commented lines
> removed) that plots the files and also creates a text table of the data.
>
>         Any help would be appreciated.
>
>         Steve
>
>
>
>         -----Original Message-----
>         From: Dennis Shea [mailto:shea at ucar.edu]
>
>         Sent: Saturday, May 23, 2015 2:30 PM
>         To: BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL
>         Cc: ncl-talk at ucar.edu
>         Subject: Re: [ncl-talk] Problems converting hdf to netcdf using NCL
>
>         I have looked at this and there are *lots* of issues.
>
>         "... taking several hdf files (200-400) and trying to generate one
> NetCDF file."
>
>
>         So ...  **Conceptually**, you want
>
>
>             AOT(time, Cell_Along_Swath, Cell_Across_Swath)
>
>
>         Thing  is your geographic coordinates are 2-dimensional. see below
>
>
>
>
>         [1]
>
>         The files contain swath data. From your dump:
>
>         path:
>  MOD04_L2.A2014306.2330.006.2015077230656.pscs_000500937810.hdf
>            file global attributes:
>               HDFEOSVersion : HDFEOS_V2.17
>               StructMetadata_0 : GROUP=SwathStructure
>                 GROUP=SWATH_1
>                         SwathName="mod04"
>
>
>         [2] Are the dimension sizes
>
>                (Cell_Along_Swath,Cell_Across_Swath) => (203,135)
>
>
>              exactly the same for *all* files?
>
>
>         [3] If [2] is satisfied  .... Are the *values* contained within
> the following variables
>
>                Latitude ( Cell_Along_Swath, Cell_Across_Swath)
>
>                Longitude ( Cell_Along_Swath, Cell_Across_Swath )
>
>
>               *exactly* the same for *all* files?  ie: homogeneous space!
>
>
>              My (limited) experience with satellite swath data is that the
> answer is "no"
>
>
>         [4] If [2] *and* [3] are not satisfied, I see no advantage of
> creating a single netCDF file.
>
>              You would end up with 200-400 sets of dimension sizes *and*
> 200-400 variables
>
>              associated 200-400two-dimensional spatial variables.
>
>
>         [5]  I have no idea why you are sub-sampling [decimating] the
> array by 5:
>                (::5,::5) the resolution from (203,135) to  ??(41,27)??.
> This seems drastic!
>
>               Sub-sampling (array decimation) is commonly used used when
> the resolution is (say) (2500,4000) and the data are reasonably smooth.
>
>
>
>         [6] In any case you should be using
>
>               ListSetType(a,"join")
>
>
>              to add a record dimension ... which you should fill in with
> 'time'
>
>              (parse file name) or some sequential number.
>
>
>              Please read
>
> https://www.ncl.ucar.edu/Document/Functions/Built-in/addfiles.shtml
>
>              Example 3 illustrates what "join" will do.
>
>
>
>         [7] I do not have your ?MODIS? data file. Based upon your dump, I
> created a script that creates a netCDF file for each HDF file and (I hope)
> a sample graphic.
>
>
>         [8] I am not sure of ultimate use of the data but from an NCL
> perspective, there is no reason to convert to netCDF.
>
>
>         ===========================
>
>
>         Please upgrade to 6.3.0
>
>         http://www.ncl.ucar.edu/Download/
>
>         ============================
>
>
>
>
>
>         On Fri, May 22, 2015 at 9:59 AM, BLIUJUS, STEVEN D CTR USAF AFMC
> AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:
>
>
>                 I wanted to provide an example of what is in the hdf file.
> See attached.
>
>                 Steve
>
>
>                 -----Original Message-----
>                 From: ncl-talk-bounces at ucar.edu [mailto:
> ncl-talk-bounces at ucar.edu] On Behalf Of BLIUJUS, STEVEN D CTR USAF AFMC
> AFLCMC/HBAW-OL
>                 Sent: Friday, May 22, 2015 10:36 AM
>                 To: ncl-talk at ucar.edu
>                 Subject: [ncl-talk] Problems converting hdf to netcdf
> using NCL
>
>                 I am taking several hdf files (200-400) and trying to
> generate one NetCDF file. I am able to plot it on a grid, but when I try to
> write to a NetCDF file, I get the following error:
>
>                 fatal:Number of dimensions in parameter (2) of
> (filedimdef) is (2), (1) dimensions were expected
>                 fatal:["Execute.c":8128]:Execute: Error occurred at or
> near line 82 in file File.ncl
>
>
>                 The data I am creating is AOD with two dimensions
> (lat/lon). I am uncertain as to why it is expecting one dimension. Below is
> my script. FYI, the original files had to be manipulated in order to make
> AOD have the lat/lon dimensions. Any help would be greatly appreciated.
>
>
>                 load
> "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>                 load
> "/home/bliujuss/ncl/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
>                 load
> "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>                 load
> "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/shea_util.ncl"
>                 begin
>
>                 diri = "/home/bliujuss/Verification_data/Test/"
>                 fili = systemfunc("ls "+diri+"*.hdf")
>                 a = addfiles(fili,"r")
>
>                 ;ListSetType(a,"cat")
>
>                 var1=a[:]->Optical_Depth_Land_And_Ocean(:,:)
>                 Optical_Depth_Land_And_Ocean = short2flt(var1)
>
>                 var2=a[:]->Longitude(:,:)
>                 Longitude=var2
>
>
>                 var3 = a[:]->Latitude(:,:)
>                 Latitude=var3
>
>                 dims = dimsizes(Optical_Depth_Land_And_Ocean)
>                 nrows = dims(0) ;203
>                 npixels = dims(1) ;135
>
>                 lat2d = new((/nrows,npixels/),float)
>                 lon2d = new((/nrows,npixels/),float)
>
>                 do i=0,npixels-1
>                   lat2d(:,i) = Latitude(:,i)
>                 end do
>
>                 do i=0,nrows-1
>                   lon2d(i,:) = Longitude(i,:)
>                 end do
>
>                 Optical_Depth_Land_And_Ocean at lat2d = lat2d(::5,::5)
> Optical_Depth_Land_And_Ocean at lon2d = lon2d(::5,::5)
>
>
>                 wks = gsn_open_wks("x11","plot_aod")
>                 gsn_define_colormap(wks,"NCV_banded")
>                 setvalues NhlGetWorkspaceObjectId()
>                   "wsMaximumSize": 300000000
>                 end setvalues
>
>                 res                   = True               ; plot mods
> desired
>                 res at cnLinesOn         = False              ; turn off
> contour lines
>                 res at cnFillOn          = True               ; color plot
> desired
>                 res at cnLineLabelsOn    = False              ; turn off
> contour lines
>                 res at gsnAddCyclic      = False             ; non-global
> data
>                 res at gsnSpreadColors   = True               ; use full
> range of colormap
>                 ;res at cnFillMode        = "RasterFill"       ; turn on
> raster mode
>                 res at lbLabelAutoStride = True               ; nice spacing
> of lables
>                 res at gsnMaximize       = True               ; blow up plot
> as much as poss
>                 ;print(min(lat2d))
>                 ;print(max(lat2d))
>                 ;print(min(lon2d))
>                 ;print(max(lon2d))
>
>                 plot =
> gsn_csm_contour_map_ce(wks,Optical_Depth_Land_And_Ocean(::5,::5),res)
>                 print(max(Optical_Depth_Land_And_Ocean(::5,::5)))
>
>                 nlat = dimsizes(lat2d(::5,::5))
>                 nlon = dimsizes(lon2d(::5,::5))
>
>                 diro = "/home/bliujuss/Plots/"
>                 filo = "example.nc"
>                 system("rm -f" + diro + filo)
>                 fout = addfile (diro + filo, "c")
>
>                 setfileoption(fout,"DefineMode",True)
>
>                 dim_names = (/"lat", "lon"/)
>                 dim_sizes = (/nlat, nlon/)
>                 dimUnlim = (/False, False/)
>                 filedimdef(fout,dim_names,dim_sizes,dimUnlim)
>
>                 filevardef(fout, "lat",
> typeof(lat2d(::5,::5)),getvardims(lat2d(::5,::5)))
>                 filevardef(fout, "lon",
> typeof(lon2d(::5,::5)),getvardims(lon2d(::5,::5)))
>                 filevardef(fout, "AOT",
> typeof(Optical_Depth_Land_And_Ocean(::5,::5)),getvardims(Optical_Depth_Land_And_Ocean(::5,::5)))
>
>                 filevarattdef(fout,"AOT",
> Optical_Depth_Land_And_Ocean(::5,::5))
>                 filevarattdef(fout,"lat", lat2d(::5,::5))
> filevarattdef(fout,"lon", lon2d(::5,::5))
>
>                 setfileoption(fout,"DefineMode",False)
>
>                 fout->Optical_Depth_Land_And_Ocean(::5,::5) =
>
>                 fout->(/Optical_Depth_Land_And_Ocean(::5,::5)/)
>
>                 fout->lat2d(::5,::5) = (/lat2d(::5,::5)/)
>                 fout->lon2d(::5,::5) = (/lon2d(::5,::5)/)
>
>                 end
>
>
>                 Steven Bliujus
>
>
>                 _______________________________________________
>                 ncl-talk mailing list
>                 List instructions, subscriber options, unsubscribe:
>                 http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>                 _______________________________________________
>                 ncl-talk mailing list
>                 List instructions, subscriber options, unsubscribe:
>                 http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150527/3521fb5a/attachment-0001.html 


More information about the ncl-talk mailing list