[ncl-talk] Problems converting hdf to netcdf using NCL

Dennis Shea shea at ucar.edu
Tue May 26 16:43:04 MDT 2015


I just looked at the 5 sample files you sent ....

The dimension sizes are *not* the same size across all files

The 1st 4files are:


MOD04_L2.A2014305.0000.006.2015077213911.pscs_000500937810.hdf
  dimensions:
      Cell_Along_Swath = 203       <=====
      Cell_Across_Swath = 135

The last file is:

MOD04_L2.A2014305.0500.006.2015077215106.pscs_000500937810
   dimensions:
      Cell_Along_Swath = 204     <====
      Cell_Across_Swath= 135

That can be 'programmed around'.

See attached



On Tue, May 26, 2015 at 6:34 AM, BLIUJUS, STEVEN D CTR USAF AFMC
AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:

> Dennis,
>
> #2 and #3: Cell_Along_Swath, and Cell_Across_Swath are exactly the same
> for every file as far as actual dimension size, the variables are not
> though.
> #4: I am trying to create a mosaic of all the files and that is why I am
> wanting to generate one NetCDF file. So when I run my script the image
> above is what comes out. This image is what I want to create in a NetCDF
> file (Globe.PNG).
> #5: I did this because I was following a script online that I found where
> I needed to give the main variable(Optical Depth) the latitude/longitude
> dimensions. It used (::5) which what I just followed. When I try plotting
> using just (:,:), the plot take an extremely long time to plot(though there
> is much more data prevalent)...If it is one file it doesn’t take long, but
> when you add them all in the mosaic form, it takes a much more substantial
> amount of time to generate the image.
> #6: I'm not sure exactly what you mean by this as there is not time field
> in the file.
> #7: See 5 attached MODIS files
> #8: I am trying to create a NetCDF file for a coworker to use for his
> verification software. He only reads in NetCDF files or text files (though
> he would prefer NetCDF files)
>
> I also attached the script (with all previous commented lines removed)
> that plots the files and also creates a text table of the data.
>
> Any help would be appreciated.
>
> Steve
>
>
>
> -----Original Message-----
> From: Dennis Shea [mailto:shea at ucar.edu]
> Sent: Saturday, May 23, 2015 2:30 PM
> To: BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL
> Cc: ncl-talk at ucar.edu
> Subject: Re: [ncl-talk] Problems converting hdf to netcdf using NCL
>
> I have looked at this and there are *lots* of issues.
>
> "... taking several hdf files (200-400) and trying to generate one NetCDF
> file."
>
>
> So ...  **Conceptually**, you want
>
>
>     AOT(time, Cell_Along_Swath, Cell_Across_Swath)
>
>
> Thing  is your geographic coordinates are 2-dimensional. see below
>
>
>
>
> [1]
>
> The files contain swath data. From your dump:
>
> path:   MOD04_L2.A2014306.2330.006.2015077230656.pscs_000500937810.hdf
>    file global attributes:
>       HDFEOSVersion : HDFEOS_V2.17
>       StructMetadata_0 : GROUP=SwathStructure
>         GROUP=SWATH_1
>                 SwathName="mod04"
>
>
> [2] Are the dimension sizes
>
>        (Cell_Along_Swath,Cell_Across_Swath) => (203,135)
>
>
>      exactly the same for *all* files?
>
>
> [3] If [2] is satisfied  .... Are the *values* contained within the
> following variables
>
>        Latitude ( Cell_Along_Swath, Cell_Across_Swath)
>
>        Longitude ( Cell_Along_Swath, Cell_Across_Swath )
>
>
>       *exactly* the same for *all* files?  ie: homogeneous space!
>
>
>      My (limited) experience with satellite swath data is that the answer
> is "no"
>
>
> [4] If [2] *and* [3] are not satisfied, I see no advantage of creating a
> single netCDF file.
>
>      You would end up with 200-400 sets of dimension sizes *and* 200-400
> variables
>
>      associated 200-400two-dimensional spatial variables.
>
>
> [5]  I have no idea why you are sub-sampling [decimating] the array by 5:
>        (::5,::5) the resolution from (203,135) to  ??(41,27)??. This seems
> drastic!
>
>       Sub-sampling (array decimation) is commonly used used when the
> resolution is (say) (2500,4000) and the data are reasonably smooth.
>
>
>
> [6] In any case you should be using
>
>       ListSetType(a,"join")
>
>
>      to add a record dimension ... which you should fill in with 'time'
>
>      (parse file name) or some sequential number.
>
>
>      Please read
>
> https://www.ncl.ucar.edu/Document/Functions/Built-in/addfiles.shtml
>
>      Example 3 illustrates what "join" will do.
>
>
>
> [7] I do not have your ?MODIS? data file. Based upon your dump, I created
> a script that creates a netCDF file for each HDF file and (I hope) a sample
> graphic.
>
>
> [8] I am not sure of ultimate use of the data but from an NCL perspective,
> there is no reason to convert to netCDF.
>
>
> ===========================
>
>
> Please upgrade to 6.3.0
>
> http://www.ncl.ucar.edu/Download/
>
> ============================
>
>
>
>
>
> On Fri, May 22, 2015 at 9:59 AM, BLIUJUS, STEVEN D CTR USAF AFMC
> AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:
>
>
>         I wanted to provide an example of what is in the hdf file. See
> attached.
>
>         Steve
>
>
>         -----Original Message-----
>         From: ncl-talk-bounces at ucar.edu [mailto:ncl-talk-bounces at ucar.edu]
> On Behalf Of BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL
>         Sent: Friday, May 22, 2015 10:36 AM
>         To: ncl-talk at ucar.edu
>         Subject: [ncl-talk] Problems converting hdf to netcdf using NCL
>
>         I am taking several hdf files (200-400) and trying to generate one
> NetCDF file. I am able to plot it on a grid, but when I try to write to a
> NetCDF file, I get the following error:
>
>         fatal:Number of dimensions in parameter (2) of (filedimdef) is
> (2), (1) dimensions were expected
>         fatal:["Execute.c":8128]:Execute: Error occurred at or near line
> 82 in file File.ncl
>
>
>         The data I am creating is AOD with two dimensions (lat/lon). I am
> uncertain as to why it is expecting one dimension. Below is my script. FYI,
> the original files had to be manipulated in order to make AOD have the
> lat/lon dimensions. Any help would be greatly appreciated.
>
>
>         load "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>         load "/home/bliujuss/ncl/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
>         load "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>         load "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/shea_util.ncl"
>         begin
>
>         diri = "/home/bliujuss/Verification_data/Test/"
>         fili = systemfunc("ls "+diri+"*.hdf")
>         a = addfiles(fili,"r")
>
>         ;ListSetType(a,"cat")
>
>         var1=a[:]->Optical_Depth_Land_And_Ocean(:,:)
>         Optical_Depth_Land_And_Ocean = short2flt(var1)
>
>         var2=a[:]->Longitude(:,:)
>         Longitude=var2
>
>
>         var3 = a[:]->Latitude(:,:)
>         Latitude=var3
>
>         dims = dimsizes(Optical_Depth_Land_And_Ocean)
>         nrows = dims(0) ;203
>         npixels = dims(1) ;135
>
>         lat2d = new((/nrows,npixels/),float)
>         lon2d = new((/nrows,npixels/),float)
>
>         do i=0,npixels-1
>           lat2d(:,i) = Latitude(:,i)
>         end do
>
>         do i=0,nrows-1
>           lon2d(i,:) = Longitude(i,:)
>         end do
>
>         Optical_Depth_Land_And_Ocean at lat2d = lat2d(::5,::5)
> Optical_Depth_Land_And_Ocean at lon2d = lon2d(::5,::5)
>
>
>         wks = gsn_open_wks("x11","plot_aod")
>         gsn_define_colormap(wks,"NCV_banded")
>         setvalues NhlGetWorkspaceObjectId()
>           "wsMaximumSize": 300000000
>         end setvalues
>
>         res                   = True               ; plot mods desired
>         res at cnLinesOn         = False              ; turn off contour
> lines
>         res at cnFillOn          = True               ; color plot desired
>         res at cnLineLabelsOn    = False              ; turn off contour
> lines
>         res at gsnAddCyclic      = False             ; non-global data
>         res at gsnSpreadColors   = True               ; use full range of
> colormap
>         ;res at cnFillMode        = "RasterFill"       ; turn on raster mode
>         res at lbLabelAutoStride = True               ; nice spacing of
> lables
>         res at gsnMaximize       = True               ; blow up plot as much
> as poss
>         ;print(min(lat2d))
>         ;print(max(lat2d))
>         ;print(min(lon2d))
>         ;print(max(lon2d))
>
>         plot =
> gsn_csm_contour_map_ce(wks,Optical_Depth_Land_And_Ocean(::5,::5),res)
>         print(max(Optical_Depth_Land_And_Ocean(::5,::5)))
>
>         nlat = dimsizes(lat2d(::5,::5))
>         nlon = dimsizes(lon2d(::5,::5))
>
>         diro = "/home/bliujuss/Plots/"
>         filo = "example.nc"
>         system("rm -f" + diro + filo)
>         fout = addfile (diro + filo, "c")
>
>         setfileoption(fout,"DefineMode",True)
>
>         dim_names = (/"lat", "lon"/)
>         dim_sizes = (/nlat, nlon/)
>         dimUnlim = (/False, False/)
>         filedimdef(fout,dim_names,dim_sizes,dimUnlim)
>
>         filevardef(fout, "lat",
> typeof(lat2d(::5,::5)),getvardims(lat2d(::5,::5)))
>         filevardef(fout, "lon",
> typeof(lon2d(::5,::5)),getvardims(lon2d(::5,::5)))
>         filevardef(fout, "AOT",
> typeof(Optical_Depth_Land_And_Ocean(::5,::5)),getvardims(Optical_Depth_Land_And_Ocean(::5,::5)))
>
>         filevarattdef(fout,"AOT", Optical_Depth_Land_And_Ocean(::5,::5))
>         filevarattdef(fout,"lat", lat2d(::5,::5))
> filevarattdef(fout,"lon", lon2d(::5,::5))
>
>         setfileoption(fout,"DefineMode",False)
>
>         fout->Optical_Depth_Land_And_Ocean(::5,::5) =
>
>         fout->(/Optical_Depth_Land_And_Ocean(::5,::5)/)
>
>         fout->lat2d(::5,::5) = (/lat2d(::5,::5)/)
>         fout->lon2d(::5,::5) = (/lon2d(::5,::5)/)
>
>         end
>
>
>         Steven Bliujus
>
>
>         _______________________________________________
>         ncl-talk mailing list
>         List instructions, subscriber options, unsubscribe:
>         http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>         _______________________________________________
>         ncl-talk mailing list
>         List instructions, subscriber options, unsubscribe:
>         http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150526/7c590edd/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bliujus_mosaic.ncl
Type: application/octet-stream
Size: 3905 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150526/7c590edd/attachment-0002.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bliujus_plot.ncl
Type: application/octet-stream
Size: 1932 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150526/7c590edd/attachment-0003.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bliujus_1.png
Type: image/png
Size: 85174 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150526/7c590edd/attachment-0001.png 


More information about the ncl-talk mailing list