[ncl-talk] Problems converting hdf to netcdf using NCL

BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL steven.bliujus.3.ctr at us.af.mil
Wed May 27 12:39:30 MDT 2015


Found the error Dennis, there was a discrepancy in the dimension size. One of the files has a 205 dimension, so I reformatted the do loop section to look like:

mAcross  = 135
  nAlong   = 203
  nAlong1  = nAlong + 2

  time  = new (   nfili                  , "double", "No_FillValue"   )
  data  = new ( (/nfili,nAlong1,mAcross/), "float" , -9999.0)
  lat2d = new ( (/nfili,nAlong1,mAcross/), "float" ,  -999.0)
  lon2d = new ( (/nfili,nAlong1,mAcross/), "float" ,  -999.0)
  print("nfili = ")
  print(nfili)
  do nf=0,nfili-1                  ; loop over files
     a = addfile(diri+fili(nf),"r")
     time(nf)  = (/ parse_MODIS( fili(nf) ) /)
     print("nf="+nf+"  time(nf)="+time(nf))

     x        := short2flt(a->$var$)
     dimx      = dimsizes(x)
     print("dimsizex="+dimx)

     if (dimx(0).eq.203 .and. dimx(1).eq.135) then
         data(nf,0:202,:)  = (/ x /)
         lat2d(nf,0:202,:) = (/ a->Latitude  /)
         lon2d(nf,0:202,:) = (/ a->Longitude /)
     else if(dimx(0).eq.204 .and. dimx(1).eq.135) then
         data(nf,0:203:,:)  = (/ x /)
         lat2d(nf,0:203,:) = (/ a->Latitude  /)
         lon2d(nf,0:203,:) = (/ a->Longitude /)
     else
         data(nf,:,:)  = (/ x /)
         lat2d(nf,:,:) = (/ a->Latitude  /)
         lon2d(nf,:,:) = (/ a->Longitude /)
     end if
     end if
  end do

The script runs all the way through but I'm still finding a couple strange things. 
#1: The NetCDF file generated only shows ranges for Longitude :  85.1406-172.263 when using ncview, though there are values in there for the entire range...same as latitude, but since I know there are values there I'm not too concerned, just find it odd.
#2: When I do plot there still looks like there is missing data, I've attached both again. Maybe it's removing duplicate values?



-----Original Message-----
From: Dennis Shea [mailto:shea at ucar.edu] 
Sent: Wednesday, May 27, 2015 12:43 PM
To: BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL
Cc: ncl-talk at ucar.edu
Subject: Re: [ncl-talk] Problems converting hdf to netcdf using NCL

There  is no reason for stopping after 34 iterations.

===
[1]  When I wrote "Each iteration thru the loop .... print the current 'time' value."


I meant ...

  do nf=0,nfili-1                  ; loop over files
     a = addfile(diri+fili(nf),"r")
     time(nf)  = (/ parse_MODIS( fili(nf) ) /)
     print("nf="+nf+"  time(nf)="+time(nf))     ; print current value
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[2]  Also ... after the loop is complete ...  
                 ^^^^^


 end do

; manually add meta data

  time!0 = "time"
  time at long_name = "current time as YYYYDDDHHMM"
  time&time      = time
  printVarSummary(time)


print(time)   ; print all values



On Wed, May 27, 2015 at 10:08 AM, BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:


	Dennis,
	
	Is there a default setting that loops kill after? The loop stops after 34 iterations, thus leaving out the other 226 files, which explains why there is no data anywhere else on the globe. There are so many files because we are doing a global analysis, so these are swaths through the entire day.
	
	As far as the plots.ncl. These values do match the values given by modis.ncl as far as lat/lon/aod. So the real problem is the loop quitting early. I attached the log file I created.
	
	Steve
	
	-----Original Message-----
	From: Dennis Shea [mailto:shea at ucar.edu]
	
	Sent: Wednesday, May 27, 2015 10:07 AM
	To: BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL
	Cc: ncl-talk at ucar.edu
	Subject: Re: [ncl-talk] Problems converting hdf to netcdf using NCL
	
	I think you have to do some debugging.
	
	===
	
	
	Look at "time":   time: [20143050010..2.426116152246728e-315]
	
	The last time is obviously wrong! How did this happen?
	
	It did not happen with the 5 sample files ... why with all the files?
	
	
	Each iteration thru the loop .... print the current 'time' value.
	
	
	Also ... after the loop is complete ...
	
	
	print(time)   ; print all values
	
	
	This issue must be cleared up!
	
	
	If 'time' is messed up ... what about other variables?
	
	
	-------
	
	
	In the code that reads/plots the netCDF ... did you do any debugging??
	
	    diri  = "./"
	    fili  = "MODIS_L2.Swath.nc"
	    f     = addfile(diri+fili, "r")
	
	
	    print(f)
	
	
	    AOT = f->AOT
	
	    printVarSummary(AOT)
	
	    printMinMax(AOT,0)
	
	
	   lat2d = f->LAT2D
	
	   printVarSummary(LAT2D)
	
	   printMinMax(lat2d,0)
	
	
	   lon2d = f->LON2D
	
	   printVarSummary(lon2d)
	   printMinMax(lon2d,0)
	
	
	Do these not match the original created values
	
	
	--------
	
	Minor: the dimension name "Cell_ACross_Swath"  should be "Cell_Across_Swath"
	
	
	On Wed, May 27, 2015 at 8:34 AM, BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:
	
	
	        Dennis the script works however for some reason the data is not getting put into the NetCDF file correctly. I have ~120 files I'm going through, and when the script prints the dimensions, the following are stated:
	
	        Variable: time
	        Type: double
	        Total Size: 2080 bytes
	                    260 values
	        Number of Dimensions: 1
	        Dimensions and sizes:   [time | 260]
	        Coordinates:
	                    time: [20143050010..2.426116152246728e-315]
	        Number Of Attributes: 1
	          long_name :   current time as YYYYDDDHHMM
	
	        Variable: data
	        Type: float
	        Total Size: 28641600 bytes
	                    7160400 values
	        Number of Dimensions: 3
	        Dimensions and sizes:   [time | 260] x [Cell_Along_Swath | 204] x [Cell_ACross_Swath | 135]
	        Coordinates:
	                    time: [20143050010..2.426116152246728e-315]
	        Number Of Attributes: 2
	          long_name :    AOT at 0.55 micron for both ocean and land
	          _FillValue :  -9999
	        (0)     data: min=-0.05   max=4.277
	
	        Variable: lat2d
	        Type: float
	        Total Size: 28641600 bytes
	                    7160400 values
	        Number of Dimensions: 3
	        Dimensions and sizes:   [time | 260] x [Cell_Along_Swath | 204] x [Cell_ACross_Swath | 135]
	        Coordinates:
	                    time: [20143050010..2.426116152246728e-315]
	        Number Of Attributes: 3
	          units :       degrees_north
	          long_name :   Geodetic Latitude
	          _FillValue :  -999
	        (0)     lat2d: min=-89.894   max=77.2616
	
	        Variable: lon2d
	        Type: float
	        Total Size: 28641600 bytes
	                    7160400 values
	        Number of Dimensions: 3
	        Dimensions and sizes:   [time | 260] x [Cell_Along_Swath | 204] x [Cell_ACross_Swath | 135]
	        Coordinates:
	                    time: [20143050010..2.426116152246728e-315]
	        Number Of Attributes: 3
	          units :       degrees_east
	          long_name :   Geodetic Longitude
	          _FillValue :  -999
	        (0)     lon2d: min=-179.999   max=179.977
	        (0)     -----
	
	
	        The thing that stands out is when the NetCDF file is created, the Longitude in the file only goes from 85.1406 - 172.263 and not -179.999 - 179.977 like the varSummary states. There is an error that gets thrown during the run:
	
	        fatal:Dimension size mismatch on subscript #1, left-hand and right-hand side dimensions do not match
	        fatal:["Execute.c":8128]:Execute: Error occurred at or near line 71 in file modis.ncl
	
	        I have attached the image created from your script, compared to the image I get from the other script. I also reattached your script as I modified some directory things and added the libraries which shifted some of the lines. It should be noted that the information present from your script does match the data location on the image I created from a different script, it's just missing everything else.
	
	
	        Steve
	
	        -----Original Message-----
	        From: Dennis Shea [mailto:shea at ucar.edu]
	
	        Sent: Tuesday, May 26, 2015 5:43 PM
	        To: BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL
	        Cc: ncl-talk at ucar.edu
	        Subject: Re: [ncl-talk] Problems converting hdf to netcdf using NCL
	
	        I just looked at the 5 sample files you sent ....
	
	        The dimension sizes are *not* the same size across all files
	
	
	        The 1st 4files are:
	
	
	        MOD04_L2.A2014305.0000.006.2015077213911.pscs_000500937810.hdf
	          dimensions:
	              Cell_Along_Swath = 203       <=====
	              Cell_Across_Swath = 135
	
	        The last file is:
	
	        MOD04_L2.A2014305.0500.006.2015077215106.pscs_000500937810
	           dimensions:
	              Cell_Along_Swath = 204     <====
	              Cell_Across_Swath= 135
	
	
	        That can be 'programmed around'.
	
	
	        See attached
	
	
	
	
	        On Tue, May 26, 2015 at 6:34 AM, BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:
	
	
	                Dennis,
	
	                #2 and #3: Cell_Along_Swath, and Cell_Across_Swath are exactly the same for every file as far as actual dimension size, the variables are not though.
	                #4: I am trying to create a mosaic of all the files and that is why I am wanting to generate one NetCDF file. So when I run my script the image above is what comes out. This image is what I want to create in a NetCDF file (Globe.PNG).
	                #5: I did this because I was following a script online that I found where I needed to give the main variable(Optical Depth) the latitude/longitude dimensions. It used (::5) which what I just followed. When I try plotting using just (:,:), the plot take an extremely long time to plot(though there is much more data prevalent)...If it is one file it doesn’t take long, but when you add them all in the mosaic form, it takes a much more substantial amount of time to generate the image.
	                #6: I'm not sure exactly what you mean by this as there is not time field in the file.
	                #7: See 5 attached MODIS files
	                #8: I am trying to create a NetCDF file for a coworker to use for his verification software. He only reads in NetCDF files or text files (though he would prefer NetCDF files)
	
	                I also attached the script (with all previous commented lines removed) that plots the files and also creates a text table of the data.
	
	                Any help would be appreciated.
	
	                Steve
	
	
	
	                -----Original Message-----
	                From: Dennis Shea [mailto:shea at ucar.edu]
	
	                Sent: Saturday, May 23, 2015 2:30 PM
	                To: BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL
	                Cc: ncl-talk at ucar.edu
	                Subject: Re: [ncl-talk] Problems converting hdf to netcdf using NCL
	
	                I have looked at this and there are *lots* of issues.
	
	                "... taking several hdf files (200-400) and trying to generate one NetCDF file."
	
	
	                So ...  **Conceptually**, you want
	
	
	                    AOT(time, Cell_Along_Swath, Cell_Across_Swath)
	
	
	                Thing  is your geographic coordinates are 2-dimensional. see below
	
	
	
	
	                [1]
	
	                The files contain swath data. From your dump:
	
	                path:   MOD04_L2.A2014306.2330.006.2015077230656.pscs_000500937810.hdf
	                   file global attributes:
	                      HDFEOSVersion : HDFEOS_V2.17
	                      StructMetadata_0 : GROUP=SwathStructure
	                        GROUP=SWATH_1
	                                SwathName="mod04"
	
	
	                [2] Are the dimension sizes
	
	                       (Cell_Along_Swath,Cell_Across_Swath) => (203,135)
	
	
	                     exactly the same for *all* files?
	
	
	                [3] If [2] is satisfied  .... Are the *values* contained within the following variables
	
	                       Latitude ( Cell_Along_Swath, Cell_Across_Swath)
	
	                       Longitude ( Cell_Along_Swath, Cell_Across_Swath )
	
	
	                      *exactly* the same for *all* files?  ie: homogeneous space!
	
	
	                     My (limited) experience with satellite swath data is that the answer is "no"
	
	
	                [4] If [2] *and* [3] are not satisfied, I see no advantage of creating a single netCDF file.
	
	                     You would end up with 200-400 sets of dimension sizes *and* 200-400 variables
	
	                     associated 200-400two-dimensional spatial variables.
	
	
	                [5]  I have no idea why you are sub-sampling [decimating] the array by 5:
	                       (::5,::5) the resolution from (203,135) to  ??(41,27)??. This seems drastic!
	
	                      Sub-sampling (array decimation) is commonly used used when the resolution is (say) (2500,4000) and the data are reasonably smooth.
	
	
	
	                [6] In any case you should be using
	
	                      ListSetType(a,"join")
	
	
	                     to add a record dimension ... which you should fill in with 'time'
	
	                     (parse file name) or some sequential number.
	
	
	                     Please read
	                        https://www.ncl.ucar.edu/Document/Functions/Built-in/addfiles.shtml
	
	                     Example 3 illustrates what "join" will do.
	
	
	
	                [7] I do not have your ?MODIS? data file. Based upon your dump, I created a script that creates a netCDF file for each HDF file and (I hope) a sample graphic.
	
	
	                [8] I am not sure of ultimate use of the data but from an NCL perspective, there is no reason to convert to netCDF.
	
	
	                ===========================
	
	
	                Please upgrade to 6.3.0
	
	                http://www.ncl.ucar.edu/Download/
	
	                ============================
	
	
	
	
	
	                On Fri, May 22, 2015 at 9:59 AM, BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:
	
	
	                        I wanted to provide an example of what is in the hdf file. See attached.
	
	                        Steve
	
	
	                        -----Original Message-----
	                        From: ncl-talk-bounces at ucar.edu [mailto:ncl-talk-bounces at ucar.edu] On Behalf Of BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL
	                        Sent: Friday, May 22, 2015 10:36 AM
	                        To: ncl-talk at ucar.edu
	                        Subject: [ncl-talk] Problems converting hdf to netcdf using NCL
	
	                        I am taking several hdf files (200-400) and trying to generate one NetCDF file. I am able to plot it on a grid, but when I try to write to a NetCDF file, I get the following error:
	
	                        fatal:Number of dimensions in parameter (2) of (filedimdef) is (2), (1) dimensions were expected
	                        fatal:["Execute.c":8128]:Execute: Error occurred at or near line 82 in file File.ncl
	
	
	                        The data I am creating is AOD with two dimensions (lat/lon). I am uncertain as to why it is expecting one dimension. Below is my script. FYI, the original files had to be manipulated in order to make AOD have the lat/lon dimensions. Any help would be greatly appreciated.
	
	
	                        load "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/gsn_code.ncl"
	                        load "/home/bliujuss/ncl/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
	                        load "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
	                        load "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/shea_util.ncl"
	                        begin
	
	                        diri = "/home/bliujuss/Verification_data/Test/"
	                        fili = systemfunc("ls "+diri+"*.hdf")
	                        a = addfiles(fili,"r")
	
	                        ;ListSetType(a,"cat")
	
	                        var1=a[:]->Optical_Depth_Land_And_Ocean(:,:)
	                        Optical_Depth_Land_And_Ocean = short2flt(var1)
	
	                        var2=a[:]->Longitude(:,:)
	                        Longitude=var2
	
	
	                        var3 = a[:]->Latitude(:,:)
	                        Latitude=var3
	
	                        dims = dimsizes(Optical_Depth_Land_And_Ocean)
	                        nrows = dims(0) ;203
	                        npixels = dims(1) ;135
	
	                        lat2d = new((/nrows,npixels/),float)
	                        lon2d = new((/nrows,npixels/),float)
	
	                        do i=0,npixels-1
	                          lat2d(:,i) = Latitude(:,i)
	                        end do
	
	                        do i=0,nrows-1
	                          lon2d(i,:) = Longitude(i,:)
	                        end do
	
	                        Optical_Depth_Land_And_Ocean at lat2d = lat2d(::5,::5) Optical_Depth_Land_And_Ocean at lon2d = lon2d(::5,::5)
	
	
	                        wks = gsn_open_wks("x11","plot_aod")
	                        gsn_define_colormap(wks,"NCV_banded")
	                        setvalues NhlGetWorkspaceObjectId()
	                          "wsMaximumSize": 300000000
	                        end setvalues
	
	                        res                   = True               ; plot mods desired
	                        res at cnLinesOn         = False              ; turn off contour lines
	                        res at cnFillOn          = True               ; color plot desired
	                        res at cnLineLabelsOn    = False              ; turn off contour lines
	                        res at gsnAddCyclic      = False             ; non-global data
	                        res at gsnSpreadColors   = True               ; use full range of colormap
	                        ;res at cnFillMode        = "RasterFill"       ; turn on raster mode
	                        res at lbLabelAutoStride = True               ; nice spacing of lables
	                        res at gsnMaximize       = True               ; blow up plot as much as poss
	                        ;print(min(lat2d))
	                        ;print(max(lat2d))
	                        ;print(min(lon2d))
	                        ;print(max(lon2d))
	
	                        plot = gsn_csm_contour_map_ce(wks,Optical_Depth_Land_And_Ocean(::5,::5),res)
	                        print(max(Optical_Depth_Land_And_Ocean(::5,::5)))
	
	                        nlat = dimsizes(lat2d(::5,::5))
	                        nlon = dimsizes(lon2d(::5,::5))
	
	                        diro = "/home/bliujuss/Plots/"
	                        filo = "example.nc"
	                        system("rm -f" + diro + filo)
	                        fout = addfile (diro + filo, "c")
	
	                        setfileoption(fout,"DefineMode",True)
	
	                        dim_names = (/"lat", "lon"/)
	                        dim_sizes = (/nlat, nlon/)
	                        dimUnlim = (/False, False/)
	                        filedimdef(fout,dim_names,dim_sizes,dimUnlim)
	
	                        filevardef(fout, "lat", typeof(lat2d(::5,::5)),getvardims(lat2d(::5,::5)))
	                        filevardef(fout, "lon", typeof(lon2d(::5,::5)),getvardims(lon2d(::5,::5)))
	                        filevardef(fout, "AOT", typeof(Optical_Depth_Land_And_Ocean(::5,::5)),getvardims(Optical_Depth_Land_And_Ocean(::5,::5)))
	
	                        filevarattdef(fout,"AOT", Optical_Depth_Land_And_Ocean(::5,::5))
	                        filevarattdef(fout,"lat", lat2d(::5,::5)) filevarattdef(fout,"lon", lon2d(::5,::5))
	
	                        setfileoption(fout,"DefineMode",False)
	
	                        fout->Optical_Depth_Land_And_Ocean(::5,::5) =
	
	                        fout->(/Optical_Depth_Land_And_Ocean(::5,::5)/)
	
	                        fout->lat2d(::5,::5) = (/lat2d(::5,::5)/)
	                        fout->lon2d(::5,::5) = (/lon2d(::5,::5)/)
	
	                        end
	
	
	                        Steven Bliujus
	
	
	                        _______________________________________________
	                        ncl-talk mailing list
	                        List instructions, subscriber options, unsubscribe:
	                        http://mailman.ucar.edu/mailman/listinfo/ncl-talk
	
	
	                        _______________________________________________
	                        ncl-talk mailing list
	                        List instructions, subscriber options, unsubscribe:
	                        http://mailman.ucar.edu/mailman/listinfo/ncl-talk
	
	
	
	
	
	
	
	
	
	
	


-------------- next part --------------
A non-text attachment was scrubbed...
Name: NetCDF_script.PNG
Type: image/png
Size: 117328 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150527/8ee9cf2c/attachment-0002.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Files1.PNG
Type: image/png
Size: 195861 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150527/8ee9cf2c/attachment-0003.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5647 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150527/8ee9cf2c/attachment-0001.bin 


More information about the ncl-talk mailing list