[ncl-talk] Problems converting hdf to netcdf using NCL
Dennis Shea
shea at ucar.edu
Wed May 27 09:07:25 MDT 2015
I think you have to do some debugging.
===
Look at "time": time: [20143050010..2.426116152246728e-315]
The last time is obviously wrong! How did this happen?
It did not happen with the 5 sample files ... why with all the files?
Each iteration thru the loop .... print the current 'time' value.
Also ... after the loop is complete ...
print(time) ; print all values
This issue must be cleared up!
If 'time' is messed up ... what about other variables?
-------
In the code that reads/plots the netCDF ... did you do any debugging??
diri = "./"
fili = "MODIS_L2.Swath.nc"
f = addfile(diri+fili, "r")
print(f)
AOT = f->AOT
printVarSummary(AOT)
printMinMax(AOT,0)
lat2d = f->LAT2D
printVarSummary(LAT2D)
printMinMax(lat2d,0)
lon2d = f->LON2D
printVarSummary(lon2d)
printMinMax(lon2d,0)
Do these not match the original created values
--------
Minor: the dimension name "Cell_ACross_Swath" should be
"Cell_Across_Swath"
On Wed, May 27, 2015 at 8:34 AM, BLIUJUS, STEVEN D CTR USAF AFMC
AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:
> Dennis the script works however for some reason the data is not getting
> put into the NetCDF file correctly. I have ~120 files I'm going through,
> and when the script prints the dimensions, the following are stated:
>
> Variable: time
> Type: double
> Total Size: 2080 bytes
> 260 values
> Number of Dimensions: 1
> Dimensions and sizes: [time | 260]
> Coordinates:
> time: [20143050010..2.426116152246728e-315]
> Number Of Attributes: 1
> long_name : current time as YYYYDDDHHMM
>
> Variable: data
> Type: float
> Total Size: 28641600 bytes
> 7160400 values
> Number of Dimensions: 3
> Dimensions and sizes: [time | 260] x [Cell_Along_Swath | 204] x
> [Cell_ACross_Swath | 135]
> Coordinates:
> time: [20143050010..2.426116152246728e-315]
> Number Of Attributes: 2
> long_name : AOT at 0.55 micron for both ocean and land
> _FillValue : -9999
> (0) data: min=-0.05 max=4.277
>
> Variable: lat2d
> Type: float
> Total Size: 28641600 bytes
> 7160400 values
> Number of Dimensions: 3
> Dimensions and sizes: [time | 260] x [Cell_Along_Swath | 204] x
> [Cell_ACross_Swath | 135]
> Coordinates:
> time: [20143050010..2.426116152246728e-315]
> Number Of Attributes: 3
> units : degrees_north
> long_name : Geodetic Latitude
> _FillValue : -999
> (0) lat2d: min=-89.894 max=77.2616
>
> Variable: lon2d
> Type: float
> Total Size: 28641600 bytes
> 7160400 values
> Number of Dimensions: 3
> Dimensions and sizes: [time | 260] x [Cell_Along_Swath | 204] x
> [Cell_ACross_Swath | 135]
> Coordinates:
> time: [20143050010..2.426116152246728e-315]
> Number Of Attributes: 3
> units : degrees_east
> long_name : Geodetic Longitude
> _FillValue : -999
> (0) lon2d: min=-179.999 max=179.977
> (0) -----
>
>
> The thing that stands out is when the NetCDF file is created, the
> Longitude in the file only goes from 85.1406 - 172.263 and not -179.999 -
> 179.977 like the varSummary states. There is an error that gets thrown
> during the run:
>
> fatal:Dimension size mismatch on subscript #1, left-hand and right-hand
> side dimensions do not match
> fatal:["Execute.c":8128]:Execute: Error occurred at or near line 71 in
> file modis.ncl
>
> I have attached the image created from your script, compared to the image
> I get from the other script. I also reattached your script as I modified
> some directory things and added the libraries which shifted some of the
> lines. It should be noted that the information present from your script
> does match the data location on the image I created from a different
> script, it's just missing everything else.
>
>
> Steve
>
> -----Original Message-----
> From: Dennis Shea [mailto:shea at ucar.edu]
> Sent: Tuesday, May 26, 2015 5:43 PM
> To: BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL
> Cc: ncl-talk at ucar.edu
> Subject: Re: [ncl-talk] Problems converting hdf to netcdf using NCL
>
> I just looked at the 5 sample files you sent ....
>
> The dimension sizes are *not* the same size across all files
>
>
> The 1st 4files are:
>
>
> MOD04_L2.A2014305.0000.006.2015077213911.pscs_000500937810.hdf
> dimensions:
> Cell_Along_Swath = 203 <=====
> Cell_Across_Swath = 135
>
> The last file is:
>
> MOD04_L2.A2014305.0500.006.2015077215106.pscs_000500937810
> dimensions:
> Cell_Along_Swath = 204 <====
> Cell_Across_Swath= 135
>
>
> That can be 'programmed around'.
>
>
> See attached
>
>
>
>
> On Tue, May 26, 2015 at 6:34 AM, BLIUJUS, STEVEN D CTR USAF AFMC
> AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:
>
>
> Dennis,
>
> #2 and #3: Cell_Along_Swath, and Cell_Across_Swath are exactly the
> same for every file as far as actual dimension size, the variables are not
> though.
> #4: I am trying to create a mosaic of all the files and that is
> why I am wanting to generate one NetCDF file. So when I run my script the
> image above is what comes out. This image is what I want to create in a
> NetCDF file (Globe.PNG).
> #5: I did this because I was following a script online that I
> found where I needed to give the main variable(Optical Depth) the
> latitude/longitude dimensions. It used (::5) which what I just followed.
> When I try plotting using just (:,:), the plot take an extremely long time
> to plot(though there is much more data prevalent)...If it is one file it
> doesn’t take long, but when you add them all in the mosaic form, it takes a
> much more substantial amount of time to generate the image.
> #6: I'm not sure exactly what you mean by this as there is not
> time field in the file.
> #7: See 5 attached MODIS files
> #8: I am trying to create a NetCDF file for a coworker to use for
> his verification software. He only reads in NetCDF files or text files
> (though he would prefer NetCDF files)
>
> I also attached the script (with all previous commented lines
> removed) that plots the files and also creates a text table of the data.
>
> Any help would be appreciated.
>
> Steve
>
>
>
> -----Original Message-----
> From: Dennis Shea [mailto:shea at ucar.edu]
>
> Sent: Saturday, May 23, 2015 2:30 PM
> To: BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL
> Cc: ncl-talk at ucar.edu
> Subject: Re: [ncl-talk] Problems converting hdf to netcdf using NCL
>
> I have looked at this and there are *lots* of issues.
>
> "... taking several hdf files (200-400) and trying to generate one
> NetCDF file."
>
>
> So ... **Conceptually**, you want
>
>
> AOT(time, Cell_Along_Swath, Cell_Across_Swath)
>
>
> Thing is your geographic coordinates are 2-dimensional. see below
>
>
>
>
> [1]
>
> The files contain swath data. From your dump:
>
> path:
> MOD04_L2.A2014306.2330.006.2015077230656.pscs_000500937810.hdf
> file global attributes:
> HDFEOSVersion : HDFEOS_V2.17
> StructMetadata_0 : GROUP=SwathStructure
> GROUP=SWATH_1
> SwathName="mod04"
>
>
> [2] Are the dimension sizes
>
> (Cell_Along_Swath,Cell_Across_Swath) => (203,135)
>
>
> exactly the same for *all* files?
>
>
> [3] If [2] is satisfied .... Are the *values* contained within
> the following variables
>
> Latitude ( Cell_Along_Swath, Cell_Across_Swath)
>
> Longitude ( Cell_Along_Swath, Cell_Across_Swath )
>
>
> *exactly* the same for *all* files? ie: homogeneous space!
>
>
> My (limited) experience with satellite swath data is that the
> answer is "no"
>
>
> [4] If [2] *and* [3] are not satisfied, I see no advantage of
> creating a single netCDF file.
>
> You would end up with 200-400 sets of dimension sizes *and*
> 200-400 variables
>
> associated 200-400two-dimensional spatial variables.
>
>
> [5] I have no idea why you are sub-sampling [decimating] the
> array by 5:
> (::5,::5) the resolution from (203,135) to ??(41,27)??.
> This seems drastic!
>
> Sub-sampling (array decimation) is commonly used used when
> the resolution is (say) (2500,4000) and the data are reasonably smooth.
>
>
>
> [6] In any case you should be using
>
> ListSetType(a,"join")
>
>
> to add a record dimension ... which you should fill in with
> 'time'
>
> (parse file name) or some sequential number.
>
>
> Please read
>
> https://www.ncl.ucar.edu/Document/Functions/Built-in/addfiles.shtml
>
> Example 3 illustrates what "join" will do.
>
>
>
> [7] I do not have your ?MODIS? data file. Based upon your dump, I
> created a script that creates a netCDF file for each HDF file and (I hope)
> a sample graphic.
>
>
> [8] I am not sure of ultimate use of the data but from an NCL
> perspective, there is no reason to convert to netCDF.
>
>
> ===========================
>
>
> Please upgrade to 6.3.0
>
> http://www.ncl.ucar.edu/Download/
>
> ============================
>
>
>
>
>
> On Fri, May 22, 2015 at 9:59 AM, BLIUJUS, STEVEN D CTR USAF AFMC
> AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:
>
>
> I wanted to provide an example of what is in the hdf file.
> See attached.
>
> Steve
>
>
> -----Original Message-----
> From: ncl-talk-bounces at ucar.edu [mailto:
> ncl-talk-bounces at ucar.edu] On Behalf Of BLIUJUS, STEVEN D CTR USAF AFMC
> AFLCMC/HBAW-OL
> Sent: Friday, May 22, 2015 10:36 AM
> To: ncl-talk at ucar.edu
> Subject: [ncl-talk] Problems converting hdf to netcdf
> using NCL
>
> I am taking several hdf files (200-400) and trying to
> generate one NetCDF file. I am able to plot it on a grid, but when I try to
> write to a NetCDF file, I get the following error:
>
> fatal:Number of dimensions in parameter (2) of
> (filedimdef) is (2), (1) dimensions were expected
> fatal:["Execute.c":8128]:Execute: Error occurred at or
> near line 82 in file File.ncl
>
>
> The data I am creating is AOD with two dimensions
> (lat/lon). I am uncertain as to why it is expecting one dimension. Below is
> my script. FYI, the original files had to be manipulated in order to make
> AOD have the lat/lon dimensions. Any help would be greatly appreciated.
>
>
> load
> "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load
> "/home/bliujuss/ncl/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
> load
> "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> load
> "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/shea_util.ncl"
> begin
>
> diri = "/home/bliujuss/Verification_data/Test/"
> fili = systemfunc("ls "+diri+"*.hdf")
> a = addfiles(fili,"r")
>
> ;ListSetType(a,"cat")
>
> var1=a[:]->Optical_Depth_Land_And_Ocean(:,:)
> Optical_Depth_Land_And_Ocean = short2flt(var1)
>
> var2=a[:]->Longitude(:,:)
> Longitude=var2
>
>
> var3 = a[:]->Latitude(:,:)
> Latitude=var3
>
> dims = dimsizes(Optical_Depth_Land_And_Ocean)
> nrows = dims(0) ;203
> npixels = dims(1) ;135
>
> lat2d = new((/nrows,npixels/),float)
> lon2d = new((/nrows,npixels/),float)
>
> do i=0,npixels-1
> lat2d(:,i) = Latitude(:,i)
> end do
>
> do i=0,nrows-1
> lon2d(i,:) = Longitude(i,:)
> end do
>
> Optical_Depth_Land_And_Ocean at lat2d = lat2d(::5,::5)
> Optical_Depth_Land_And_Ocean at lon2d = lon2d(::5,::5)
>
>
> wks = gsn_open_wks("x11","plot_aod")
> gsn_define_colormap(wks,"NCV_banded")
> setvalues NhlGetWorkspaceObjectId()
> "wsMaximumSize": 300000000
> end setvalues
>
> res = True ; plot mods
> desired
> res at cnLinesOn = False ; turn off
> contour lines
> res at cnFillOn = True ; color plot
> desired
> res at cnLineLabelsOn = False ; turn off
> contour lines
> res at gsnAddCyclic = False ; non-global
> data
> res at gsnSpreadColors = True ; use full
> range of colormap
> ;res at cnFillMode = "RasterFill" ; turn on
> raster mode
> res at lbLabelAutoStride = True ; nice spacing
> of lables
> res at gsnMaximize = True ; blow up plot
> as much as poss
> ;print(min(lat2d))
> ;print(max(lat2d))
> ;print(min(lon2d))
> ;print(max(lon2d))
>
> plot =
> gsn_csm_contour_map_ce(wks,Optical_Depth_Land_And_Ocean(::5,::5),res)
> print(max(Optical_Depth_Land_And_Ocean(::5,::5)))
>
> nlat = dimsizes(lat2d(::5,::5))
> nlon = dimsizes(lon2d(::5,::5))
>
> diro = "/home/bliujuss/Plots/"
> filo = "example.nc"
> system("rm -f" + diro + filo)
> fout = addfile (diro + filo, "c")
>
> setfileoption(fout,"DefineMode",True)
>
> dim_names = (/"lat", "lon"/)
> dim_sizes = (/nlat, nlon/)
> dimUnlim = (/False, False/)
> filedimdef(fout,dim_names,dim_sizes,dimUnlim)
>
> filevardef(fout, "lat",
> typeof(lat2d(::5,::5)),getvardims(lat2d(::5,::5)))
> filevardef(fout, "lon",
> typeof(lon2d(::5,::5)),getvardims(lon2d(::5,::5)))
> filevardef(fout, "AOT",
> typeof(Optical_Depth_Land_And_Ocean(::5,::5)),getvardims(Optical_Depth_Land_And_Ocean(::5,::5)))
>
> filevarattdef(fout,"AOT",
> Optical_Depth_Land_And_Ocean(::5,::5))
> filevarattdef(fout,"lat", lat2d(::5,::5))
> filevarattdef(fout,"lon", lon2d(::5,::5))
>
> setfileoption(fout,"DefineMode",False)
>
> fout->Optical_Depth_Land_And_Ocean(::5,::5) =
>
> fout->(/Optical_Depth_Land_And_Ocean(::5,::5)/)
>
> fout->lat2d(::5,::5) = (/lat2d(::5,::5)/)
> fout->lon2d(::5,::5) = (/lon2d(::5,::5)/)
>
> end
>
>
> Steven Bliujus
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150527/3521fb5a/attachment-0001.html
More information about the ncl-talk
mailing list