[ncl-talk] Problems converting hdf to netcdf using NCL
Dennis Shea
shea at ucar.edu
Tue May 26 16:43:04 MDT 2015
I just looked at the 5 sample files you sent ....
The dimension sizes are *not* the same size across all files
The 1st 4files are:
MOD04_L2.A2014305.0000.006.2015077213911.pscs_000500937810.hdf
dimensions:
Cell_Along_Swath = 203 <=====
Cell_Across_Swath = 135
The last file is:
MOD04_L2.A2014305.0500.006.2015077215106.pscs_000500937810
dimensions:
Cell_Along_Swath = 204 <====
Cell_Across_Swath= 135
That can be 'programmed around'.
See attached
On Tue, May 26, 2015 at 6:34 AM, BLIUJUS, STEVEN D CTR USAF AFMC
AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:
> Dennis,
>
> #2 and #3: Cell_Along_Swath, and Cell_Across_Swath are exactly the same
> for every file as far as actual dimension size, the variables are not
> though.
> #4: I am trying to create a mosaic of all the files and that is why I am
> wanting to generate one NetCDF file. So when I run my script the image
> above is what comes out. This image is what I want to create in a NetCDF
> file (Globe.PNG).
> #5: I did this because I was following a script online that I found where
> I needed to give the main variable(Optical Depth) the latitude/longitude
> dimensions. It used (::5) which what I just followed. When I try plotting
> using just (:,:), the plot take an extremely long time to plot(though there
> is much more data prevalent)...If it is one file it doesn’t take long, but
> when you add them all in the mosaic form, it takes a much more substantial
> amount of time to generate the image.
> #6: I'm not sure exactly what you mean by this as there is not time field
> in the file.
> #7: See 5 attached MODIS files
> #8: I am trying to create a NetCDF file for a coworker to use for his
> verification software. He only reads in NetCDF files or text files (though
> he would prefer NetCDF files)
>
> I also attached the script (with all previous commented lines removed)
> that plots the files and also creates a text table of the data.
>
> Any help would be appreciated.
>
> Steve
>
>
>
> -----Original Message-----
> From: Dennis Shea [mailto:shea at ucar.edu]
> Sent: Saturday, May 23, 2015 2:30 PM
> To: BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL
> Cc: ncl-talk at ucar.edu
> Subject: Re: [ncl-talk] Problems converting hdf to netcdf using NCL
>
> I have looked at this and there are *lots* of issues.
>
> "... taking several hdf files (200-400) and trying to generate one NetCDF
> file."
>
>
> So ... **Conceptually**, you want
>
>
> AOT(time, Cell_Along_Swath, Cell_Across_Swath)
>
>
> Thing is your geographic coordinates are 2-dimensional. see below
>
>
>
>
> [1]
>
> The files contain swath data. From your dump:
>
> path: MOD04_L2.A2014306.2330.006.2015077230656.pscs_000500937810.hdf
> file global attributes:
> HDFEOSVersion : HDFEOS_V2.17
> StructMetadata_0 : GROUP=SwathStructure
> GROUP=SWATH_1
> SwathName="mod04"
>
>
> [2] Are the dimension sizes
>
> (Cell_Along_Swath,Cell_Across_Swath) => (203,135)
>
>
> exactly the same for *all* files?
>
>
> [3] If [2] is satisfied .... Are the *values* contained within the
> following variables
>
> Latitude ( Cell_Along_Swath, Cell_Across_Swath)
>
> Longitude ( Cell_Along_Swath, Cell_Across_Swath )
>
>
> *exactly* the same for *all* files? ie: homogeneous space!
>
>
> My (limited) experience with satellite swath data is that the answer
> is "no"
>
>
> [4] If [2] *and* [3] are not satisfied, I see no advantage of creating a
> single netCDF file.
>
> You would end up with 200-400 sets of dimension sizes *and* 200-400
> variables
>
> associated 200-400two-dimensional spatial variables.
>
>
> [5] I have no idea why you are sub-sampling [decimating] the array by 5:
> (::5,::5) the resolution from (203,135) to ??(41,27)??. This seems
> drastic!
>
> Sub-sampling (array decimation) is commonly used used when the
> resolution is (say) (2500,4000) and the data are reasonably smooth.
>
>
>
> [6] In any case you should be using
>
> ListSetType(a,"join")
>
>
> to add a record dimension ... which you should fill in with 'time'
>
> (parse file name) or some sequential number.
>
>
> Please read
>
> https://www.ncl.ucar.edu/Document/Functions/Built-in/addfiles.shtml
>
> Example 3 illustrates what "join" will do.
>
>
>
> [7] I do not have your ?MODIS? data file. Based upon your dump, I created
> a script that creates a netCDF file for each HDF file and (I hope) a sample
> graphic.
>
>
> [8] I am not sure of ultimate use of the data but from an NCL perspective,
> there is no reason to convert to netCDF.
>
>
> ===========================
>
>
> Please upgrade to 6.3.0
>
> http://www.ncl.ucar.edu/Download/
>
> ============================
>
>
>
>
>
> On Fri, May 22, 2015 at 9:59 AM, BLIUJUS, STEVEN D CTR USAF AFMC
> AFLCMC/HBAW-OL <steven.bliujus.3.ctr at us.af.mil> wrote:
>
>
> I wanted to provide an example of what is in the hdf file. See
> attached.
>
> Steve
>
>
> -----Original Message-----
> From: ncl-talk-bounces at ucar.edu [mailto:ncl-talk-bounces at ucar.edu]
> On Behalf Of BLIUJUS, STEVEN D CTR USAF AFMC AFLCMC/HBAW-OL
> Sent: Friday, May 22, 2015 10:36 AM
> To: ncl-talk at ucar.edu
> Subject: [ncl-talk] Problems converting hdf to netcdf using NCL
>
> I am taking several hdf files (200-400) and trying to generate one
> NetCDF file. I am able to plot it on a grid, but when I try to write to a
> NetCDF file, I get the following error:
>
> fatal:Number of dimensions in parameter (2) of (filedimdef) is
> (2), (1) dimensions were expected
> fatal:["Execute.c":8128]:Execute: Error occurred at or near line
> 82 in file File.ncl
>
>
> The data I am creating is AOD with two dimensions (lat/lon). I am
> uncertain as to why it is expecting one dimension. Below is my script. FYI,
> the original files had to be manipulated in order to make AOD have the
> lat/lon dimensions. Any help would be greatly appreciated.
>
>
> load "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load "/home/bliujuss/ncl/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl"
> load "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> load "/home/bliujuss/ncl/lib/ncarg/nclscripts/csm/shea_util.ncl"
> begin
>
> diri = "/home/bliujuss/Verification_data/Test/"
> fili = systemfunc("ls "+diri+"*.hdf")
> a = addfiles(fili,"r")
>
> ;ListSetType(a,"cat")
>
> var1=a[:]->Optical_Depth_Land_And_Ocean(:,:)
> Optical_Depth_Land_And_Ocean = short2flt(var1)
>
> var2=a[:]->Longitude(:,:)
> Longitude=var2
>
>
> var3 = a[:]->Latitude(:,:)
> Latitude=var3
>
> dims = dimsizes(Optical_Depth_Land_And_Ocean)
> nrows = dims(0) ;203
> npixels = dims(1) ;135
>
> lat2d = new((/nrows,npixels/),float)
> lon2d = new((/nrows,npixels/),float)
>
> do i=0,npixels-1
> lat2d(:,i) = Latitude(:,i)
> end do
>
> do i=0,nrows-1
> lon2d(i,:) = Longitude(i,:)
> end do
>
> Optical_Depth_Land_And_Ocean at lat2d = lat2d(::5,::5)
> Optical_Depth_Land_And_Ocean at lon2d = lon2d(::5,::5)
>
>
> wks = gsn_open_wks("x11","plot_aod")
> gsn_define_colormap(wks,"NCV_banded")
> setvalues NhlGetWorkspaceObjectId()
> "wsMaximumSize": 300000000
> end setvalues
>
> res = True ; plot mods desired
> res at cnLinesOn = False ; turn off contour
> lines
> res at cnFillOn = True ; color plot desired
> res at cnLineLabelsOn = False ; turn off contour
> lines
> res at gsnAddCyclic = False ; non-global data
> res at gsnSpreadColors = True ; use full range of
> colormap
> ;res at cnFillMode = "RasterFill" ; turn on raster mode
> res at lbLabelAutoStride = True ; nice spacing of
> lables
> res at gsnMaximize = True ; blow up plot as much
> as poss
> ;print(min(lat2d))
> ;print(max(lat2d))
> ;print(min(lon2d))
> ;print(max(lon2d))
>
> plot =
> gsn_csm_contour_map_ce(wks,Optical_Depth_Land_And_Ocean(::5,::5),res)
> print(max(Optical_Depth_Land_And_Ocean(::5,::5)))
>
> nlat = dimsizes(lat2d(::5,::5))
> nlon = dimsizes(lon2d(::5,::5))
>
> diro = "/home/bliujuss/Plots/"
> filo = "example.nc"
> system("rm -f" + diro + filo)
> fout = addfile (diro + filo, "c")
>
> setfileoption(fout,"DefineMode",True)
>
> dim_names = (/"lat", "lon"/)
> dim_sizes = (/nlat, nlon/)
> dimUnlim = (/False, False/)
> filedimdef(fout,dim_names,dim_sizes,dimUnlim)
>
> filevardef(fout, "lat",
> typeof(lat2d(::5,::5)),getvardims(lat2d(::5,::5)))
> filevardef(fout, "lon",
> typeof(lon2d(::5,::5)),getvardims(lon2d(::5,::5)))
> filevardef(fout, "AOT",
> typeof(Optical_Depth_Land_And_Ocean(::5,::5)),getvardims(Optical_Depth_Land_And_Ocean(::5,::5)))
>
> filevarattdef(fout,"AOT", Optical_Depth_Land_And_Ocean(::5,::5))
> filevarattdef(fout,"lat", lat2d(::5,::5))
> filevarattdef(fout,"lon", lon2d(::5,::5))
>
> setfileoption(fout,"DefineMode",False)
>
> fout->Optical_Depth_Land_And_Ocean(::5,::5) =
>
> fout->(/Optical_Depth_Land_And_Ocean(::5,::5)/)
>
> fout->lat2d(::5,::5) = (/lat2d(::5,::5)/)
> fout->lon2d(::5,::5) = (/lon2d(::5,::5)/)
>
> end
>
>
> Steven Bliujus
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150526/7c590edd/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bliujus_mosaic.ncl
Type: application/octet-stream
Size: 3905 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150526/7c590edd/attachment-0002.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bliujus_plot.ncl
Type: application/octet-stream
Size: 1932 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150526/7c590edd/attachment-0003.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bliujus_1.png
Type: image/png
Size: 85174 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150526/7c590edd/attachment-0001.png
More information about the ncl-talk
mailing list