[ncl-talk] read variable with different size dimension in multiple files
Dennis Shea
shea at ucar.edu
Mon Oct 8 09:31:02 MDT 2018
The short answer is "no". There is no automatic method to do this.
However, 'brute force' can be used. It may be good to test on a 'few'
files (eg; ntim=nfil=20)
If really slow; create a netCDF file and use that for subsequent analysis.
Each swath has different latitude/longitude arrays associated with it.
I speculate that these will be needed also.
*The following is not tested. *
varname = "pr_dbz"
tStrt = *get_cpu_time*() ; timing information
.... ; your code to acces file names
nfil = *dimsizes*( fili )
ntim = nfil ; clarity; each 'file' corresponds to
a different 'time'
*NSCAN = 50000 ; arbitrary large number*
NRAY = 80 ; fixed
ntim = 20 ; for testing; override the original
ntim=nfil
; quick turn around ; number is
arbitrary
VAR = *new*((/ntim,NRAY,NSCAN/),"float") ; 'SUPER' variable
;LAT = new((/ntim,NRAY,NSCAN/),"float") ; 'SUPER' variable
;LON = new((/ntim,NRAY,NSCAN/),"float") ; 'SUPER' variable
;TIME = new( ntim, ???, "No_FillValue") ; integer, float, double ;
recommended; create a time variable
nscan_max = -1
do nt=0,ntim-1 ; loop over each file (time)
f = *addfile*( fili(nt), "r")
x = f->*$*varname*$* ; (nray,nscan); integer?; why
(::-1,:) ?
dimx = *dimsizes*(x)
nray = dimx(0)
nscan = dimx(1)
if (nray.ne.NRAY .or. nscan.gt.NSCAN) then
*print*("FATAL: unexpected array size: nray="+nray+":
nscan="+nscan+": "+fili(nt))
exit
end if
nscan_max = *max*((/nscan_max,nscan/)) ; can be used to set NSCAN
VAR(nt,:,0:nscan-1) = *tofloat*( x )
;time: create a 'time' variable ; this is optional but recommended
;time = *cd_calendar*(.....)
;TIME(nt) = time
;LAT(nt,:,0:nscan-1) = (/ f->Latitude /)
;LON(nt,:,0:nscan-1) = (/ f->Longitude /)
end do
VAR!0 = "time"
;;VAR&time = TIME
VAR!1 = "nray"
VAR&nray = *ispan*(1,NRAY,1) ; or * ispan*(0,NRAY-1,1)
VAR!2 = "NSCAN"
VAR&NSCAN = ispan(1,NSCAN,1) ; or ispan(0,NSCAN-1,1)
VAR at long_name = x at long_name
VAR at hdf_name = x at hdf_name
if (isatt(x,"units")) then
VAR at units = x at units
end if
*printVarSummary*(VAR)
*printMinMax*(VAR,0)
print("===")
; same approach for LAT (units: "degrees_north") and LON (units:
degrees_east")
etime = *get_cpu_time*() -tStrt
print("Elapsed time: " + etime+" nfil="+nfil+":
nscan_max="+nscan_max)
print("===")
;; Create a netCDF file using VAR
netCDF = True
if (netCDF) then
dirNc = "./"
filNc = "Super_"+varname+".1Z09.nc" ; whatever you
want
pthNc = dirNc + filNc ;
file path
system("/bin/rm -f "+pthNc) ; remove any pre-existing file
*setfileoption*("nc","Format","NetCDF4Classic") ; variable (likely) >
2GB
ncdf = addfile(pthNc ,"c") ; open output netCDF file
;===================================================================
; create global attributes of the file (optional)
;===================================================================
fAtt = True ; assign file
attributes
fAtt at Title = "TRMM-PF: 1Z09: Super Variable: "+varname
fAtt at Number_of_files = nfil
fAtt at Conventions = "None"
fAtt at Creation_date = *systemfunc* ("date")
fileattdef( ncdf, fAtt ) ; copy file attributes
;===================================================================
; make time an UNLIMITED dimension; recommended for most applications
;===================================================================
*filedimdef*(ncdf,"time",-1,True
;===================================================================
; output variables directly; NCL will call appropriate functions
; to write the meta data associated with each variable
;===================================================================
var_super = varname_super
ncdf->$var_super$ = VAR ; (time,nray,NSCAN)
;;ncdf->Latitude = LAT ; (time,nray,NSCAN)
;;ncdf->Longitude = LON
end if ; netCDF
On Sat, Oct 6, 2018 at 7:48 PM Stefane Fonseca Freitas <
fane.freitas at hotmail.com> wrote:
> Hi users,
>
>
>
> I'm using TRMM precipitation features swath data. In this data, for each
> orbit, the variables size changes (dimensions number remains same) because
> only precipitation pixels are recording in the file. I want read
> reflectivity profile in multiple files (~7000 files) using addfiles and
> join operation. I know that join requires that variables in each file must
> be the same size. So, my question is: is there a way to conform the
> dimension sizes of variables while reading the files? Is it possible to
> read in NCL a variable with different dimension size in each file? Or any
> other suggestion how could I manipulate this variable as one file for
> statistical analysis?
>
>
>
>
> Script:
>
>
> ;***********************************************************************************************************************************************************************************************
>
> begin
>
>
> diri = "/media/stefaneff/Seagate\ Expansion\ Drive/TRMM-PF/199801/"
>
>
> cmd = "find "+ diri + "1Z09.* -type f -print"
> fili = systemfunc(cmd)
>
> nfil = dimsizes( fili )
>
> varname="pr_dbz"
>
>
> f=addfiles(fili, "r")
>
> ListSetType (f, "join")
>
>
> var=int2flt(f[:]->$varname$(::-1,:))
>
> end
>
> ;***********************************************************************************************************************************************************************************************
>
> Error message:
> warning:File /media/stefaneff/Seagate Expansion
> Drive/TRMM-PF/199801/1Z09.19980101.00545.7.HDF dimension sizes do not
> conform to others in list; skipping file
> warning:File /media/stefaneff/Seagate Expansion
> Drive/TRMM-PF/199801/1Z09.19980101.00544.7.HDF dimension sizes do not
> conform to others in list; skipping file
> warning:File /media/stefaneff/Seagate Expansion
> Drive/TRMM-PF/199801/1Z09.19980101.00543.7.HDF dimension sizes do not
> conform to others in list; skipping file
> warning:File /media/stefaneff/Seagate Expansion
> Drive/TRMM-PF/199801/1Z09.19980101.00542.7.HDF dimension sizes do not
> conform to others in list; skipping file
> warning:File /media/stefaneff/Seagate Expansion
> Drive/TRMM-PF/199801/1Z09.19980101.00541.7.HDF dimension sizes do not
> conform to others in list; skipping file
> warning:File /media/stefaneff/Seagate Expansion
> Drive/TRMM-PF/199801/1Z09.19980101.00540.7.HDF dimension sizes do not
> conform to others in list; skipping file
> warning:File /media/stefaneff/Seagate Expansion
> Drive/TRMM-PF/199801/1Z09.19980101.00539.7.HDF dimension sizes do not
> conform to others in list; skipping file
> warning:A valid instance of variable pr_dbz was not found in one or more
> elements of the file list
> fatal:Number of subscripts on rhs do not match number of dimensions of
> aggregated join type variable, (2) subscripts used, (3) subscripts expected
> fatal:["Execute.c":8637]:Execute: Error occurred at or near line 29 in
> file reflectivity_profile.ncl
>
>
> ;***********************************************************************************************************************************************************************************************
>
> PrintVarSummary of pr_dbz in one file: first dimension has always the same
> size (levels), second dimension not (lat,lon)
>
>
> Variable: pr_profile
> Type: integer
> Total Size: 10451520 bytes
> 2612880 values
> Number of Dimensions: 2
> Dimensions and sizes: [fakeDim6 | 80] x [fakeDim7 | 32661]
> Coordinates:
> Number Of Attributes: 2
> long_name : 2A25 REFLECTIVITY profiles (dBZ*100)
> hdf_name : pr_dbz
>
>
>
> Best regards,
>
>
> Stefane Fonseca Freitas
>
> Doutoranda em Meteorologia - PGMET - CPTEC / INPE
> Centro de Previsão de Tempo e Estudos Climáticos / Instituto Nacional de
> Pesquisas Espaciais
> Cachoeira Paulista - SP - Brasil
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20181008/72d1b7c8/attachment.html>
More information about the ncl-talk
mailing list