[ncl-talk] netcdf file creation and order of dimensions
Gurer, Kemal@ARB
kemal.gurer at arb.ca.gov
Fri Oct 23 09:27:33 MDT 2020
Hello Dennis,
Thank you for the wonderful guide and help all in all to solve my problem. You are amazing! God bless you.
I agree with you that I should use ncl syntax. Thank you.
Kemal.
From: Dennis Shea <shea at ucar.edu>
Sent: Thursday, October 22, 2020 8:38 PM
To: Gurer, Kemal at ARB <kemal.gurer at arb.ca.gov>
Cc: ncl-talk at ucar.edu
Subject: Re: [ncl-talk] netcdf file creation and order of dimensions
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
You csn use functions/procedures but I prefer using NCL syntax:
See: https://www.ncl.ucar.edu/Applications/method_1.shtml<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncl.ucar.edu%2FApplications%2Fmethod_1.shtml&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cfd0f32fff188434c0e5708d8770513e0%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637390211018890535%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Fumx2Di7g2V0scEQSTi3qkuQPCJymhxRb3HVlfbfzEA%3D&reserved=0>
zmid(latitude,longitude,levz_mid)
========
work = zmid(levz_mid|:,latitude|:,longitude|:) ; reorder via NCL syntax: sizes: (klev,nlat,mlon)
printVarSummary(work)
print("=====")
ntim = 1 ; 0 1 2 3
WORK = conform_dims<https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ncl.ucar.edu%2FDocument%2FFunctions%2FBuilt-in%2Fconform_dims.shtml&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cfd0f32fff188434c0e5708d8770513e0%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637390211018890535%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yUZSomE45tysTe8G4gWTplWWkEfiwca97L8CGg3tpis%3D&reserved=0>((/ntim,klev,nlat,mlon/), work, (/1,2,3/))
printVarSummary(WORK)
print("=====")
copy_VarMeta(x, WORK(0,:,:,:)) ; coordinates and attributes
WORK!0 = "time" ; name dimension
;; WORK&time = (/...../) ; add time coordinate
printVarSummary(WORK)
print("=====")
setfileoption<https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ncl.ucar.edu%2FDocument%2FFunctions%2FBuilt-in%2Fsetfileoption.shtml&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cfd0f32fff188434c0e5708d8770513e0%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637390211018900499%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=RmcczTdNTT8N5ldCjvd%2Fmq0n7QUFS1cDS%2Fa3JcinNjY%3D&reserved=0>("nc","Format","NetCDF4Classic")
diro = "./"
filo = "SampleFile_KemalGurer.nc"
ptho = doro+filo
fo = addfile<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncl.ucar.edu%2FDocument%2FFunctions%2FBuilt-in%2Faddfile.shtml&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cfd0f32fff188434c0e5708d8770513e0%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637390211018900499%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5iOSVh80lc3By61vc9zhVv0qI%2BiO3C%2FLsIFR2ftiFGg%3D&reserved=0>(ptho,"c")
;===================================================================
; create global attributes of the file (optional)
;===================================================================
fAtt = True ; assign file attributes
fAtt at title = "NCL netCDF Creation: Kemal Gurer"
fAtt at source_file = "original-file.nc<https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Foriginal-file.nc%2F&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cfd0f32fff188434c0e5708d8770513e0%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637390211018910425%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=PiPXCZDt1Kg9hcPc8q412O%2BPQPZeqlA5lF3YtljjnFQ%3D&reserved=0>"
fAtt at Conventions = "None"
fAtt at creation_date = systemfunc<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncl.ucar.edu%2FDocument%2FFunctions%2FBuilt-in%2Fsystemfunc.shtml&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cfd0f32fff188434c0e5708d8770513e0%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637390211018910425%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ReKQ3RCn5i1UHKqLv1P1GDgEABzdb2w2W26tKmyorWg%3D&reserved=0> ("date")
fileattdef<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncl.ucar.edu%2FDocument%2FFunctions%2FBuilt-in%2Ffileattdef.shtml&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cfd0f32fff188434c0e5708d8770513e0%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637390211018920403%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=W8MBM%2FN3VHaXMciobkngnbZh5Uf6%2BcVNYdMbweBkJoE%3D&reserved=0>( fo, fAtt ) ; copy file attributes
;===================================================================
; make time an UNLIMITED dimension; recommended for most applications
;===================================================================
filedimdef<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncl.ucar.edu%2FDocument%2FFunctions%2FBuilt-in%2Ffiledimdef.shtml&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cfd0f32fff188434c0e5708d8770513e0%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637390211018920403%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qA7mVMgMoeOh0oW18Flz6OoSMGvlAj88JVoAUsBlUcs%3D&reserved=0>(fo,"time",-1,True)
;===================================================================
; output variables directly; NCL will call appropriate functions
; to write the meta data associated with each variable
;===================================================================
fo->ZMID = WORK ; 4D
==========
After file is created, examine file contents
%> ncdump -h SampleFile_KemalGurer.nc
or
%> ncl_filedump SampleFile_KemalGurer.nc
On Thu, Oct 22, 2020 at 5:28 PM Gurer, Kemal at ARB via ncl-talk <ncl-talk at mailman.ucar.edu<mailto:ncl-talk at mailman.ucar.edu>> wrote:
Hello,
I am writing some variables into a netcdf file, but for my problem, let me show one variable here, zmid, with the netcdf name ZH, using the following netcdf commands:
filevardef(f,"ZH","float",(/"TSTEP","LAY","ROW","COL"/))
ZH_atts = 0.0
ZH_atts at long_name = "ZH"
ZH_atts at var_desc = "mid-layer height above ground"
ZH_atts at units = "M"
ZH_atts at _FillValue = 1e32
filevarattdef(f,"ZH",ZH_atts)
ZH = new((/nlev, nlat, nlon/), "float", ZH_atts at _FillValue)
do n = 0, ntim-1
f->ZH(n,:,:,:) = (/zmid/)
end do
Here, TSTEP is Time, LAY is levels, ROW is latitude, and COL is longitude with the exact dimensions of the dimensions given in the original data.
printVarSummary(zmid) gives the following information:
-----------------------------------------------------------------------------------
Variable: zmid
Type: float
Total Size: 2328480 bytes
582120 values
Number of Dimensions: 3
Dimensions and sizes: [latitude | 98] x [longitude | 108] x [levz_mid | 55]
Coordinates:
latitude: [31.6178..42.50189]
longitude: [-129.0292..-113.0208]
Number Of Attributes: 2
units : m MSL
long_name : Geometric height of mid-levels
(0)
(0) Geometric height of mid-levels (m MSL) : min=-64.3161 max=29608.8
-----------------------------------------------------------------------------------
Since I need to write the variable “zmid” as well as other variables into the netcdf file with the following order of dimensions:
("Time","nVertLevels","latitude","longitude") with the dimension names
I reorder “zmid” via:
zmid_reorder = zmid(levz_mid|:,latitude|:,longitude|:)
printVarSummary(zmid_reorder),
which gives the following output:
-----------------------------------------------------------------------------------
Variable: zmid_reorder
Type: float
Total Size: 2328480 bytes
582120 values
Number of Dimensions: 3
Dimensions and sizes: [levz_mid | 55] x [latitude | 98] x [longitude | 108]
Coordinates:
latitude: [31.6178..42.50189]
longitude: [-129.0292..-113.0208]
(0) min=-64.3161 max=29608.8
-----------------------------------------------------------------------------------
However, the reordered zmid output in netcdf file is wrong. The output of zmid without reordering of dimensions is:
float ZH(Time, LAY, ROW, COL) ;
ZH:_FillValue = 1.e+32f ;
string ZH:units = "M" ;
string ZH:var_desc = "mid-layer height above ground" ;
string ZH:long_name = "ZH" ;
data:
ZH =
9.999999, 32, 58.4, 90.09999, 128.1, 173.7, 228.45, 294.15, 373, 467.6,
581.1, 717.3, 880.75, 1076.9, 1312.3, 1594.75, 1933.7, 2340.45, 2828.55,
3414.3, 4108.8, 4858.8, 5608.799, 6358.8, 7108.8, 7858.8, 8608.8, 9358.8,
10108.8, 10858.8, 11608.8, 12358.8, 13108.8, 13858.8, 14608.8, 15358.8,
16108.8, 16858.8, 17608.8, 18358.8, 19108.8, 19858.8, 20608.8, 21358.8,
22108.8, 22858.8, 23608.8, 24358.8, 25108.8, 25858.8, 26608.8, 27358.8,
28108.8, 28858.8, 29608.8, 10, 32, 58.4, 90.10001, 128.1, 173.7, 228.45,
294.15, 373, 467.6, 581.1, 717.3, 880.75, 1076.9, 1312.3, 1594.75,
1933.7, 2340.45, 2828.55, 3414.3, 4108.8, 4858.8, 5608.8, 6358.8, 7108.8,
7858.8, 8608.8, 9358.8, 10108.8, 10858.8, 11608.8, 12358.8, 13108.8,
13858.8, 14608.8, 15358.8, 16108.8, 16858.8, 17608.8, 18358.8, 19108.8,
19858.8, 20608.8, 21358.8, 22108.8, 22858.8, 23608.8, 24358.8, 25108.8,
25858.8, 26608.8, 27358.8, 28108.8,
28858.8, 29608.8,
----------------------------------------------------------
The netcdf output of zmid_reorder with netcdf variable name ZH1 as test is:
float ZH1(TSTEP, LAY, ROW, COL) ;
ZH1:_FillValue = 1.e+32f ;
string ZH1:units = "M" ;
string ZH1:var_desc = "mid-layer height above ground" ;
string ZH1:long_name = "ZH1" ;
data:
ZH1 =
9.999999, 10, 10, 10, 10, 10, 10, 10, 10, 10, 9.999999, 10, 10, 10, 10, 10,
10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,
10, 10, 10, 9.999999, 10, 10, 10, 9.999999, 10, 10, 10, 10, 10, 10, 10,
10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,
10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 423.6477,
378.6578, 662.8945, 1097.154, 1152.52, 1072.004, 1299.672, 1005.835,
592.6567, 270.4732, 318.5765, 13.87211, 13.37675, 10, 10, 49.30841,
48.51982, 38.2547, 10.43509, 20.66064, 89.90919, 139.163, 114.4501,
201.582, 276.1719,
10, 10, 10, 10, 10, 10, 10,
---------------------------------------------------------------
As another example of for the impact of variable reordering on the output, here is the information about code to read potential temperature and reordering it as well as its results:
tk1 = a->theta
; copy_VarMeta(a->theta,tk1)
; tk1 at units = ""
printVarSummary(tk1)
printMinMax(tk1,0)
tk = tk1(Time|:,nVertLevels|:,latitude|:,longitude|:)
printVarSummary(tk)
printMinMax(tk,0)
--------------------------------------------------------------------
Variable: tk1
Type: float
Total Size: 55883520 bytes
13970880 values
Number of Dimensions: 4
Dimensions and sizes: [Time | 24] x [latitude | 98] x [longitude | 108] x [nVertLevels | 55]
Coordinates:
latitude: [31.6178..42.50189]
longitude: [-129.0292..-113.0208]
Number Of Attributes: 2
units : K
long_name : Potential temperature
(0) Potential temperature (K) : min=282.766 max=797.725
data:
TK1 =
290.8131, 290.7571, 290.7412, 290.7352, 290.734, 290.7361, 290.7417,
290.7557, 290.9228, 293.661, 296.6131, 298.7729, 301.4621, 303.3734,
305.5786, 307.7134, 309.639, 311.1685, 312.9564, 315.1082, 317.7267,
320.5912, 323.4933, 326.3149, 329.7065, 333.1702, 337.303, 339.9579,
341.5059, 344.3701, 347.8319, 354.1167, 358.0386, 360.7633, 364.5301,
372.0264, 381.8416, 392.6732, 410.2569, 427.9963, 446.3387, 469.1274,
492.7356, 513.7944, 534.12, 553.8484, 575.1737, 600.4743, 628.2712,
658.1637, 681.6721, 702.9726, 730.4858, 756.6381, 787.6335, 290.8405,
290.7827, 290.7664, 290.7603, 290.7591, 290.7614, 290.7675, 290.7849,
290.9932, 293.7723, 296.4947, 298.9371, 301.5804, 303.4898, 305.7006,
307.7661, 309.706, 311.2367, 313.1123, 315.1071, 317.8082, 320.5843,
323.5288, 326.2887, 329.711, 333.046, 337.2808, 339.8705, 341.3407,
344.2988, 348.018, 354.2007, 357.992, 360.691, 364.6261, 371.9153,
381.6589, 392.6092, 410.5144, 428.3927, 446.4816, 469.0464, 491.7896,
513.2549, 534.1764, 555.1635, 576.1888, 599.7042, 626.2191, 656.9671,
681.5657, 704.613, 733.2448,
-----------------------------------------------------------------------------------
Variable: tk
Type: float
Total Size: 55883520 bytes
13970880 values
Number of Dimensions: 4
Dimensions and sizes: [Time | 24] x [nVertLevels | 55] x [latitude | 98] x [longitude | 108]
Coordinates:
latitude: [31.6178..42.50189]
longitude: [-129.0292..-113.0208]
Number Of Attributes: 2
long_name : Reordered potential temperature
units : K
(0) Potential temperature (K) : min=282.766 max=797.725
data:
TK =
290.8131, 290.8405, 290.8532, 290.874, 290.8821, 290.8575, 290.9009,
290.9425, 290.9517, 290.9512, 290.9521, 290.9454, 290.9456, 290.9441,
290.9357, 290.9396, 290.9547, 290.9457, 290.9522, 290.9639, 290.9545,
290.9518, 290.9407, 290.9241, 290.9189, 290.9194, 290.8954, 290.8531,
290.8095, 290.7653, 290.6684, 290.5908, 290.5103, 290.4197, 290.3355,
290.2524, 290.0889, 289.9354, 289.8426, 289.769, 289.7196, 289.5693,
289.5384, 289.5408, 289.3453, 289.1881, 289.1, 289.0808, 289.054,
289.0443, 289.0854, 288.9913, 289.123, 289.17, 289.3506, 289.4048,
289.4511, 289.4771, 289.5936, 289.642, 289.163, 289.2242, 289.2056,
289.2532, 289.3674, 289.5267, 289.6508, 289.8015, 289.9764, 290.2227,
290.4557, 290.7174, 290.9393, 291.1483, 291.4482, 291.6609, 291.8963,
292.1499, 292.4067, 292.6108, 292.777, 292.8965, 293.0784, 299.2768,
299.7407, 305.4052, 313.3188, 313.5712, 314.6627, 313.7572, 313.2654,
312.6786, 311.9484, 311.588, 311.4529, 308.0678, 307.5142, 307.9211,
308.1941, 309.1768, 309.8638, 310.4225, 309.4877, 310.032, 311.1737,
311.9905, 312.4248, 311.6823,
---------------------------------------------------------------------------------------
I use the following netcdf file definitions:
theflnm = "./nc4_write.nc<https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fnc4_write.nc%2F&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cfd0f32fff188434c0e5708d8770513e0%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637390211018920403%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=h4Hqt536TRhPqJvdeNfZIthg4LI5LPAfGRtpk9MUKu0%3D&reserved=0>"
setfileoption("nc","format","netcdf4")
setfileoption("nc","headerReserveSpace",64000)
setfileoption("nc","preFill",False)
setfileoption("nc","defineMode",True)
I also tried to use the following chunking definitions, but commented them out because of the errors that I get and let the internal default definitions take over:
;mtim = 1
;mlev = nlev/10
;mlat = nlat/10
;mlon = nlon/10
;chunk_sizes = (/mtim,mlev,mlat,mlon/)
;filechunkdimdef(f,dim_names,chunk_sizes,dim_unlimited)
;filevarchunkdef(f, "TA", chunk_sizes)
;filevarcompressleveldef(f, "TA", 2)
Is my definition of netcdf file and/or variables in netcdf definitions wrong? Are my definitions for the order of dimensions using reorder and the order of dimensions in netcdf declarations not supposed to be the same? Just as a summary, original variable, reordered variable and dimension declarations in netcdf attributes are (in order):
Potential Temperature - [Time | 24] x [latitude | 98] x [longitude | 108] x [nVertLevels | 55]
Reordered Potential Temperature - [Time | 24] x [nVertLevels | 55] x [latitude | 98] x [longitude | 108]
float TK1(TSTEP, LAY, ROW, COL) ;
TK1:_FillValue = 1.e+32f ;
string TK1:units = "K" ;
string TK1:var_desc = "potential temperature" ;
string TK1:long_name = "TK1" ;
float TK(TSTEP, LAY, ROW, COL) ;
TK:_FillValue = 1.e+32f ;
string TK:units = "K" ;
string TK:var_desc = "reordered potential temperature" ;
Thank you for your help.
Kemal.
_______________________________________________
ncl-talk mailing list
ncl-talk at mailman.ucar.edu<mailto:ncl-talk at mailman.ucar.edu>
List instructions, subscriber options, unsubscribe:
https://mailman.ucar.edu/mailman/listinfo/ncl-talk<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.ucar.edu%2Fmailman%2Flistinfo%2Fncl-talk&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cfd0f32fff188434c0e5708d8770513e0%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637390211018930337%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FmzA2BsMAZA1Ih7abhIHBdXlcWRMwnSy0uWssLI%2BqoI%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20201023/beae033e/attachment.html>
More information about the ncl-talk
mailing list