[ncl-talk] Order of dimensions of an array in netcdf

[Gurer, Kemal@ARB]

Thank you very much Dave and Dennis. I do appreciate the detailed explanations that both of you gave. I wasn’t concerned with the internal storage order of the dimensions, but I wanted to make sure I am writing the data out in the way that it is supposed to be and in the way that I understand. Let alone, necessary reordering of the dimensions that I have to do for my project makes this writing the data out in correct dimensions even more challenging. In the end, if I give a data that is wrongly put together to my colleagues, that means I didn’t do my scientific job properly. So, please excuse me if I ask some very simple questions that everyone else may understand right away. Again, I appreciate both of your detailed responses very much.

With my sincere regards to you both.

Kemal.

From: Dennis Shea <shea at ucar.edu>
Sent: Wednesday, October 21, 2020 8:49 PM
To: Dave Allured - NOAA Affiliate <dave.allured at noaa.gov>
Cc: Gurer, Kemal at ARB <kemal.gurer at arb.ca.gov>; ncl-talk at ucar.edu
Subject: Re: [ncl-talk] Order of dimensions of an array in netcdf

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Additional information to Dave's netCDF overview:

NCL is row major like C. A language like fortran is column major.

On 'paper' the following look opposite however they are arranged exactly the same in memory:

NCL:     TEMP(ntim,klev,nlat,mlon)     <== fastest varying dimension is the rightmost; row major
Fortran: TEMP(mlon,nlat,klev,ntim)     <== fastest varying dimension is the leftmost  ; column major


===========
NCAR's  Climate Analysis Section provides the  Climate Data Guide<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fclimatedataguide.ucar.edu%2F&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cd4f5906be86947cfb26608d8763d708d%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637389353567837763%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=lm8xE7HZUcDN84zdGg8PYYfkQW%2FhIzo32%2FFA%2Fgnuwjc%3D&reserved=0>
You should explore it. There is a lot of information.

Specifically:
https://climatedataguide.ucar.edu/climate-data-tools-and-analysis/common-climate-data-formats-overview<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fclimatedataguide.ucar.edu%2Fclimate-data-tools-and-analysis%2Fcommon-climate-data-formats-overview&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cd4f5906be86947cfb26608d8763d708d%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637389353567837763%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5YOQCYnAorTFFjbp9W0E1%2F3UhHRClIQ6%2Bz27X51uhe8%3D&reserved=0>
https://climatedataguide.ucar.edu/climate-data-tools-and-analysis/netcdf-overview<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fclimatedataguide.ucar.edu%2Fclimate-data-tools-and-analysis%2Fnetcdf-overview&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cd4f5906be86947cfb26608d8763d708d%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637389353567847718%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vz7JrW%2Fk%2BlByVty5AlgMYD5lGNxoYoEtqZZxGPY5G7U%3D&reserved=0>

On Wed, Oct 21, 2020 at 8:25 PM Dave Allured - NOAA Affiliate via ncl-talk <ncl-talk at mailman.ucar.edu<mailto:ncl-talk at mailman.ucar.edu>> wrote:
NCL uses zero-based subscripting for both memory variables and file variables.  Therefore the first element in your example is temp [0] [0] [0] [0].

There are some problems with the Netcdf users guide, which you cited.  You found the Netcdf-4 data model, which is rather complicated.  Here is a simplified summary of the data storage model for the three most common types of Netcdf storage.  For small, simple applications, the user should not be at all concerned with the internal file format.  When performance becomes an issue, it is good to have a general understanding of format issues, so as to help optimize read and write strategies.

Netcdf-3 fixed size variable.  The entire N-dimensional array is stored contiguously in the file, with the NCL rightmost (last) dimension varying the fastest.  The physical layout is generally the same as you would have for a simple array in program memory.  This storage is not expandable.

Netcdf-3 record variable.  The array is stored in chunks across the leftmost (first, slowest varying) dimension, with all the other dimensions stored contiguously in each chunk.  The first dimension is also called the record or unlimited dimension.  This storage can be expanded by appending on the leftmost (slowest) dimension only.  A typical application is a gridded data set where new grids are added over time, as new data become available.

Netcdf-4 chunked variable.  The array is subdivided and stored in equal-sized chunks across all dimensions.  The individual chunk dimension sizes may be arbitrarily chosen when a new file variable is first created.  Inside each chunk, storage is contiguous across dimensions, just like in the other two storage models.  There is hidden infrastructure to keep track of all the chunks.  This strategy has some advantages for arbitrary unlimited (expandable) dimensions, and automatic internal data compression.

There are two simple approaches to optimizing file access.  Either read and write the entire variable in a single statement, or else read and write contiguous file chunks or contiguous subsets of file chunks.  A common example is to set up a gridded file variable so that each 2-D grid is stored contiguously, then read and write a whole grid each time.  This is why you see many examples where lat and lon are the rightmost dimensions; that generally results in contiguous grids.


On Wed, Oct 21, 2020 at 6:10 PM Gurer, Kemal at ARB via ncl-talk <ncl-talk at mailman.ucar.edu<mailto:ncl-talk at mailman.ucar.edu>> wrote:
Hello ncl’ers,

I have been searching a detailed description for the dimensional order of the data points of a multi-dimensional variable to be written
to a netcdf file. I found some locations, but am also equally confused with different explanations or my lack of understanding. One example
for such an explanation for the variable:

temp [TIMES] [LEVELS] [LATS] [LONS];

with

TIMES = 3,
LEVELS = 1,
LATS = 5,
LONS = 10

being written into the netcdf file in the order of dimensions is given at:

https://www.unidata.ucar.edu/software/netcdf/docs/netcdf_data_set_components.html<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.unidata.ucar.edu%2Fsoftware%2Fnetcdf%2Fdocs%2Fnetcdf_data_set_components.html&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cd4f5906be86947cfb26608d8763d708d%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637389353567847718%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=b2zZFCQ%2Bi8LEeJtSRWwQk4x8zCjfA3VC%2BO%2FQNjNAXig%3D&reserved=0>

and mentions “The order in which the data will be returned is with the last dimension, LONS, varying fastest:”
at the time of retrieval of the data.

What I am wondering is when I write the variable “temp” above into the netcdf file, is the first data element of temp:

Temp [2] [1] [4] [9]

or,

temp [0] [1] [0] [0]

Thank you for your help.

Kemal.
_______________________________________________
ncl-talk mailing list
ncl-talk at mailman.ucar.edu<mailto:ncl-talk at mailman.ucar.edu>
List instructions, subscriber options, unsubscribe:
https://mailman.ucar.edu/mailman/listinfo/ncl-talk<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.ucar.edu%2Fmailman%2Flistinfo%2Fncl-talk&data=04%7C01%7Ckemal.gurer%40arb.ca.gov%7Cd4f5906be86947cfb26608d8763d708d%7C9de5aaee778840b1a438c0ccc98c87cc%7C0%7C0%7C637389353567857674%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hJGWYBhi8BD7oK3d%2F0tlsMzlxq2eWy%2BwYEZ2Ry5SWRI%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20201022/8a223813/attachment.html>