[ncl-talk] Reading Multiple Files stored in Short Format

[Dennis Shea]

The "addfiles_GetVarShort" function can be modified


Copy and the original script and rename it to (say)


Change:
       function *addfiles_GetVarShor*t (fils[*]:string, varName:string)
To:
       function *addfiles_GetVarShort_Level*s(fils[*]:string, varName:string*,
lev1[1]:numeric, lev2[1]:numeric*)

Change:
  xs    = fs[:]->$varName$            ; with original atts + coord vars
To
   if (lev2.lt.0) then
        xs = fs[:]->$varName$(:*,{lev1}*,:,:)            ; 3D; original
atts + coord vars
    else
        xs = fs[:]->$varName$(:*,{lev1:lev2}*,:,:)    ; 4D;  original atts
+ coord vars
    end if

Change
  X at NCL_tag = "addfiles_GetVarShort"
To
  X at NCL_tag = "addfiles_GetVarShort_Levels"

====
To make the function 'robust' [eg, handle 1D, 2D, 3D, 5D] would require mor
coding and dimension checking/


On Thu, Aug 9, 2018 at 10:10 PM, Bappaditya Nag <bappaditya.nag82 at gmail.com>
wrote:

> Hi Dennis,
>
> Thank you again for the script. That has helped me a lot. I had a
> follow-up question on that. For large datasets, it is easy to read-in only
> the required levels and time period to reduce memory usage. Hence is it
> possible to club the following two lines together
>
> u_all = addfiles_GetVarShort (filesu, "u")
> u = u_all(iStrt:iLast,{lev1:lev2},:,:)
>
> Thank you,
> Bappaditya Nag
> Post-doctoral Research Scholar
> Earth, Ocean and Atmospheric Science
> Florida State University
> Ph. No. : +91-8697458936
>
> On Thu, Aug 2, 2018 at 10:37 AM, Dennis Shea <shea at ucar.edu> wrote:
>
>> [1] *If necessary*, NCL's  *addfiles*
>> <http://www.ncl.ucar.edu/Document/Functions/Built-in/addfiles.shtml>
>> will rebase the 'time'  to the time units on the initial file.
>> [2] *addfiles* will not apply the scale_factor and add_offset. The
>> short2flt
>> <http://www.ncl.ucar.edu/Document/Functions/Contributed/short2flt.shtml>
>> function will use the initial scale_factor and add_offset only.
>>
>> I had an old function (*addfiles_getVarShort*) that does just what Toni
>> suggested. Offline, I sent it  to Bappa. Luckily, it worked  :-)
>>
>> *load* "./addfiles_GetVarShort.ncl"
>>
>>    diri = "./"
>>    fili  =* systemfunc
>> <http://www.ncl.ucar.edu/Document/Functions/Built-in/systemfunc.shtml>*("cd
>> " + diri + ";  ls ERA_Interim_20*.nc")
>>    pthi = diri+fili
>>
>>    print(pthi)
>>    print("=============================================================")
>>
>>    varName = "WhatEverVariable"
>>    x = *addfiles_GetVarShort* (pthi, varName)
>>    printVarSummary(x)
>>    printMinMax(x,0)
>>
>>    print("=============================================================")
>>    ymdh = *cd_calendar
>> <http://www.ncl.ucar.edu/Document/Functions/Built-in/cd_calendar.shtml>(*x&time,
>> -3)
>>    print(x&time+"   "+ymdh)                ; show times
>>
>>
>> On Sun, Jul 29, 2018 at 11:18 PM, Toni Klemm <toni-klemm at tamu.edu> wrote:
>>
>>> Hi Bappaditya,
>>>
>>> Have you tried creating a new, empty array of the same size of all the
>>> files you want to concatenate and then create a loop in which you read in a
>>> single file, convert it to from short to float and then “add” it to your
>>> array?
>>>
>>> I’m not familiar with your data, so my idea might not work for you. Good
>>> luck though!
>>>
>>> Greetings from Texas,
>>> Toni
>>>
>>>
>>> *Toni Klemm**, Ph.D.*
>>> Postdoctoral Research Associate
>>> Department of Ecosystem Science and Management
>>> Texas A&M University, College Station, TX
>>> Contributor at the Early Career Climate Forum
>>> <https://www.eccforum.org/>
>>> www.toni-klemm.de | @toniklemm <https://twitter.com/ToniKlemm>
>>>
>>>
>>> On Jul 30, 2018, at 12:00 AM, Bappaditya Nag <bappaditya.nag82 at gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I have 6 hrly ERA-Interim data stored in chunks of monthly files in
>>> short format (i.e., each file have their own scale and offset factor). When
>>> I read such multiple files in ncl (i.e., using v->f[:]) and convert to
>>> float (i.e., using short2flt), I do not get the correct result since the
>>> scale and offset of the first file is used for all the files.
>>>
>>> Solution : I can unpack and repack the files in a single file, but that
>>> is against the purpose of storing the data in optimal compact format. I am
>>> looking for a solution to read every file, convert to float and
>>> then concatenate the data. Can this be done? Or any other more
>>> effcient solution. Any help is appreciated.
>>>
>>> Present Code is as follows:
>>>
>>> files = systemfunc("ls " + dir + "ERA_Interim/ERA_Interim_v_"+y
>>> ear+"*.nc")
>>> fil = addfiles(files,"r")
>>> ListSetType (fil, "cat")
>>> v6hrly = short2flt(fil[:]->v(iStrt:iLast,{lev1:lev2},:,:))
>>>
>>> Also attached is a figure generated from the code.
>>>
>>> Warm Regards,
>>> Bappaditya Nag
>>> Post-doctoral Scholar
>>> Florida State University
>>>
>>> <xt_Meridional Wind.png>
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk at ucar.edu
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk at ucar.edu
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180810/cd154710/attachment.html>