[ncl-talk] Reading Multiple Files stored in Short Format

[Bappaditya Nag]

Hi Dennis,

Thank you again for the script. That has helped me a lot. I had a follow-up
question on that. For large datasets, it is easy to read-in only the
required levels and time period to reduce memory usage. Hence is it
possible to club the following two lines together

u_all = addfiles_GetVarShort (filesu, "u")
u = u_all(iStrt:iLast,{lev1:lev2},:,:)

Thank you,
Bappaditya Nag
Post-doctoral Research Scholar
Earth, Ocean and Atmospheric Science
Florida State University
Ph. No. : +91-8697458936

On Thu, Aug 2, 2018 at 10:37 AM, Dennis Shea <shea at ucar.edu> wrote:

> [1] *If necessary*, NCL's  *addfiles*
> <http://www.ncl.ucar.edu/Document/Functions/Built-in/addfiles.shtml>
> will rebase the 'time'  to the time units on the initial file.
> [2] *addfiles* will not apply the scale_factor and add_offset. The
> short2flt
> <http://www.ncl.ucar.edu/Document/Functions/Contributed/short2flt.shtml>
> function will use the initial scale_factor and add_offset only.
>
> I had an old function (*addfiles_getVarShort*) that does just what Toni
> suggested. Offline, I sent it  to Bappa. Luckily, it worked  :-)
>
> *load* "./addfiles_GetVarShort.ncl"
>
>    diri = "./"
>    fili  =* systemfunc
> <http://www.ncl.ucar.edu/Document/Functions/Built-in/systemfunc.shtml>*("cd
> " + diri + ";  ls ERA_Interim_20*.nc")
>    pthi = diri+fili
>
>    print(pthi)
>    print("=============================================================")
>
>    varName = "WhatEverVariable"
>    x = *addfiles_GetVarShort* (pthi, varName)
>    printVarSummary(x)
>    printMinMax(x,0)
>
>    print("=============================================================")
>    ymdh = *cd_calendar
> <http://www.ncl.ucar.edu/Document/Functions/Built-in/cd_calendar.shtml>(*x&time,
> -3)
>    print(x&time+"   "+ymdh)                ; show times
>
>
> On Sun, Jul 29, 2018 at 11:18 PM, Toni Klemm <toni-klemm at tamu.edu> wrote:
>
>> Hi Bappaditya,
>>
>> Have you tried creating a new, empty array of the same size of all the
>> files you want to concatenate and then create a loop in which you read in a
>> single file, convert it to from short to float and then “add” it to your
>> array?
>>
>> I’m not familiar with your data, so my idea might not work for you. Good
>> luck though!
>>
>> Greetings from Texas,
>> Toni
>>
>>
>> *Toni Klemm**, Ph.D.*
>> Postdoctoral Research Associate
>> Department of Ecosystem Science and Management
>> Texas A&M University, College Station, TX
>> Contributor at the Early Career Climate Forum <https://www.eccforum.org/>
>> www.toni-klemm.de | @toniklemm <https://twitter.com/ToniKlemm>
>>
>>
>> On Jul 30, 2018, at 12:00 AM, Bappaditya Nag <bappaditya.nag82 at gmail.com>
>> wrote:
>>
>> Hi,
>>
>> I have 6 hrly ERA-Interim data stored in chunks of monthly files in short
>> format (i.e., each file have their own scale and offset factor). When I
>> read such multiple files in ncl (i.e., using v->f[:]) and convert to float
>> (i.e., using short2flt), I do not get the correct result since the scale
>> and offset of the first file is used for all the files.
>>
>> Solution : I can unpack and repack the files in a single file, but that
>> is against the purpose of storing the data in optimal compact format. I am
>> looking for a solution to read every file, convert to float and
>> then concatenate the data. Can this be done? Or any other more
>> effcient solution. Any help is appreciated.
>>
>> Present Code is as follows:
>>
>> files = systemfunc("ls " + dir + "ERA_Interim/ERA_Interim_v_"+y
>> ear+"*.nc")
>> fil = addfiles(files,"r")
>> ListSetType (fil, "cat")
>> v6hrly = short2flt(fil[:]->v(iStrt:iLast,{lev1:lev2},:,:))
>>
>> Also attached is a figure generated from the code.
>>
>> Warm Regards,
>> Bappaditya Nag
>> Post-doctoral Scholar
>> Florida State University
>>
>> <xt_Meridional Wind.png>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180810/1ff44a34/attachment.html>