[ncl-talk] Subsetting problem across files when using addfiles and ListSetType("cat")

[Mary Haley]

Elinor,

Thanks for providing the files.  As I stated offline (and am restating here
for the benefit of ncl-talk), the issue is that the two time arrays have
different units.

Addfiles is not handling this gracefully.  I believe it looks at the time
units of the first file, and assumes that the time units are the same on
subsequent files. It shouldn't seg fault, so I will a create a ticket on
this.

Meanwhile, I wrote a script to "fix" the time array on one of the files to
match the time array on the other file, so that addfiles will work
properly. Please see the attached fix_time.ncl script. One you run that
script, you can then use read_fixed_files_and_plot.ncl to read and plot the
subsetted data.

This process uncovered a bug in cd_convert (which is why I didn't use this
function to do the conversion) which doesn't recognize the "calendar"
option, so I will make sure that gets fixed as well.

Let me know if you have any questions or issues with the scripts.

--Mary

On Mon, Aug 29, 2016 at 2:47 PM, Martin, Elinor R. <elinor.martin at ou.edu>
wrote:

> I have put the two sample files in the ftp account. The names are:
>
> ta_6hrLev_GISS-E2-H_historical_r6i1p1_195001010600-195007010000.nc
>
> ta_6hrLev_GISS-E2-H_historical_r6i1p1_195007010600-195101010000.nc
>
> They are large files (1.5GB each), which may be the cause of the problem.
> If there is a straightforward way to break them up into time chunks then I
> can do that in the meantime.
>
> Elinor
>
>
> *******************************************
> Dr. Elinor Martin
> Assistant Professor
> School of Meteorology
> University of Oklahoma
> 120 David L. Boren Blvd. Suite 5900
> Norman, OK 73072-7307
>
> Office: NWC 5642
> Email: elinor.martin at ou.edu
> Phone: 405 325 7392
> Web: http://weather.ou.edu/~ermartin
> *******************************************
>
>
>
>
>
>
>
> On Aug 29, 2016, at 3:32 PM, Mary Haley <haley at ucar.edu> wrote:
>
> Hi Elinor,
>
> I think we'll need to see your files to debug this one.  Are the files
> really large?  If not, you can upload them to our ftp account:
>
> ftp ftp.cgd.ucar.edu
> anonymous
> <use your email address for the password>
> cd incoming
> put temp_195001010600-195007010000.nc
> put temp_195007010600-195107010000.nc
> .
> .
> .
> quit
>
> Otherwise, if you can put the files somewhere that I can download them,
> that's okay too.
>
> The seg fault could either be coming from a memory issue (if the files are
> large), or NCL could just be struggling with organizing the time array.
>
> Thanks,
>
> --Mary
>
>
> On Mon, Aug 29, 2016 at 1:44 PM, Martin, Elinor R. <elinor.martin at ou.edu>
> wrote:
>
>> I have a large set of files (time,level,lat,lon and 6 hourly) that span
>> multiple years, but they are not divided nicely into months/years. I only
>> want to read in one month at a time (and only 00Z), which in some cases,
>> spans only one file, but in other cases spans two.
>>
>> My code works perfectly for the months that come from only one file, but
>> I get a “Segmentation fault (core dumped)” when the month spans two files.
>> See brief code example below (happy to send more and actual files if
>> needed).
>>
>> example files:
>> temp_195001010600-195007010000.nc
>> temp_195007010600-195107010000.nc
>>
>> example code snipet:
>>
>> …….
>> all_files = systemfunc ("ls “+in_path+"temp_195*.nc")
>> f            = addfiles (all_files, "r")
>>
>> ListSetType(f,"cat")
>> time            =       f[:]->time
>>
>> utc_date = cd_calendar(time, 0)
>> yr = tointeger(utc_date(:,0))
>> mm = tointeger(utc_date(:,1))
>> hour =     tointeger(utc_date(:,3))
>>
>> index = ind(yr.eq.1950 .and. mm.eq.7  .and. hour.eq.00)
>>
>> ListSetType(f,"cat")
>> ta      =       f[:]->ta(index,:,:,:)
>> …………
>>
>> As I mentioned, this works great if mm.eq.1, mm.eq.2 etc., but not when
>> mm.eq.7. When reading in ta(index,:,:,:) I get the segmentation fault.
>>
>>
>> Thank you!
>>
>> Elinor
>>
>>
>>
>>
>>
>> *******************************************
>> Dr. Elinor Martin
>> Assistant Professor
>> School of Meteorology
>> University of Oklahoma
>> 120 David L. Boren Blvd. Suite 5900
>> Norman, OK 73072-7307
>>
>> Office: NWC 5642
>> Email: elinor.martin at ou.edu
>> Phone: 405 325 7392
>> Web: http://weather.ou.edu/~ermartin
>> *******************************************
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160830/6f813a4f/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: read_fixed_files_and_plot.ncl
Type: application/octet-stream
Size: 1044 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160830/6f813a4f/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fix_time.ncl
Type: application/octet-stream
Size: 2119 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160830/6f813a4f/attachment-0001.obj