[ncl-talk] subsetting across the date line

Sat Jan 12 18:29:25 MST 2019

Frequently the most relevant error message is the first one printed.  That
is true in this case: "NetCDF: One or more variable sizes violate format
constraints".  The message about define mode is not relevant.  However, the
later reference to line 89 is important.

"Violate format constraints" probably means that the write statement at
line 89 (fishing_hours) is trying to expand your output file past the
2-Gbyte normal limit for a netcdf-3 file.  Use printVarSummary just before
that write statement, to examine the dimensions of the output variable.

Also try inserting a temporary "exit" statement just before the write
statement, to examine the format and current dimensions of the output file
at an intermediate stage, with the ncdump -hs shell command.  Between the
file dump and size information from printVarSummary above, you may find the
cause of this particular error.

If you need to write more that 2 Gbytes, then switch to "netcdf-4 classic"
output format.  See documentation for addfile and setfileoption for how to
select the output format when creating a new file.

I offer these debugging suggestions because I can not analyze your script
for you.  I can help interpret diagnostic messages.

Also please note that you copied one of my previous corrections
improperly.  In all cases the longitude offset should be +360, never +180.

Good luck.

--Dave

On Sat, Jan 12, 2019 at 5:32 PM Sam McClatchie <smcclatchie at fishocean.info>
wrote:

> Thanks Dave and Dennis for your suggestions.
>
> I have tried all three suggested methods, but I am getting the same
> strange error message when any of these indexed substitutions is added to
> my code (which works without these additions to cross the dateline).
>
> Here are the errors returned following each suggestion:
>
> ; Dennis Shea suggestion
>  lon             = f->lon
>  lon             = where(lon.lt.0, lon+360, lon)   ; lon => 0-to-360
>
> ncendef: ncid 131072: NetCDF: One or more variable sizes violate format
> constraints
> fatal:NetCDF: Operation not allowed in define mode: error attempting to
> write variable (fishing_hours) to file (../data/Socal_GFW.nc)
> fatal:["Execute.c":8637]:Execute: Error occurred at or near line 89 in
> file selection_by_polygon_netcdf_data.ncl
>
> ncclose: ncid 131072: NetCDF: One or more variable sizes violate format
> constraints
>
> ------------------------------------
>
> ; Dave Allured suggestions
> xind = ind(lon.lt.0)
> lon(xind) = lon(xind) + 360
>
> or
>
> lon(ind(lon.lt.0)) = lon(ind(lon.lt.0)) + 180
>
> ncendef: ncid 131072: NetCDF: One or more variable sizes violate format
> constraints
> fatal:NetCDF: Operation not allowed in define mode: error attempting to
> write variable (fishing_hours) to file (../data/Socal_GFW.nc)
> fatal:["Execute.c":8637]:Execute: Error occurred at or near line 89 in
> file selection_by_polygon_netcdf_data.ncl
>
> ncclose: ncid 131072: NetCDF: One or more variable sizes violate format
> constraints
> ----------------------------------
>
> These errors strike me as strange because the fishing hour variable writes
> out just fine without the suggested additions to cross the dateline. Do the
> suggested new additions also require a shift from "define mode" before
> writing to netcdf? I do close the output file after writing.
>
> Puzzled,
> Sam
>
> On 13/01/19 11:18 AM, Dennis Shea wrote:
>
> Rather than:
>      lon(ind(lon.lt.0)) = -lon(ind(lon.lt.0)) + 180
> which invokes the *ind*  function* twice*, you could do
>      ilon0 := *ind*
> <https://www.ncl.ucar.edu/Document/Functions/Built-in/ind.shtml>(lon.lt.0)
>      lon(ilon0) = -lon(ilon0) + 180
>
> Personally, I would do the following upon input
>
> lon           = f->lon         ; input
> lon           = *where*
> <https://www.ncl.ucar.edu/Document/Functions/Built-in/where.shtml>(lon.lt.0,
> lon+360, lon)   ; lon => 0-to-360
>
> D
>
>
> On Sat, Jan 12, 2019 at 12:01 AM Sam McClatchie <
> smcclatchie at fishocean.info> wrote:
>
>> Colleagues
>>
>> I have a new question that is related to my original question. Initially,
>> I wanted to subset the large Global Fishing Watch dataset into regional
>> datasets, and then plot the unstructured station data as fishing hours
>> colored by their magnitude. Rick pointed me to the gc_inout function, and
>> Dennis wrote some code with nested loops to do the subsetting and plotting
>> at the same time.
>>
>> This was extremely helpful, but the loops were exceedingly slow because
>> there are over 250,000 points per region, so I sped up the plots by doing
>> the subsetting with a separate script before doing the plotting. I also
>> figured out how to use the new cmocean colour tables. The data, code and
>> figure produced are here:
>> <https://my.pcloud.com/publink/show?code=kZsanC7ZlKPCnfV5wnLFWm4C2yCww4mtUIYV>
>> <https://my.pcloud.com/publink/show?code=kZsanC7ZlKPCnfV5wnLFWm4C2yCww4mtUIYV>
>> .
>>
>> My new question is how to subset the data across the dateline (in the New
>> Zealand plot) when the longitude data run from -180:180? As you can see
>> from the plot, my plotted fishing hours data stops at the dateline.
>>
>> To conveniently subset beyond 180 degrees, I tried this to convert lon to
>> 0:360, but I'm getting a variable unassigned error for lon:
>>
>>                 ; test ************************************************
>>                 lon(ind(lon.lt.0)) = -lon(ind(lon.lt.0)) + 180
>>                 print("%%%%%%%%%%%%%%%%%%%%%%%%")
>>                 printVarSummary(lon)
>>                 print("%%%%%%%%%%%%%%%%%%%%%%%%")
>>                 ; end test **********************************************
>>
>> Using the reassignment operator (:=) does not fix the problem.
>>
>> Any hints would be most gratefully received.
>>
>> Best fishes
>> Sam
>> --
>> Sam McClatchie (fisheries oceanographer)
>> & Elena Turin (accounting & auditing)
>> FishOcean Enterprises
>> 38 Upland Rd, Huia, Auckland 0604, New Zealand
>> cell: 027 752 8495
>> Internet <http://www.fishocean.info>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190112/8df31ea2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SamMcClatchie2_small.jpg
Type: image/jpeg
Size: 45362 bytes
Desc: not available
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190112/8df31ea2/attachment.jpg>