[ncl-talk] error due to write more than 2 Gb values to a file

[Mary Haley]

Debasish,

I looked at the "fbinwrite" code, and it has a test in it that explicitly
doesn't allow you to write more than 2 GB to a file:

        if(totalsize > INT_MAX) {

          NhlPError(NhlFATAL,NhlEUNKNOWN,"fbinwrite: cannot write more than
2 Gb values to a file.");
          return(NhlFATAL);

        }

So, Dennis' suggestion is not going to work unfortunately.

Also, I found an older ncl-talk message on a similar issue, and there's a
statement, "Apparently, this is a gcc (or more precisely a gfortran) issue,
which internally, each written block has a limit of 2G."

You can see the full post and a possible work-around here:

https://www.ncl.ucar.edu/Support/talk_archives/2014/0209.html

I've added your email to a ticket that we already had open on
"fbindirwrite" (NCL-1889). I marked it for a 6.4.0 release, but I can't
guarantee this.  I'm not sure how hard the fix will be to allow all of our
binary write functions to write larger than 2 GB.

--Mary


On Mon, Mar 7, 2016 at 12:03 PM, Debasish Pai Mazumder <debasish at ucar.edu>
wrote:

> Hi Dennis,
>
> Thanks for your response. I am trying to write a variable with dimension
> (year = 16, hour = 368, lev = 19, lat = 66, lon = 80). Total file size I am
> trying to write is 2.3 GB. I have tried the setfileoption but getting same
> error message (fatal:fbinwrite: cannot write more than 2 Gb values to a
> file).
>
> outfilename = diri + "CESM_ASO_Tcor_" + mk + "_1990010100Z-2005123118Z.bin"
> system("/bin/rm -f "+ outfilename)
> setfileoption("bin", "RecordMarkerSize", 8)
> fbinwrite(outfilename, (/Tcor/))
>
> with regards
> -Deb
>
>
> On 3/7/16 10:40 AM, Dennis Shea wrote:
>
> Hi Debasish,
>
>
> Please read the description at:
> https://www.ncl.ucar.edu/Document/Functions/Built-in/fbinwrite.shtml
>
> Specifically:
>
> "By default, the record marker size at the beginning and end of Fortran
> sequential files is assumed to be 4 bytes. In V6.1.1 and later, you can
> change this to 8 bytes using the "RecordMarkerSize" option in the
> *setfileoption*
> <https://www.ncl.ucar.edu/Document/Functions/Built-in/setfileoption.shtml>
> procedure."
>
> [1] fortran's default integer is 4-bytes (signed). The range is
> -2,147,483,648 to 2,147,483,647
> [2] by default, fortran sequential records are preceded and postpended by
> a 'hidden' 4-byte value indicating the size of the record.
>               [hidden 4-byte value][data_record][hidden 4-byte value]
> [3] The max value for a signed 4-byte integer is 2,147,483,647  (think
> 2GB).
>
> So, in a sense, this is a fortran based issue.
>
> ---
> NCL's setfileoption allows the hidden record marker to be set to 8-bytes
>
>      setfileoption("bin", "RecordMarkerSize", 8)
>      fbinwrite(file_name, foo)                                 ; foo is
> variable
>
> This can be read via NCL but you must specify that an 8-byte record
> separator is needed.
>
>      setfileoption("bin", "RecordMarkerSize", 8)
>      foo = fbinread(file_name, rec_dims, "float")
>
>
> * --- *
> If reading the binary file in a fortran code, it is likely that either (i)
> a compiler level option will have to be used so that it recognizes that
> 8-byte separator is being used or (ii) a language extension function
> (similar to 'setfileoption') must be used.
>
> Cheers
>
> On Sat, Mar 5, 2016 at 6:40 PM, Debasish Pai Mazumder <
> <debasish at ucar.edu>debasish at ucar.edu> wrote:
>
>> Hi All,
>>
>> I am trying to write more than 2 Gb values to a binary file and I
>> received following error message
>> fatal:fbinwrite: cannot write more than 2 Gb values to a file.
>>
>> Is there any way to create this binary file without splitting in to two
>> files.
>>
>> with regards
>> -Debasish
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160309/329a4ee7/attachment.html