[ncl-talk] Crash with eofunc_varimax_Wrap function
Allison Serakos
sera0021 at umn.edu
Wed Aug 3 13:11:26 MDT 2016
Hi Dennis,
The output for uname -a is: Darwin soil201051006.cfans.umn.edu 15.4.0
Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016;
root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64
The output for gcc --version is: Configured with:
--prefix=/Applications/Xcode.app/Contents/Developer/usr
--with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 7.3.0 (clang-703.0.29)
Target: x86_64-apple-darwin15.4.0
Thread model: posix
InstalledDir:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
I am able to obtain the same output and the rotation works fine from the
modified script you sent me of the annually averaged data. However, now I
am trying to produce REOFs of monthly and seasonally averaged data. I
noticed the main thing you modified in my original script for the annual
REOFs is the incorporation of the 'month_to_annual_weighted' function as
opposed to the 'month_to_annual' function that I was using to average my
monthly data into annual data. So I thought maybe that was the reason why
the 'eofunc_varimax_Wrap' function was not finishing, my data was not a
weighted average. So now to compute seasonal REOFs I am doing the following
to have a weighted seasonally averaged dataset (showing just Dec, Jan, Feb
as an example):
wgt = (/0.08493151, 0.08493151, 0.076712325/)
djf = new((/3,114,180,360/),float)
djf(0,:,:,:) = SST(11::12,:,:) ;December
djf(1,:,:,:) = SST(0::12,:,:) ;January
djf(2,:,:,:) = SST(1::12,:,:) ;February
djf_wgt_avg = dim_avg_wgt_n(djf,wgt,0,0)
copy_VarMeta(SST(11::12,:,:),djf_wgt_avg)
I then area weight my dfj_wgt_avg data, calculate the EOFs, and the REOFs
(using the 'eofunc_varimax_Wrap' function), however, the rotation will not
complete and stalls just like my original problem.
On another note, I also would like to create monthly REOFs and when I use
the raw data (which is monthly average values) the rotation stalls as well.
If you'd like I can send my scripts (one for the seasonal average, and one
for the monthly) securely to you.
Let me know if you have any suggestions. I appreciate your help.
Thank you,
Allison
On Tue, Aug 2, 2016 at 4:20 PM, Dennis Shea <shea at ucar.edu> wrote:
> Hi Allison
>
> I have run (see below) the script on my old MAC using NCL 6.3.0 and the
> beta 6.4.0 with no problems.
>
> ---------------------------
>
> %> ncl ncl-talk_reof.ncl
>
> Specifically:
>
> neval = 6
> optEOF = True
>
> ev = eofunc_Wrap(av,neval,optEOF)
> printVarSummary(ev)
> printMinMax(ev,0)
> print("===")
>
> ;***** VARIMAX ROTATION *****
>
> ev_rot = eofunc_varimax_Wrap(ev,-1)
> printVarSummary(ev_rot)
> printMinMax(ev_rot,0)
>
> ============ OUTPUT ===========
>
>
> Variable: ev
> Type: float
> Total Size: 1555200 bytes
> 388800 values
> Number of Dimensions: 3
> Dimensions and sizes: [evn | 6] x [latitude | 180] x [longitude | 360]
> Coordinates:
> evn: [1..6]
> latitude: [89.5..-89.5]
> longitude: [-179.5..179.5]
> Number Of Attributes: 7
> eval_transpose : ( 4.416919, 2.388321, 0.7132505, 0.5452091,
> 0.4846498, 0.388064 )
> eval : ( 1492.919, 807.2526, 241.0787, 184.2807, 163.8116, 131.1656 )
> pcvar : ( 34.8554, 18.84705, 5.6285, 4.302428, 3.824535, 3.062344 )
> matrix : covariance
> method : transpose
> _FillValue : -1000
> long_name : EOF: sst
> (0) EOF: sst : min=-0.0294479 max=0.0344466
> (0) ===
>
>
> Variable: ev_rot
> Type: float
> Total Size: 1555200 bytes
> 388800 values
> Number of Dimensions: 3
> Dimensions and sizes: [evn | 6] x [latitude | 180] x [longitude | 360]
> Coordinates:
> evn: [1..6]
> latitude: [89.5..-89.5]
> longitude: [-179.5..179.5]
> Number Of Attributes: 4
> pcvar_varimax : ( 20.40459, 19.06581, 13.1998, 9.32391, 4.929915,
> 3.596234 )
> variance_varimax : ( 873.9649, 816.6224, 565.3708, 399.3596, 211.157,
> 154.0331 )
> _FillValue : -1000
> op : Kaiser Varimax Rotation: opt=-1
> (0) min=-0.527019 max=0.743203
> (0) ===
>
> ==============
> What do the following return?
>
> %> uname -a
> %> gcc --version
>
> D
>
>
> On Mon, Aug 1, 2016 at 12:48 PM, Allison Serakos <sera0021 at umn.edu> wrote:
>
>> Hi Dennis,
>>
>> Neither 'eofunc_varimax' or 'eofunc_varimax_Wrap' will work.
>>
>> Let me know if you are able to take another look.
>>
>> Thanks for your help,
>> Allison
>>
>> On Wed, Jul 27, 2016 at 11:55 AM, Dennis Shea <shea at ucar.edu> wrote:
>>
>>> I'm am not sure why 'eofunc_varimax' would work and
>>> 'eofunc_varimax_Wrap' would not. The *_Wrap invokes the function and, where
>>> appropriate, adds attributes and coordinates. Here is the code used:
>>>
>>> undef ("eofunc_varimax_Wrap")
>>> function eofunc_varimax_Wrap (eof:numeric, optEVX:integer)
>>> local eofEVX
>>> begin
>>> eofEVX = eofunc_varimax(eof, optEVX) ; invoke built-in function
>>> eofEVX at op = "Kaiser Varimax Rotation: opt="+optEVX
>>> copy_VarCoords(eof, eofEVX)
>>>
>>> return (eofEVX) ; return
>>> end
>>>
>>> ===
>>>
>>> I am busy and will look later.
>>>
>>> On Tue, Jul 26, 2016 at 5:04 PM, Allison Serakos <sera0021 at umn.edu>
>>> wrote:
>>>
>>>> Hi Dennis,
>>>>
>>>> Thank you again for your assistance with my issue of the
>>>> eofunc_varimax_Wrap function stalling. I was able to produce REOF plots of
>>>> the annually averaged data. However, I am running into the same issue of
>>>> the eofunc_varimax_Wrap function stalling when I use the original monthly
>>>> data as well as using a weighted seasonal average instead of a weighted
>>>> annual average.
>>>>
>>>> I am wondering if you could explain your modifications to the script I
>>>> sent you. I am also wondering if you have an idea as to why now the
>>>> original monthly dataset and the weighted seasonally averaged dataset stall
>>>> with the eofunc_varimax_Wrap function.
>>>>
>>>> Thanks again for all your help.
>>>>
>>>> Kind regards,
>>>> Allison
>>>>
>>>> On Thu, Jun 30, 2016 at 8:09 AM, Dennis Shea <shea at ucar.edu> wrote:
>>>>
>>>>> offline from ncl-talk
>>>>>
>>>>> re my comment:
>>>>>
>>>>> I am not sure why you want to rotate.
>>>>> The 1st two modes look very distinct.
>>>>>
>>>>> ===
>>>>>
>>>>> Deriving a "global" eof is rather unusual for ocean grids. The
>>>>> atmosphere is 'continuous'. The oceans are 'discontinuous' due to the
>>>>> continents. The EOFs may be though of as orthogonal structures that result
>>>>> from covariances. Deriving a 'global' EOF is implicitly 'saying' that SST
>>>>> variations in the Kuroshio may be related to the Gulf Stream.
>>>>>
>>>>> An Example of using an EOF analysis over the ocean is:
>>>>>
>>>>> http://www.ncl.ucar.edu/Applications/eof.shtml
>>>>> Example 3
>>>>>
>>>>> Good Luck
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jun 29, 2016 at 4:17 PM, Dennis Shea <shea at ucar.edu> wrote:
>>>>>
>>>>>> Hi Allison,
>>>>>>
>>>>>> Can you please try the attached?
>>>>>>
>>>>>> I am not sure why you want to rotate.
>>>>>> The 1st two modes look very distinct.
>>>>>>
>>>>>> I did not plot anything. I'll let that to you
>>>>>>
>>>>>>
>>>>>> THX
>>>>>> D
>>>>>>
>>>>>> On Tue, Jun 28, 2016 at 8:07 AM, Allison Serakos <sera0021 at umn.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> script name: ncl-talk_reof.ncl
>>>>>>> data file name: HadISST_sst.nc.gz
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Allison
>>>>>>>
>>>>>>> On Mon, Jun 27, 2016 at 1:49 PM, Allison Serakos <sera0021 at umn.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I am using the eofunc_varimax_Wrap function to analyze global
>>>>>>>> rotated EOFs produced from the eofunc_Wrap function. However, when I run my
>>>>>>>> script, it hangs when it reaches the eofunc_varimax_Wrap function. The
>>>>>>>> script does not finish and I get no error message.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> The dataset that I am using is the HadISST 1.1 monthly average sea
>>>>>>>> surface temperature dataset. It is a 3-dimensional (time,
>>>>>>>> latitude, longitude) global 1° x 1° grid dataset with 1755 time
>>>>>>>> values, 180 latitude, and 360 longitude. I produce annually averaged values
>>>>>>>> and spatially weighted it prior to the EOF analysis.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I have been using a Mac machine with NCL version 6.3.0 and a Linux
>>>>>>>> machine with NCL version 6.2.1 and the program hangs when conducting the
>>>>>>>> rotation.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I tried running the script with global data for different time
>>>>>>>> periods (all between 113 and 102 years of global data) on both the Mac and
>>>>>>>> Linux machines and found that the function works only for some
>>>>>>>> time periods and crashes with others. These periods differ between the two
>>>>>>>> Mac and Linux machines that I used. I also tried reshaping the
>>>>>>>> data from a 3D matrix to a 2D matrix using the reshape function and
>>>>>>>> combining the spatial components. In this situation, the rotation output
>>>>>>>> was still the same as above for the respective time periods and machines.
>>>>>>>> Additionally, I tried reducing the spatial domain and found that when I
>>>>>>>> reduce the latitude range (anything smaller than 69S-90N) for any
>>>>>>>> time period the script runs on both machines and the function does not
>>>>>>>> crash.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I used ‘ncl –x’ to see if there is an error in my script, but the
>>>>>>>> script only crashes before the eofunc_varimax_Warp function with no
>>>>>>>> messages. When I run the script, NCL uses about 592.5MB of the 12GB of
>>>>>>>> memory on the Mac machine, so I do not believe this an issue of lack of
>>>>>>>> memory space.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I am wondering if there is a known problem with the
>>>>>>>> eofunc_varimax_Wrap function? I am also wondering if you have any thoughts
>>>>>>>> as to why the function may be crashing under certain situations and not
>>>>>>>> others?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Here is a copy of my EOF and Varimax Rotation script:
>>>>>>>>
>>>>>>>> ;----------------------------------------------------------------
>>>>>>>>
>>>>>>>> ;***** EOF *****
>>>>>>>>
>>>>>>>> neval = 6
>>>>>>>>
>>>>>>>> optEOF = True
>>>>>>>>
>>>>>>>> optEOF at method = "transpose"
>>>>>>>>
>>>>>>>> optEOF at jopt = 0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> eof = eofunc_Wrap(av,neval,optEOF)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ;***** VARIMAX ROTATION *****
>>>>>>>>
>>>>>>>> eof_rot = eofunc_varimax_Wrap(eof,-1)
>>>>>>>>
>>>>>>>> eof_varimax_reorder(eof_rot)
>>>>>>>>
>>>>>>>> ;----------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks for your help in advance.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Kind regards,
>>>>>>>>
>>>>>>>> Allison
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Allison Serakos
>>>>>>>> Graduate Research Assistant
>>>>>>>> Department of Soil, Water, and Climate
>>>>>>>> University of Minnesota
>>>>>>>> 1991 Upper Buford Circle, Rm S535
>>>>>>>> St. Paul, MN 55108
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Allison Serakos
>>>>>>> Graduate Research Assistant
>>>>>>> Department of Soil, Water, and Climate
>>>>>>> University of Minnesota
>>>>>>> 1991 Upper Buford Circle, Rm S535
>>>>>>> St. Paul, MN 55108
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ncl-talk mailing list
>>>>>>> ncl-talk at ucar.edu
>>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Allison Serakos
>>>> Graduate Research Assistant
>>>> Department of Soil, Water, and Climate
>>>> University of Minnesota
>>>> 1991 Upper Buford Circle, Rm S535
>>>> St. Paul, MN 55108
>>>>
>>>
>>>
>>
>>
>> --
>> Allison Serakos
>> Graduate Research Assistant
>> Department of Soil, Water, and Climate
>> University of Minnesota
>> 1991 Upper Buford Circle, Rm S535
>> St. Paul, MN 55108
>>
>
>
--
Allison Serakos
Graduate Research Assistant
Department of Soil, Water, and Climate
University of Minnesota
1991 Upper Buford Circle, Rm S535
St. Paul, MN 55108
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160803/26c6d8c8/attachment.html
More information about the ncl-talk
mailing list