[ncl-talk] clarity on missing values in svdcov_sv and other lapack rountines

[Adam Herrington]

just to keep anyone from looking into this, I've found a solution that I'm
happy with. If I don't remove the zero's, I actually get a cleaner signal:

;;;DON'T REMOVE ZEROS
;;;print("remove zeros")
;;;Xp at _FillValue = default_fillvalue("float")
;;;Yp at _FillValue = default_fillvalue("float")

;;;remove fillvalue attribute
  delete(Xp at _FillValue)
  delete(Yp at _FillValue)

  lapack = svdcov_sv(Xp,Yp,nsvd,L,R)

Gives me the same number of warnings as when I had missing values, because
the same warning applies to a bunch of zeros as it does to a bunch of
missing values....however, the results look a little cleaner when I don't
use missing values.

Adam


On Sat, May 27, 2017 at 12:21 PM, Adam Herrington <
adam.herrington at stonybrook.edu> wrote:

> the svdcov_sv documentation (http://www.ncl.ucar.edu/
> Document/Functions/Built-in/svdcov_sv.shtml) states that "This function
> does not allow for missing data." However, the function will not exit, but
> rather gives you a warning. For example,
>
> print("remove zeros")
>   Xp at _FillValue = default_fillvalue("float")
>   Yp at _FillValue = default_fillvalue("float")
>
>   Xp = where(Xp.ne.0.,Xp,Xp at _FillValue)
>   Yp = where(Yp.ne.0.,Yp,Yp at _FillValue)
>
>   lapack = svdcov_sv(Xp,Yp,nsvd,L,R)
>
> I get about 4000 of these warnings:
>  "SVD: Warning all X values in column are missing or are constant"
>
> Followed by about 1000 of these warnings
>  "SVD: Warning all Y values in column are missing or are constant"
>
> I'm trying to figure out how much of the data are being ignored, because I
> have reason to believe the L and R singular vectors that come out of this
> example are using quite abit of the data still.
>
> In the example, Xp and Yp are dimensioned (/ntime,ncols/), where ntime =
> 980 and ncols = 30*48602. Since the number of warnings is much larger than
> ntime, I don't know how to interpret what the warning is referring to when
> they say "column".
>
> In trying to track down the code, I've scanned the entire directory
> $NCARG_ROOT/lib/ncarg/nclscripts/csm/ and can't find any references to
> the wrappers used to create svdcov_sv.
>
> The "eofunc" also uses lapack and the documentation (
> http://www.ncl.ucar.edu/Document/Functions/Built-in/eofunc.shtml) states:
>
> "Missing values are ignored when computing the covariance or correlation
> matrix"
>
> I've actually had some success in passing missing values using this
> function. Is the handling of missing values the same for the svdcov_sv
> function?
>
> In contrast, the function "dgeevx_lapack" is very sensitive to missing
> values. In fact, the function will exit even if a missing value attribute
> is set, and there are no actual missing values.
>
> So I'm hoping to get some clarity on how missing values are handled. Is it
> possible to get the FORTRAN lapack code to understand the error messages
> and potentially make modifications?
>
> Adam
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170527/0808df93/attachment.html