<div dir="ltr">Hi Sri,<div>I've again pasted my code below but added numerous comments to answer your questions. I had a mistake in my original code, and changed this:</div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">finobs = tarr(0,:,:)</span><br></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">to this:</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">finobs = tarr(:,:,0)</span><br></span></div><div><br></div><div><br></div><div>;======================================================</div><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">tarr = obs_ann(:,:,5::12) ; tarr will be dimensioned lat x lon x time</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">tttt = dtrend_msg(ispan(0,dimsizes(<wbr>tarr&time)-1,1),tarr,True,<wbr>True) ; I am using dtrend_msg to calculate the linear trends of the tarr array. This has nothing to do with the significance calculation.</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">finobs = tarr(:,:,0) ; preallocate space for finobs (lat x lon)</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">finobs_rtt = finobs ; preallocate space for finobs_rtt (lat x lon)</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">finobs = (/ onedtond(tttt@slope, (/dimsizes(obs_seas&lat),<wbr>dimsizes(obs_seas&lon)/) ) /) ; reform the slope attribute to 2D (lat x lon) array. </span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px"> </span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">tval = tarr(:,:,0) ; preallocate space for tval (lat x lon)</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">df = new((/dimsizes(tarr&lat),<wbr>dimsizes(tarr&lon)/),"integer"<wbr>) ; preallocate space for df</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">rc = regcoef(ispan(0,dimsizes(tarr&<wbr>time)-1,1),tarr,tval,df) ; tval and df are filled in by regcoef</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">df = equiv_sample_size(tarr,0.05,0)<wbr> ; Optional! over rule df returned from regcoef. U<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">p to you whether you want to use equiv_sample_size or not</span></span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">df = df-2<span style="white-space:pre-wrap"> </span>;regcoef/equiv_sample_size return N, need N-2</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">beta_b = tval</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">beta_b = 0.5</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">finobs_rtt = (/ (1.-betainc(df/(df+tval^2), df/2.0, beta_b))*100. /) ; probability in %</span></div><div><br></div><div><br></div>In the above finobs will have units of (for SST) degrees C per year.</div><div><br></div><div>Note that there are multiple ways to compute the linear trend. I choose to use dtrend (or dtrend_msg or dtrend_msg_n). You can regress your data against a straight line (as your regress_4 example does), or you can use the slopes returned from dtrend/dtrend_msg as is illustrated in example #2 here:<br><a href="https://www.ncl.ucar.edu/Document/Functions/Built-in/dtrend.shtml">https://www.ncl.ucar.edu/Document/Functions/Built-in/dtrend.shtml</a><br class="gmail-Apple-interchange-newline"><br></div><div>Note that on the same page example #4 shows one using dtrend (for the trends) and then regcoef/betainc to calculate the significance of the trends. </div><div>Hope that makes sense. If not, please respond to the ncl-talk email list.</div><div>Adam</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 6, 2018 at 1:18 AM, Sri Nandini <span dir="ltr"><<a href="mailto:snandini@marum.de" target="_blank">snandini@marum.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Hello<br><br>Indeed i was<span style="font-size:12.8px"> regressing a EOF principal component timeseries on a detrended time x lat x lon array (temperature). </span><blockquote class="m_1488878025774862876felamimail-body-blockquote"><div dir="ltr"><div><span style="font-size:12.8px">However, i managed to make that work by setting the right metadata.</span> <br></div></div></blockquote><blockquote class="m_1488878025774862876felamimail-body-blockquote"><div dir="ltr"><div><span style="font-size:12.8px"><br></span></div></div></blockquote><p>Now is for the calculation of significance of trends.</p><p>I have some questions regarding the code below, posted next to the line in the code:<br></p><blockquote class="m_1488878025774862876felamimail-body-blockquote"><div dir="ltr"><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Here is some code that I use to calculate the significance of trends (using the betainc function as you were doing):</span></div><div><span style="font-size:12.8px"><br></span></div><div><div><span style="font-size:12.8px">tarr = obs_ann(:,:,5::12) <wbr> <wbr> =====>what is the data format? [lat, lon, time?]<br></span></div><div><span style="font-size:12.8px">tttt = dtrend_msg(ispan(0,dimsizes(<wbr>tarr&time)-1,1),tarr,True,<wbr>True) ====>Are you using the </span><span style="font-size:12.8px"><span style="font-size:12.8px">dtrend_msg function because it returns attributes like @slope which is what i need for graphical purposes? </span>why not just use dtrend function?<br></span></div></div></div></blockquote><p><br></p><blockquote class="m_1488878025774862876felamimail-body-blockquote"><div dir="ltr"><div><div><span style="font-size:12.8px">finobs = tarr(0,:,:) <wbr> <wbr> ====>have you rearranged the inital data here?<br></span></div><div><span style="font-size:12.8px">finobs_rtt = finobs</span></div><div><span style="font-size:12.8px">finobs = (/ onedtond(tttt@slope, (/dimsizes(obs_seas&lat),<wbr>dimsizes(obs_seas&lon)/) ) /) ===>i understand we get the slope from the detrend function and not from </span><span style="font-size:12.8px"><span style="font-size:12.8px">regcoef?</span></span></div><div><span style="font-size:12.8px"> </span></div><div><span style="font-size:12.8px">tval = tarr(:,:,0) ====>is this still arranged as lat, lon, time?<br></span></div><div><span style="font-size:12.8px">df = new((/dimsizes(tarr&lat),<wbr>dimsizes(tarr&lon)/),"integer"<wbr>) =====>defining new variable===degrees of freedom<br></span></div></div></div></blockquote><p> </p><blockquote class="m_1488878025774862876felamimail-body-blockquote"><div dir="ltr"><div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">rc = regcoef(ispan(0,dimsizes(tarr&<wbr>time)-1,1),tarr,tval,df) ======> i dont understand what is happening here? from NCL online, if you want to calculate trends <a href="https://www.ncl.ucar.edu/Applications/Scripts/regress_4.ncl" target="_blank">https://www.ncl.ucar.edu/<wbr>Applications/Scripts/regress_<wbr>4.ncl</a> shows to use </span><br><pre>;*****************************<wbr>*******************
; Calculate the regression coefficients (slopes)
;*****************************<wbr>*******************
rc = regCoef(year,xann(lat|:,lon|:,<wbr>year|:))
rc@long_name = "Trend"
rc@units = xann@units+"/year"
copy_VarCoords(xann(0,:,:), rc) ; copy lat,lon coords
printVarSummary(rc)<br></pre></div></div></div></blockquote><blockquote class="m_1488878025774862876felamimail-body-blockquote"><div dir="ltr"><div><div><pre><br></pre><span style="font-size:12.8px"></span></div><div><span style="font-size:12.8px">df = equiv_sample_size(tarr,0.05,0)<wbr> ; Optional! over rule df returned from regcoef. U<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none">p to you whether you want to use equiv_sample_size or not</span></span></div><div><span style="font-size:12.8px">df = df-2<span style="white-space:pre-wrap"> </span>;regcoef/equiv_sample_size return N, need N-2</span></div><div><span style="font-size:12.8px">beta_b = tval</span></div><div><span style="font-size:12.8px">beta_b = 0.5</span></div><div><span style="font-size:12.8px">finobs_rtt = (/ (1.-betainc(df/(df+tval^2), df/2.0, beta_b))*100. /) ====> is this ttest? with a 2 tailed probability?<br></span></div><div><span style="font-size:12.8px"><br></span></div></div></div></blockquote><p>Would appreciate is someone would help guide me on this.</p><p>The idea is to just to do a simple trend analysis on temperature from 2020-2100 (80years) and see if its significant or not?</p><p><br></p><blockquote class="m_1488878025774862876felamimail-body-blockquote"><div dir="ltr"><div><div><span style="font-size:12.8px"><br></span></div><br><div><br></div></div><div><span style="font-size:12.8px"><br></span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 5, 2018 at 2:36 AM, Sri Nandini <span dir="ltr"><<a href="mailto:snandini@marum.de" target="_blank">snandini@marum.de</a>></span> wrote:<br><blockquote class="m_1488878025774862876felamimail-body-blockquote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Hello<br><br>I am trying to <span class="m_1488878025774862876m_-1226483225063639934st"><em>calculate statistical significance</em> of a <em>linear</em> trend (Regression analysis).<br>The regression analysis simple enough.<br><br>I have troubles understanding how the probabilities are computed based on the trend i calculate?<br>What i actually wanted to do is calculate the 2 tailed ttest probabilities but it doesn't seem possible with NCL, unless someone else knows of a way?<br>Or does </span><span class="m_1488878025774862876m_-1226483225063639934st"><em class="m_1488878025774862876m_-1226483225063639934quotelev3">using the NCL function "betainc" to get the p-value
for a student test is the same thing?<br><br></em>
<em class="m_1488878025774862876m_-1226483225063639934quotelev3">>>></em>My code is below:<br><br><br>f = addfile ("<a href="http://air.mon.mean.nc" target="_blank">air.mon.mean.nc</a>", "r")<br> latS = 29.3<br> latN = 67.26 <br> lonL = 19.<br> lonR = 60.<br><br> TIME1 = f->time<br> YYYY1 = cd_calendar(TIME1,-1)/100 <wbr> ; entire file<br> iYYYY1 = ind(YYYY1.ge.yrStrt .and. YYYY1.le.yrLast)<br> t2m = f->air(iYYYY1,:,:)<br> printVarSummary(t2m)<br><br> T2M = month_to_season (t2m, season)<br> printVarSummary(T2M)<br> nyrs = dimsizes(T2M&time)<br><br> t2mw = T2M({lat|latS:latN},{lon|lonL:<wbr>lonR},time|:)<br> printVarSummary(t2mw)<br><br> t2mw = dtrend(t2mw,False)<br> printVarSummary(t2mw)<br><br> eofT = eofunc_Wrap(t2mw, neof, optEOF)<br> printVarSummary(eofT)<br><br>;=============================<wbr>==============================<wbr>===============<br>eof_regres= eofT <wbr> ;create an array with meta data <br>do ne=0,neof-1<br>eof_regres(ne,:,:)= regCoef(eof_ts(ne,:), t2mw(lat|:,lon|:,time|:));<br> end do<br> printVarSummary(eof_regres)<br><br>;=============================<wbr>==============================<wbr>===============<br>;Calculate significance at 95% confidence<br>;The regCoef function returns the following attributes: yintercept (y-intercept), tval (t-statistic), rstd (standard error of the estimated regression coefficient) and nptxy (number of elements used). <br>;=============================<wbr>==============================<wbr>===============<br><br> df = onedtond(eof_regres@nptxy,<wbr>dimsizes(eof_regres))-2 ; degrees of freedom<br> tval = onedtond(eof_regres@tval, dimsizes(eof_regres)) ; t-statistic<br><br> b = tval <wbr> ; b must be same size as tval (and df)<br> b = 0.5<br> prob = betainc(df/(df+tval^2),df/2.0,<wbr>b)<br> printVarSummary(prob) <wbr> ; prob(nlat,nlon)<br><br> ;copy_VarCoords(eof_regres,<wbr>prob)<br> prob@long_name = "probability"<br> ;prob@units = "fraction: [0,1]"<br><br> prob_95=where (prob.ge.0.95,1,0) ; **** note 0.95 ****<br> copy_VarCoords(prob, prob_95)<br> printVarSummary(prob_95) <br><br> eof_regres@long_name = "regression coefficient"<br> prob@long_name = "probability"<br> <br>;=============================<wbr>=====The error message i get is:<br><br>warning:VarVarWrite: rhs has no dimension name or coordinate variable, deleting name of lhs dimension number (1) and destroying coordinate var, use "(/../)" if this is not desired outcome<br>warning:VarVarWrite: rhs has no dimension name or coordinate variable, deleting name of lhs dimension number (2) and destroying coordinate var, use "(/../)" if this is not desired outcome<br>warning:["Execute.c":8640]:<wbr>Execute: Error occurred at or near line 135 in file NAOregreT2M_obs.ncl<br><br><br>Variable: eof_regres<br>Type: float<br>Total Size: 49856 bytes<br> 12464 values<br>Number of Dimensions: 3<br>Dimensions and sizes: [evn | 2] x [76] x [82]<br>Coordinates: <br> evn: [1..2]<br>Number Of Attributes: 11<br> tval : <ARRAY of 6232 elements><br> yintercept : <ARRAY of 6232 elements><br> rstd : <ARRAY of 6232 elements><br> nptxy : <ARRAY of 6232 elements><br> long_name : EOF: DJF: Monthly mean of surface temperature<br> _FillValue : -9.96921e+36<br> method : transpose<br> matrix : covariance<br> pcvar : ( 59.16475, 20.32462 )<br> eval : ( 18982.65, 6521.03 )<br> eval_transpose : ( 169.488, 58.22348 )<br><br><br></span></div>
<br>______________________________<wbr>_________________<br>
ncl-talk mailing list<br><a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br><a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" target="_blank">http://mailman.ucar.edu/<wbr>mailman/listinfo/ncl-talk</a><br><br></blockquote></div><br><br style="clear:both"><span class="HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="m_1488878025774862876gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div><div><span><span style="color:#888888">Adam Phillips <br></span></span></div><span><span style="color:#888888">Associate Scientist, </span></span><span><span style="color:#888888">Climate and Global Dynamics Laboratory, NCAR<br></span></span></div></div><div><span><span style="color:#888888"><a href="http://www.cgd.ucar.edu/staff/asphilli/" target="_blank">www.cgd.ucar.edu/staff/<wbr>asphilli/</a> </span></span><span><span style="color:#888888"><a href="tel:(303)%20497-1726" value="+13034971726" target="_blank">303-497-1726</a> </span></span></div><span><span style="color:#888888"></span></span><div><div><span><span style="color:#888888"><br></span></span><div><span><span style="color:#888888"><a href="http://www.cgd.ucar.edu/staff/asphilli" target="_blank"></a></span></span></div></div></div></div></div></div></div></div></div></div></div>
</font></span></div>
</blockquote></div>
<br>______________________________<wbr>_________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/<wbr>mailman/listinfo/ncl-talk</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div><div><span><font color="#888888">Adam Phillips <br></font></span></div><span><font color="#888888">Associate Scientist, </font></span><span><font color="#888888">Climate and Global Dynamics Laboratory, NCAR<br></font></span></div></div><div><span><font color="#888888"><a href="http://www.cgd.ucar.edu/staff/asphilli/" target="_blank">www.cgd.ucar.edu/staff/asphilli/</a> </font></span><span><font color="#888888">303-497-1726 </font></span></div><span><font color="#888888"></font></span><div><div><span><font color="#888888"><br></font></span><div><span><font color="#888888"><a href="http://www.cgd.ucar.edu/staff/asphilli" target="_blank"></a></font></span></div></div></div></div></div></div></div></div></div></div></div>
</div>