<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><title></title><style type="text/css">.felamimail-body-blockquote {margin: 5px 10px 0 3px;padding-left: 10px;border-left: 2px solid #000088;} </style></head><body>Thank you<br><br>Just to clarify here <span class="_Tgc _s8w">the <b>regression coefficient</b> is synonymous with the slope or <b>trend</b>?<br>The main aim is to plot out the trend and overlay with p values.<br><br></span>Is there another way to calculate the significance apart from: <br><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">finobs_rtt = (/ (1.-betainc(df/(df+tval^2), df/2.0, beta_b))*100. /) ; probability in %?<br><br>Is it possible to do ttest on the trend analysis?<br></span></div><br><br>On Mar 6, 2018 10:30:30 PM, Adam Phillips wrote:
<br><blockquote class="felamimail-body-blockquote"><div dir="ltr">Hi Sri,<div>I've again pasted my code below but added numerous comments to answer your questions. I had a mistake in my original code, and changed this:</div><div><span style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;">finobs = tarr(0,:,:)</span><br></div><div><span style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;">to this:</span></div><div><span style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;"><span style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;">finobs = tarr(:,:,0)</span><br></span></div><div><br></div><div><br></div><div>;======================================================</div><div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">tarr = obs_ann(:,:,5::12) ; tarr will be dimensioned lat x lon x time</span></div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">tttt = dtrend_msg(ispan(0,dimsizes(tarr&time)-1,1),tarr,True,True) ; I am using dtrend_msg to calculate the linear trends of the tarr array. This has nothing to do with the significance calculation.</span></div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">finobs = tarr(:,:,0) ; preallocate space for finobs (lat x lon)</span></div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">finobs_rtt = finobs ; preallocate space for finobs_rtt (lat x lon)</span></div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">finobs = (/ onedtond(tttt@slope, (/dimsizes(obs_seas&lat),dimsizes(obs_seas&lon)/) ) /) ; reform the slope attribute to 2D (lat x lon) array. </span></div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;"> </span></div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">tval = tarr(:,:,0) ; preallocate space for tval (lat x lon)</span></div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">df = new((/dimsizes(tarr&lat),dimsizes(tarr&lon)/),"integer") ; preallocate space for df</span></div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">rc = regcoef(ispan(0,dimsizes(tarr&time)-1,1),tarr,tval,df) ; tval and df are filled in by regcoef</span></div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">df = equiv_sample_size(tarr,0.05,0) ; Optional! over rule df returned from regcoef. U<span style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;">p to you whether you want to use equiv_sample_size or not</span></span></div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">df = df-2<span style="white-space:pre-wrap;"> </span>;regcoef/equiv_sample_size return N, need N-2</span></div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">beta_b = tval</span></div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">beta_b = 0.5</span></div><div style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);"><span style="font-size:12.8px;">finobs_rtt = (/ (1.-betainc(df/(df+tval^2), df/2.0, beta_b))*100. /) ; probability in %</span></div><div><br></div><div><br></div>In the above finobs will have units of (for SST) degrees C per year.</div><div><br></div><div>Note that there are multiple ways to compute the linear trend. I choose to use dtrend (or dtrend_msg or dtrend_msg_n). You can regress your data against a straight line (as your regress_4 example does), or you can use the slopes returned from dtrend/dtrend_msg as is illustrated in example #2 here:<br><a href="https://www.ncl.ucar.edu/Document/Functions/Built-in/dtrend.shtml" target="_blank">https://www.ncl.ucar.edu/Document/Functions/Built-in/dtrend.shtml</a><br class="gmail-Apple-interchange-newline"><br></div><div>Note that on the same page example #4 shows one using dtrend (for the trends) and then regcoef/betainc to calculate the significance of the trends. </div><div>Hope that makes sense. If not, please respond to the ncl-talk email list.</div><div>Adam</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 6, 2018 at 1:18 AM, Sri Nandini <span dir="ltr"><<a href="mailto:snandini@marum.de" target="_blank">snandini@marum.de</a>></span> wrote:<br><blockquote class="felamimail-body-blockquote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div>Hello<br><br>Indeed i was<span style="font-size:12.8px;"> regressing a EOF principal component timeseries on a detrended time x lat x lon array (temperature). </span><blockquote class="felamimail-body-blockquote"><div dir="ltr"><div><span style="font-size:12.8px;">However, i managed to make that work by setting the right metadata.</span> <br></div></div></blockquote><blockquote class="felamimail-body-blockquote"><div dir="ltr"><div><span style="font-size:12.8px;"><br></span></div></div></blockquote><p>Now is for the calculation of significance of trends.</p><p>I have some questions regarding the code below, posted next to the line in the code:<br></p><blockquote class="felamimail-body-blockquote"><div dir="ltr"><div><span style="font-size:12.8px;"><br></span></div><div><span style="font-size:12.8px;">Here is some code that I use to calculate the significance of trends (using the betainc function as you were doing):</span></div><div><span style="font-size:12.8px;"><br></span></div><div><div><span style="font-size:12.8px;">tarr = obs_ann(:,:,5::12) =====>what is the data format? [lat, lon, time?]<br></span></div><div><span style="font-size:12.8px;">tttt = dtrend_msg(ispan(0,dimsizes(tarr&time)-1,1),tarr,True,True) ====>Are you using the </span><span style="font-size:12.8px;"><span style="font-size:12.8px;">dtrend_msg function because it returns attributes like @slope which is what i need for graphical purposes? </span>why not just use dtrend function?<br></span></div></div></div></blockquote><p><br></p><blockquote class="felamimail-body-blockquote"><div dir="ltr"><div><div><span style="font-size:12.8px;">finobs = tarr(0,:,:) ====>have you rearranged the inital data here?<br></span></div><div><span style="font-size:12.8px;">finobs_rtt = finobs</span></div><div><span style="font-size:12.8px;">finobs = (/ onedtond(tttt@slope, (/dimsizes(obs_seas&lat),dimsizes(obs_seas&lon)/) ) /) ===>i understand we get the slope from the detrend function and not from </span><span style="font-size:12.8px;"><span style="font-size:12.8px;">regcoef?</span></span></div><div><span style="font-size:12.8px;"> </span></div><div><span style="font-size:12.8px;">tval = tarr(:,:,0) ====>is this still arranged as lat, lon, time?<br></span></div><div><span style="font-size:12.8px;">df = new((/dimsizes(tarr&lat),dimsizes(tarr&lon)/),"integer") =====>defining new variable===degrees of freedom<br></span></div></div></div></blockquote><p> </p><blockquote class="felamimail-body-blockquote"><div dir="ltr"><div><div><span style="font-size:12.8px;"><br></span></div><div><span style="font-size:12.8px;">rc = regcoef(ispan(0,dimsizes(tarr&time)-1,1),tarr,tval,df) ======> i dont understand what is happening here? from NCL online, if you want to calculate trends <a href="https://www.ncl.ucar.edu/Applications/Scripts/regress_4.ncl" target="_blank">https://www.ncl.ucar.edu/Applications/Scripts/regress_4.ncl</a> shows to use </span><br><pre>;************************************************
; Calculate the regression coefficients (slopes)
;************************************************
rc = regCoef(year,xann(lat|:,lon|:,year|:))
rc@long_name = "Trend"
rc@units = xann@units+"/year"
copy_VarCoords(xann(0,:,:), rc) ; copy lat,lon coords
printVarSummary(rc)<br></pre></div></div></div></blockquote><blockquote class="felamimail-body-blockquote"><div dir="ltr"><div><div><pre><br></pre><span style="font-size:12.8px;"></span></div><div><span style="font-size:12.8px;">df = equiv_sample_size(tarr,0.05,0) ; Optional! over rule df returned from regcoef. U<span style="color:rgb(34,34,34);font-family:arial, sans-serif;font-size:12.8px;font-style:normal;font-weight:400;letter-spacing:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;">p to you whether you want to use equiv_sample_size or not</span></span></div><div><span style="font-size:12.8px;">df = df-2<span style="white-space:pre-wrap;"> </span>;regcoef/equiv_sample_size return N, need N-2</span></div><div><span style="font-size:12.8px;">beta_b = tval</span></div><div><span style="font-size:12.8px;">beta_b = 0.5</span></div><div><span style="font-size:12.8px;">finobs_rtt = (/ (1.-betainc(df/(df+tval^2), df/2.0, beta_b))*100. /) ====> is this ttest? with a 2 tailed probability?<br></span></div><div><span style="font-size:12.8px;"><br></span></div></div></div></blockquote><p>Would appreciate is someone would help guide me on this.</p><p>The idea is to just to do a simple trend analysis on temperature from 2020-2100 (80years) and see if its significant or not?</p><p><br></p><blockquote class="felamimail-body-blockquote"><div dir="ltr"><div><div><span style="font-size:12.8px;"><br></span></div><br><div><br></div></div><div><span style="font-size:12.8px;"><br></span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 5, 2018 at 2:36 AM, Sri Nandini <span dir="ltr"><<a href="mailto:snandini@marum.de" target="_blank">snandini@marum.de</a>></span> wrote:<br><blockquote class="felamimail-body-blockquote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div>Hello<br><br>I am trying to <span class="m_1488878025774862876m_-1226483225063639934st"><em>calculate statistical significance</em> of a <em>linear</em> trend (Regression analysis).<br>The regression analysis simple enough.<br><br>I have troubles understanding how the probabilities are computed based on the trend i calculate?<br>What i actually wanted to do is calculate the 2 tailed ttest probabilities but it doesn't seem possible with NCL, unless someone else knows of a way?<br>Or does </span><span class="m_1488878025774862876m_-1226483225063639934st"><em class="m_1488878025774862876m_-1226483225063639934quotelev3">using the NCL function "betainc" to get the p-value
for a student test is the same thing?<br><br></em>
<em class="m_1488878025774862876m_-1226483225063639934quotelev3">>>></em>My code is below:<br><br><br>f = addfile ("<a href="http://air.mon.mean.nc" target="_blank">air.mon.mean.nc</a>", "r")<br> latS = 29.3<br> latN = 67.26 <br> lonL = 19.<br> lonR = 60.<br><br> TIME1 = f->time<br> YYYY1 = cd_calendar(TIME1,-1)/100 ; entire file<br> iYYYY1 = ind(YYYY1.ge.yrStrt .and. YYYY1.le.yrLast)<br> t2m = f->air(iYYYY1,:,:)<br> printVarSummary(t2m)<br><br> T2M = month_to_season (t2m, season)<br> printVarSummary(T2M)<br> nyrs = dimsizes(T2M&time)<br><br> t2mw = T2M({lat|latS:latN},{lon|lonL:lonR},time|:)<br> printVarSummary(t2mw)<br><br> t2mw = dtrend(t2mw,False)<br> printVarSummary(t2mw)<br><br> eofT = eofunc_Wrap(t2mw, neof, optEOF)<br> printVarSummary(eofT)<br><br>;==========================================================================<br>eof_regres= eofT ;create an array with meta data <br>do ne=0,neof-1<br>eof_regres(ne,:,:)= regCoef(eof_ts(ne,:), t2mw(lat|:,lon|:,time|:));<br> end do<br> printVarSummary(eof_regres)<br><br>;==========================================================================<br>;Calculate significance at 95% confidence<br>;The regCoef function returns the following attributes: yintercept (y-intercept), tval (t-statistic), rstd (standard error of the estimated regression coefficient) and nptxy (number of elements used). <br>;==========================================================================<br><br> df = onedtond(eof_regres@nptxy,dimsizes(eof_regres))-2 ; degrees of freedom<br> tval = onedtond(eof_regres@tval, dimsizes(eof_regres)) ; t-statistic<br><br> b = tval ; b must be same size as tval (and df)<br> b = 0.5<br> prob = betainc(df/(df+tval^2),df/2.0,b)<br> printVarSummary(prob) ; prob(nlat,nlon)<br><br> ;copy_VarCoords(eof_regres,prob)<br> prob@long_name = "probability"<br> ;prob@units = "fraction: [0,1]"<br><br> prob_95=where (prob.ge.0.95,1,0) ; **** note 0.95 ****<br> copy_VarCoords(prob, prob_95)<br> printVarSummary(prob_95) <br><br> eof_regres@long_name = "regression coefficient"<br> prob@long_name = "probability"<br> <br>;==================================The error message i get is:<br><br>warning:VarVarWrite: rhs has no dimension name or coordinate variable, deleting name of lhs dimension number (1) and destroying coordinate var, use "(/../)" if this is not desired outcome<br>warning:VarVarWrite: rhs has no dimension name or coordinate variable, deleting name of lhs dimension number (2) and destroying coordinate var, use "(/../)" if this is not desired outcome<br>warning:["Execute.c":8640]:Execute: Error occurred at or near line 135 in file NAOregreT2M_obs.ncl<br><br><br>Variable: eof_regres<br>Type: float<br>Total Size: 49856 bytes<br> 12464 values<br>Number of Dimensions: 3<br>Dimensions and sizes: [evn | 2] x [76] x [82]<br>Coordinates: <br> evn: [1..2]<br>Number Of Attributes: 11<br> tval : <ARRAY of 6232 elements><br> yintercept : <ARRAY of 6232 elements><br> rstd : <ARRAY of 6232 elements><br> nptxy : <ARRAY of 6232 elements><br> long_name : EOF: DJF: Monthly mean of surface temperature<br> _FillValue : -9.96921e+36<br> method : transpose<br> matrix : covariance<br> pcvar : ( 59.16475, 20.32462 )<br> eval : ( 18982.65, 6521.03 )<br> eval_transpose : ( 169.488, 58.22348 )<br><br><br></span></div>
<br>_______________________________________________<br>
ncl-talk mailing list<br><a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br><a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a><br><br></blockquote></div><br><br style="clear:both;"><span class="HOEnZb"><span style="color:#888888;"></span></span><div><br></div>-- <br><div class="m_1488878025774862876gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div><div><span><span style="color:#888888;">Adam Phillips <br></span></span></div><span><span style="color:#888888;">Associate Scientist, </span></span><span><span style="color:#888888;">Climate and Global Dynamics Laboratory, NCAR<br></span></span></div></div><div><span><span style="color:#888888;"><a href="http://www.cgd.ucar.edu/staff/asphilli/" target="_blank">www.cgd.ucar.edu/staff/asphilli/</a> </span></span><span><span style="color:#888888;"><a target="_blank">303-497-1726</a> </span></span></div><span><span style="color:#888888;"></span></span><div><div><span><span style="color:#888888;"><br></span></span><div><span><span style="color:#888888;"><a href="http://www.cgd.ucar.edu/staff/asphilli" target="_blank"></a></span></span></div></div></div></div></div></div></div></div></div></div></div>
</div>
</blockquote></div>
<br>_______________________________________________<br>
ncl-talk mailing list<br><a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br><a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a><br><br></blockquote></div><br><br style="clear:both;"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div><div><span><span style="color:#888888;">Adam Phillips <br></span></span></div><span><span style="color:#888888;">Associate Scientist, </span></span><span><span style="color:#888888;">Climate and Global Dynamics Laboratory, NCAR<br></span></span></div></div><div><span><span style="color:#888888;"><a href="http://www.cgd.ucar.edu/staff/asphilli/" target="_blank">www.cgd.ucar.edu/staff/asphilli/</a> </span></span><span><span style="color:#888888;">303-497-1726 </span></span></div><span><span style="color:#888888;"></span></span><div><div><span><span style="color:#888888;"><br></span></span><div><span><span style="color:#888888;"><a href="http://www.cgd.ucar.edu/staff/asphilli" target="_blank"></a></span></span></div></div></div></div></div></div></div></div></div></div></div>
</div>
</blockquote><br></body></html>