[ncl-talk] Adding time coordinate
Dave Allured - NOAA Affiliate
dave.allured at noaa.gov
Thu Jan 14 11:24:45 MST 2021
Glad it is working. Here is a minor suggestion not related to time of
day. In your current version, the grid cells around the edges of the
bounding box are cut in half, so a few obs data points might be missed.
Add one-half grid cell size along all 4 edges, and use these expanded
limits in the bounding box statement.
lon_min = lonE - 0.5 * dgrad
lon_max = lonW + 0.5 * dgrad
lat_min = latS - 0.5 * dgrad
lat_max = latN + 0.5 * dgrad
I am not sure this will make any difference in your case, but it is good
practice.
On Thu, Jan 14, 2021 at 5:01 AM Vanúcia Schumacher <
vanucia-schumacher at hotmail.com> wrote:
> This worked!
> Thank you so much for advancing and learning
> The NCL team is incredible!
> ------------------------------
> *De:* Dave Allured - NOAA Affiliate <dave.allured at noaa.gov>
> *Enviado:* quarta-feira, 13 de janeiro de 2021 19:08
> *Para:* Vanúcia Schumacher <vanucia-schumacher at hotmail.com>
> *Cc:* ncl-talk at ucar.edu <ncl-talk at ucar.edu>
> *Assunto:* Re: [ncl-talk] Adding time coordinate
>
> Good, now the desired file layout is clear. Let's start with the script
> version that you sent this morning.
>
> First compute time of day for each input obs, as number of minutes from
> start of day. You could also use hours or seconds here if you want, the
> code is about the same, however you want to do it. Add this right after
> you extract hours, minutes, seconds from the input data. You will not need
> to call cd_inv_calendar; this time of day is simple because all obs are on
> the same day.
>
> time_of_day = (hh * 60) + mm + (sc / 60.0)
>
> Now add two more grids in memory to compute the averaged time of day. Do
> this after you create the new_data array. Also add the same lat and lon
> coordinates to both, and set both new grids to zero.
>
> time_sum = new((/nlat,mlon/),"double")
> nobs = new ((/nlat,mlon/),"integer")
>
> Now add these two lines after "lon := lon(ii)". The purpose is to exclude
> obs outside of the selected bounding box.
>
> VAL2 := VAL(ii)
> time_of_day2 := time_of_day(ii)
>
> Inside the loop, change the first line to use VAL2 as shown here. Also
> add two more lines which sum up the time of day and take care of multiple
> obs at the same grid point.
>
> do i=0,N-1
> new_data({lat(i)},{lon(i)}) = new_data({lat(i)},{lon(i)}) +
> (/VAL2(i)/)
> time_sum({lat(i)},{lon(i)}) = time_sum({lat(i)},{lon(i)}) +
> time_of_day2(i)
> nobs({lat(i)},{lon(i)}) = nobs({lat(i)},{lon(i)}) +
> 1
> end do
>
> After the loop, compute the final time of day average values. The first
> line protects against divide by zero, and the second line preserves the
> attached coordinates:
>
> nobs2 = where (nobs.eq.0, nobs at _FillValue, nobs)
> avg_time_of_day = time_sum
> avg_time_of_day = time_sum / nobs2
>
> Now write avg_time_of_day to your output file, and you should have the
> time grid that you want. The units is "minutes since start of day". Use
> printVarSummary, print, and printMinMax as needed, to check intermediate
> results. Try to debug, but check back here if you get stuck.
>
>
> On Wed, Jan 13, 2021 at 11:53 AM Vanúcia Schumacher <
> vanucia-schumacher at hotmail.com> wrote:
>
> Yes, exactly.
> When multiple obs at the same grid point, the obs are summed, and the time
> of day can be averaged.
> A single grid with all obs combined for the whole day, and a second grid
> with time of day corresponding to each cell of lat and lon.
> This is what I'm trying to get
>
>
> ------------------------------
> *Enviado:* quarta-feira, 13 de janeiro de 2021 15:42
> *Para:* Vanúcia Schumacher <vanucia-schumacher at hotmail.com>
>
> So you want two grids total? A single grid with all obs combined for the
> whole day, and a second grid with time of day?
>
> When there are multiple obs at the same grid point, the obs are summed,
> and the time of day is averaged?
>
>
> On Wed, Jan 13, 2021 at 10:23 AM Vanúcia Schumacher <
> vanucia-schumacher at hotmail.com> wrote:
>
> About the data, is correct:
> yyyy-mm-dd hh:mn:sc (fractional part of the second)
> lat lon value
> 2015-01-25 00:04:34 ;239963527;
> -18.3521115; -59.866315; 18
> 2015-01-25 00:04:34 ;568370796; -15.2493556
> -58.7402566; -18
>
> There are several cases in which the hour and minute will be the same, but
> the lat and lon information changes, although often close.
> How could I attach as extra information of hh-mn, average value in the
> grid, with the same size of lat and lon without error
>
> ------------------------------
> *De:* Dennis Shea <shea at ucar.edu>
> *Enviado:* quarta-feira, 13 de janeiro de 2021 13:57
>
> What to say .....
>
> Here are two lines at beginning of the file.
>
> yyyy-mm-dd
> hh:mn:sc
> ????????? lat
> lon value
> 2015-01-25 00:04:34;239963527;-18.3521115;-59.866315;18
> 2015-01-25 00:04:34;568370796;-15.2493556;-58.7402566;-18
>
> As you can see the yyyy-mm-dd-hh-mb-sc are identical. Hence, as noted
> in a previous email the calculated times are identical. Hence, 'time' in
> not monotonic.
>
> I ***speculate*** that the ????????? is actually a fractional part of the
> second.
>
> %> ncl vanucia.dup_time_example.ncl | less
>
> Is that correct?
>
> It is the user's responsibility to know about and explore the data.
>
> On Wed, Jan 13, 2021 at 6:44 AM Vanúcia Schumacher <
> vanucia-schumacher at hotmail.com> wrote:
>
> I detailed the attached script better with comments on each step.
> The script's job is to take the data from the ascii file and create a
> regular grid with 1 km of spatial resolution.
> The variable (new_data) varies with values above 1 at some grid points.
> However, what I need to include in this script and I am not succeeding, is
> to include the hh and mn information for each variable in the grid point,
> when there are no missing values. If I have a value above 1 at the grid
> point, I want to know the hour and minute of that information.
>
> I made the corrections that Dennis suggested (attachment)
>
> I appreciate the progress
> Thanks
> ------------------------------
> *De:* Dennis Shea <shea at ucar.edu>
> *Enviado:* quarta-feira, 13 de janeiro de 2021 02:14
>
> I agree with DaveA.
>
> I , also, do not understand your code. You must do a better job of
> explaining the objective.
>
> [1] Do you just want to convert the text file to a netCDF file?
>
> [2] Some sort of grid???
>
> You define a grid style variable "new_data"
> new_data = new((/nlat,mlon,ntimes/),"float")
>
> new_data!0 = "lat2"
> new_data!1 = "lon2"
> new_data!2 = "time"
>
> new_data&lat2 = lat2
> new_data&lon2 = lon2
> new_data&time = time
>
> To me, it looks like you are trying to directly place the raw values into
> the 'new_data' array.
>
> In fact, because each time is unique, there will be ntime=dimsizes(time)=
> *6024 * grids with one value per grid.
>
> ;;;;new_data({lat(i)},{lon(i)},{time(i)}) =
> new_data({LAT(i)},{LON(i)},{time(i)})+(/var(i)/)
>
> ---
> Also, the way you are accessing the various fields is wrong.
>
> NCL is confusing the - delimeter embedded with thh yyyy-mm-dd with a
> numeric minus sign.
>
> 2015-02-19 00:01:27;733939380;-18.4340369;-56.829563;-28
> ^ ^
> ===================================================
>
> datas := asciiread(pthasc(j),-1,"string")
> nline = dimsizes(datas)
> print("nline="+nline)
>
> ; NCL code is confusing the - delimeter with a minus sign
> ; Change delimeter embedded with yyyy-mm-dd to yyyy mm dd
> ; There may be better ways but (1) converting string to character
> ; (2) converting - character to : character
> ; (3) change : character to : string
>
> datac := tochar(datas) ; (nline,:)
> datac(:,4) = tochar(":")
> datac(:,7) = tochar(":")
> datas = tostring(datac) ; overwrite original
> ;;print(datas)
> nfld = str_fields_count(datas(0), delim) ; look at 1st line only
> print("nfld="+nfld)
>
> yyyy := toint(str_get_field(datas, 1, delim) )
> mm := toint(str_get_field(datas, 2, delim) )
> dd := toint(str_get_field(datas, 3, delim) )
> hh := toint(str_get_field(datas, 4, delim) )
> mn := toint(str_get_field(datas, 5, delim) )
> sc := toint(str_get_field(datas, 6, delim) )
>
> time := cd_inv_calendar(yyyy,mm,dd,hh,mn,sc,tunits, 0)
> time!0 = "time"
> ntimes = dimsizes(time)
> printVarSummary(time)
> LAT := todouble( str_get_field(datas, 8, delin) ) ; fields
> count starts at 1
> LON := todouble( str_get_field(datas, 9, delim) )
> VAL := todouble( str_get_field(datas,10, delim) )
> print("time="+time+" yyyy="+yyyy+" mm="+mm+" dd="+dd+" hh="+hh \
> +" mn="+mn+" sc="+sc+" LAT="+LAT+" LON="+LON+" VAL="+VAL)
>
>
> On Tue, Jan 12, 2021 at 7:58 PM Dave Allured - NOAA Affiliate via ncl-talk
> <ncl-talk at mailman.ucar.edu> wrote:
>
> Sure, one netcdf file per day, 1 km grid. But simple application of the
> cd_inv_calendar function only gets you a long 1-D array of exact times. It
> is not sufficient to determine the structure of the desired result file.
>
> This still leaves the question of how you want to carry the time-of-day
> information inside that one file. The related questions are, do you want a
> 2-D or 3-D array in the file, and what are the times of day on the third
> dimension. If the array is 3-D, then that means multiple grids within a
> single day. There are many examples of multiple grids per day in other
> data sets, and they are almost always regularly spaced over time, such as
> one grid per hour.
>
> It is even possible to have a single data grid, and a companion grid that
> has exact time of day at each grid point. This would be unusual, but not
> difficult. Also, for this you would need to decide what to do about
> multiple obs at the same grid point.
>
> So forget the details of NCL for a minute. Please describe specifically
> the grid or grids that you would like, and their spacing across time of
> day. This will work any way you like, as long as you can fully describe
> the shape of the desired end product.
>
>
> On Tue, Jan 12, 2021 at 4:17 PM Vanúcia Schumacher <
> vanucia-schumacher at hotmail.com> wrote:
>
> So, in fact what I need is just the hour and minute information, but the
> easiest way I found was to use the cd_inv_calendar function.
> How do you suggest that I can attach this information to the new_data
> variable?
> The purpose of the script is to spatialize the input data to a regular 1km
> grid.
> The input data is just one example of several data in which I use to
> generate a netcdf file per day with this script.
>
>
> ------------------------------
> *De:* Dave Allured - NOAA Affiliate <dave.allured at noaa.gov>
> *Enviado:* terça-feira, 12 de janeiro de 2021 16:39
>
> Okay. It looks like your original dates and times are not regularly
> spaced, and you are trying to make a new set of time coordinates by setting
> seconds = 0. This results in more irregular time values with some
> duplicate values, which are not legal for coordinate subscripting. This is
> the error "Non-monotonic coordinate value".
>
> It looks like your input file spans all 24 hours in one day. Can you
> please explain exactly what you want the new time coordinates to look
> like? Are they regularly spaced or irregular? Start and end times?
>
>
> On Tue, Jan 12, 2021 at 4:51 AM Vanúcia Schumacher <
> vanucia-schumacher at hotmail.com> wrote:
>
> Using your sugestion:
> ntimes = dimsizes(time)
> new_data = new((/nlat,mlon,ntimes/),"float")
> I got this error:
> fatal:NclOneDValGetClosestIndex: Non-monotonic coordinate value being
> used, can't complete coordinate subscript
> fatal:Could not obtain coordinate indexes, unable to perform subscript
> fatal:["Execute.c":8640]:Execute: Error occurred at or near line 89 in
> file test.ncl
> The line 89: new_data({lat(i)},{lon(i)},{time(i)}) =
> new_data({lat(i)},{lon(i)},{time(i)})+(/var(i)/)
>
> Using this:
> ntimes = dimsizes(time(1))
> new_data = new((/nlat,mlon,ntimes/),"float")
> I got this error:
> fatal:Coordinate variables must be the same dimension as their dimension
> fatal:No coordinate variable exists for dimension (time) in variable
> (new_data)
> fatal:["Execute.c":8637]:Execute: Error occurred at or near line 86 in
> file test.ncl
> The line 86: new_data&time = time
>
> Attached is the script and a test data.
> Thanks
> ------------------------------
> *De:* Dave Allured - NOAA Affiliate <dave.allured at noaa.gov>
> *Enviado:* segunda-feira, 11 de janeiro de 2021 22:55
>
> Please show the complete text for line 75, the complete error message, and
> printVarSummary for the array new_data, just before line 75 is executed.
> Also please attach the current version of your script. Thank you.
>
>
> On Mon, Jan 11, 2021 at 5:39 PM Vanúcia Schumacher <
> vanucia-schumacher at hotmail.com> wrote:
>
> Thanks for the help, but the error persists in the line 75
>
> PrintVarSummary:
>
> Variable: nlat
> Type: integer
> Total Size: 4 bytes
> 1 values
> Number of Dimensions: 1
> Dimensions and sizes: [1]
> Coordinates:
>
> Variable: mlon
> Type: integer
> Total Size: 4 bytes
> 1 values
> Number of Dimensions: 1
> Dimensions and sizes: [1]
> Coordinates:
>
> Variable: ntimes
> Type: integer
> Total Size: 4 bytes
> 1 values
> Number of Dimensions: 1
> Dimensions and sizes: [1]
> Coordinates:
> (0) 6024
>
> *De:* Dave Allured - NOAA Affiliate <dave.allured at noaa.gov>
> *Enviado:* segunda-feira, 11 de janeiro de 2021 20:27
> *Para:* Vanúcia Schumacher <vanucia-schumacher at hotmail.com>
> *Cc:* ncl-talk at ucar.edu <ncl-talk at ucar.edu>
> *Assunto:* Re: [ncl-talk] Adding time coordinate
>
> Your script looks pretty good so far. This does not look right:
> new_data = new((/nlat,mlon,toint(time(1))/),"float")
>
> Try this:
> ntimes = dimsizes (time)
> new_data = new((/nlat,mlon,ntimes/),"float")
>
> Always use printVarSummary when you are having problems with arrays; and
> look at dimensions, coordinates, attributes, and such.
>
> Also, in the future, please show the *first* error received, and be sure
> to show exactly which line it occurs on. Usually the first error causes
> more problems later, and it is harder to sort out. I am not sure what
> happened this time, but that does not matter now. Good luck.
>
>
> On Mon, Jan 11, 2021 at 2:51 PM Vanúcia Schumacher via ncl-talk <
> ncl-talk at mailman.ucar.edu> wrote:
>
> Hi users,
>
> I need to add the time information (preference in hh and mn) extracted
> from an ascii file to a new variable (new_data from script), but I am not
> having success.
> Attached is the script I am trying using the function cd_inv_calendar and
> the error obtained:
> fatal:No coordinate variable exists for dimension (time) in variable
> (new_data)
>
> I appreciate some help to attach this information to my new variable
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210114/83345310/attachment.html>
More information about the ncl-talk
mailing list