[ncl-talk] Adding time coordinate
Dave Allured - NOAA Affiliate
dave.allured at noaa.gov
Wed Jan 13 15:08:32 MST 2021
Good, now the desired file layout is clear. Let's start with the script
version that you sent this morning.
First compute time of day for each input obs, as number of minutes from
start of day. You could also use hours or seconds here if you want, the
code is about the same, however you want to do it. Add this right after
you extract hours, minutes, seconds from the input data. You will not need
to call cd_inv_calendar; this time of day is simple because all obs are on
the same day.
time_of_day = (hh * 60) + mm + (sc / 60.0)
Now add two more grids in memory to compute the averaged time of day. Do
this after you create the new_data array. Also add the same lat and lon
coordinates to both, and set both new grids to zero.
time_sum = new((/nlat,mlon/),"double")
nobs = new ((/nlat,mlon/),"integer")
Now add these two lines after "lon := lon(ii)". The purpose is to exclude
obs outside of the selected bounding box.
VAL2 := VAL(ii)
time_of_day2 := time_of_day(ii)
Inside the loop, change the first line to use VAL2 as shown here. Also add
two more lines which sum up the time of day and take care of multiple obs
at the same grid point.
do i=0,N-1
new_data({lat(i)},{lon(i)}) = new_data({lat(i)},{lon(i)}) +
(/VAL2(i)/)
time_sum({lat(i)},{lon(i)}) = time_sum({lat(i)},{lon(i)}) +
time_of_day2(i)
nobs({lat(i)},{lon(i)}) = nobs({lat(i)},{lon(i)}) + 1
end do
After the loop, compute the final time of day average values. The first
line protects against divide by zero, and the second line preserves the
attached coordinates:
nobs2 = where (nobs.eq.0, nobs at _FillValue, nobs)
avg_time_of_day = time_sum
avg_time_of_day = time_sum / nobs2
Now write avg_time_of_day to your output file, and you should have the time
grid that you want. The units is "minutes since start of day". Use
printVarSummary, print, and printMinMax as needed, to check intermediate
results. Try to debug, but check back here if you get stuck.
On Wed, Jan 13, 2021 at 11:53 AM Vanúcia Schumacher <
vanucia-schumacher at hotmail.com> wrote:
> Yes, exactly.
> When multiple obs at the same grid point, the obs are summed, and the time
> of day can be averaged.
> A single grid with all obs combined for the whole day, and a second grid
> with time of day corresponding to each cell of lat and lon.
> This is what I'm trying to get
>
>
> ------------------------------
> *Enviado:* quarta-feira, 13 de janeiro de 2021 15:42
> *Para:* Vanúcia Schumacher <vanucia-schumacher at hotmail.com>
>
> So you want two grids total? A single grid with all obs combined for the
> whole day, and a second grid with time of day?
>
> When there are multiple obs at the same grid point, the obs are summed,
> and the time of day is averaged?
>
>
> On Wed, Jan 13, 2021 at 10:23 AM Vanúcia Schumacher <
> vanucia-schumacher at hotmail.com> wrote:
>
> About the data, is correct:
> yyyy-mm-dd hh:mn:sc (fractional part of the second)
> lat lon value
> 2015-01-25 00:04:34 ;239963527;
> -18.3521115; -59.866315; 18
> 2015-01-25 00:04:34 ;568370796; -15.2493556
> -58.7402566; -18
>
> There are several cases in which the hour and minute will be the same, but
> the lat and lon information changes, although often close.
> How could I attach as extra information of hh-mn, average value in the
> grid, with the same size of lat and lon without error
>
> ------------------------------
> *De:* Dennis Shea <shea at ucar.edu>
> *Enviado:* quarta-feira, 13 de janeiro de 2021 13:57
>
> What to say .....
>
> Here are two lines at beginning of the file.
>
> yyyy-mm-dd
> hh:mn:sc
> ????????? lat
> lon value
> 2015-01-25 00:04:34;239963527;-18.3521115;-59.866315;18
> 2015-01-25 00:04:34;568370796;-15.2493556;-58.7402566;-18
>
> As you can see the yyyy-mm-dd-hh-mb-sc are identical. Hence, as noted
> in a previous email the calculated times are identical. Hence, 'time' in
> not monotonic.
>
> I ***speculate*** that the ????????? is actually a fractional part of the
> second.
>
> %> ncl vanucia.dup_time_example.ncl | less
>
> Is that correct?
>
> It is the user's responsibility to know about and explore the data.
>
> On Wed, Jan 13, 2021 at 6:44 AM Vanúcia Schumacher <
> vanucia-schumacher at hotmail.com> wrote:
>
> I detailed the attached script better with comments on each step.
> The script's job is to take the data from the ascii file and create a
> regular grid with 1 km of spatial resolution.
> The variable (new_data) varies with values above 1 at some grid points.
> However, what I need to include in this script and I am not succeeding, is
> to include the hh and mn information for each variable in the grid point,
> when there are no missing values. If I have a value above 1 at the grid
> point, I want to know the hour and minute of that information.
>
> I made the corrections that Dennis suggested (attachment)
>
> I appreciate the progress
> Thanks
> ------------------------------
> *De:* Dennis Shea <shea at ucar.edu>
> *Enviado:* quarta-feira, 13 de janeiro de 2021 02:14
>
> I agree with DaveA.
>
> I , also, do not understand your code. You must do a better job of
> explaining the objective.
>
> [1] Do you just want to convert the text file to a netCDF file?
>
> [2] Some sort of grid???
>
> You define a grid style variable "new_data"
> new_data = new((/nlat,mlon,ntimes/),"float")
>
> new_data!0 = "lat2"
> new_data!1 = "lon2"
> new_data!2 = "time"
>
> new_data&lat2 = lat2
> new_data&lon2 = lon2
> new_data&time = time
>
> To me, it looks like you are trying to directly place the raw values into
> the 'new_data' array.
>
> In fact, because each time is unique, there will be ntime=dimsizes(time)=
> *6024 * grids with one value per grid.
>
> ;;;;new_data({lat(i)},{lon(i)},{time(i)}) =
> new_data({LAT(i)},{LON(i)},{time(i)})+(/var(i)/)
>
> ---
> Also, the way you are accessing the various fields is wrong.
>
> NCL is confusing the - delimeter embedded with thh yyyy-mm-dd with a
> numeric minus sign.
>
> 2015-02-19 00:01:27;733939380;-18.4340369;-56.829563;-28
> ^ ^
> ===================================================
>
> datas := asciiread(pthasc(j),-1,"string")
> nline = dimsizes(datas)
> print("nline="+nline)
>
> ; NCL code is confusing the - delimeter with a minus sign
> ; Change delimeter embedded with yyyy-mm-dd to yyyy mm dd
> ; There may be better ways but (1) converting string to character
> ; (2) converting - character to : character
> ; (3) change : character to : string
>
> datac := tochar(datas) ; (nline,:)
> datac(:,4) = tochar(":")
> datac(:,7) = tochar(":")
> datas = tostring(datac) ; overwrite original
> ;;print(datas)
> nfld = str_fields_count(datas(0), delim) ; look at 1st line only
> print("nfld="+nfld)
>
> yyyy := toint(str_get_field(datas, 1, delim) )
> mm := toint(str_get_field(datas, 2, delim) )
> dd := toint(str_get_field(datas, 3, delim) )
> hh := toint(str_get_field(datas, 4, delim) )
> mn := toint(str_get_field(datas, 5, delim) )
> sc := toint(str_get_field(datas, 6, delim) )
>
> time := cd_inv_calendar(yyyy,mm,dd,hh,mn,sc,tunits, 0)
> time!0 = "time"
> ntimes = dimsizes(time)
> printVarSummary(time)
> LAT := todouble( str_get_field(datas, 8, delin) ) ; fields
> count starts at 1
> LON := todouble( str_get_field(datas, 9, delim) )
> VAL := todouble( str_get_field(datas,10, delim) )
> print("time="+time+" yyyy="+yyyy+" mm="+mm+" dd="+dd+" hh="+hh \
> +" mn="+mn+" sc="+sc+" LAT="+LAT+" LON="+LON+" VAL="+VAL)
>
>
> On Tue, Jan 12, 2021 at 7:58 PM Dave Allured - NOAA Affiliate via ncl-talk
> <ncl-talk at mailman.ucar.edu> wrote:
>
> Sure, one netcdf file per day, 1 km grid. But simple application of the
> cd_inv_calendar function only gets you a long 1-D array of exact times. It
> is not sufficient to determine the structure of the desired result file.
>
> This still leaves the question of how you want to carry the time-of-day
> information inside that one file. The related questions are, do you want a
> 2-D or 3-D array in the file, and what are the times of day on the third
> dimension. If the array is 3-D, then that means multiple grids within a
> single day. There are many examples of multiple grids per day in other
> data sets, and they are almost always regularly spaced over time, such as
> one grid per hour.
>
> It is even possible to have a single data grid, and a companion grid that
> has exact time of day at each grid point. This would be unusual, but not
> difficult. Also, for this you would need to decide what to do about
> multiple obs at the same grid point.
>
> So forget the details of NCL for a minute. Please describe specifically
> the grid or grids that you would like, and their spacing across time of
> day. This will work any way you like, as long as you can fully describe
> the shape of the desired end product.
>
>
> On Tue, Jan 12, 2021 at 4:17 PM Vanúcia Schumacher <
> vanucia-schumacher at hotmail.com> wrote:
>
> So, in fact what I need is just the hour and minute information, but the
> easiest way I found was to use the cd_inv_calendar function.
> How do you suggest that I can attach this information to the new_data
> variable?
> The purpose of the script is to spatialize the input data to a regular 1km
> grid.
> The input data is just one example of several data in which I use to
> generate a netcdf file per day with this script.
>
>
> ------------------------------
> *De:* Dave Allured - NOAA Affiliate <dave.allured at noaa.gov>
> *Enviado:* terça-feira, 12 de janeiro de 2021 16:39
>
> Okay. It looks like your original dates and times are not regularly
> spaced, and you are trying to make a new set of time coordinates by setting
> seconds = 0. This results in more irregular time values with some
> duplicate values, which are not legal for coordinate subscripting. This is
> the error "Non-monotonic coordinate value".
>
> It looks like your input file spans all 24 hours in one day. Can you
> please explain exactly what you want the new time coordinates to look
> like? Are they regularly spaced or irregular? Start and end times?
>
>
> On Tue, Jan 12, 2021 at 4:51 AM Vanúcia Schumacher <
> vanucia-schumacher at hotmail.com> wrote:
>
> Using your sugestion:
> ntimes = dimsizes(time)
> new_data = new((/nlat,mlon,ntimes/),"float")
> I got this error:
> fatal:NclOneDValGetClosestIndex: Non-monotonic coordinate value being
> used, can't complete coordinate subscript
> fatal:Could not obtain coordinate indexes, unable to perform subscript
> fatal:["Execute.c":8640]:Execute: Error occurred at or near line 89 in
> file test.ncl
> The line 89: new_data({lat(i)},{lon(i)},{time(i)}) =
> new_data({lat(i)},{lon(i)},{time(i)})+(/var(i)/)
>
> Using this:
> ntimes = dimsizes(time(1))
> new_data = new((/nlat,mlon,ntimes/),"float")
> I got this error:
> fatal:Coordinate variables must be the same dimension as their dimension
> fatal:No coordinate variable exists for dimension (time) in variable
> (new_data)
> fatal:["Execute.c":8637]:Execute: Error occurred at or near line 86 in
> file test.ncl
> The line 86: new_data&time = time
>
> Attached is the script and a test data.
> Thanks
> ------------------------------
> *De:* Dave Allured - NOAA Affiliate <dave.allured at noaa.gov>
> *Enviado:* segunda-feira, 11 de janeiro de 2021 22:55
>
> Please show the complete text for line 75, the complete error message, and
> printVarSummary for the array new_data, just before line 75 is executed.
> Also please attach the current version of your script. Thank you.
>
>
> On Mon, Jan 11, 2021 at 5:39 PM Vanúcia Schumacher <
> vanucia-schumacher at hotmail.com> wrote:
>
> Thanks for the help, but the error persists in the line 75
>
> PrintVarSummary:
>
> Variable: nlat
> Type: integer
> Total Size: 4 bytes
> 1 values
> Number of Dimensions: 1
> Dimensions and sizes: [1]
> Coordinates:
>
> Variable: mlon
> Type: integer
> Total Size: 4 bytes
> 1 values
> Number of Dimensions: 1
> Dimensions and sizes: [1]
> Coordinates:
>
> Variable: ntimes
> Type: integer
> Total Size: 4 bytes
> 1 values
> Number of Dimensions: 1
> Dimensions and sizes: [1]
> Coordinates:
> (0) 6024
>
> *De:* Dave Allured - NOAA Affiliate <dave.allured at noaa.gov>
> *Enviado:* segunda-feira, 11 de janeiro de 2021 20:27
> *Para:* Vanúcia Schumacher <vanucia-schumacher at hotmail.com>
> *Cc:* ncl-talk at ucar.edu <ncl-talk at ucar.edu>
> *Assunto:* Re: [ncl-talk] Adding time coordinate
>
> Your script looks pretty good so far. This does not look right:
> new_data = new((/nlat,mlon,toint(time(1))/),"float")
>
> Try this:
> ntimes = dimsizes (time)
> new_data = new((/nlat,mlon,ntimes/),"float")
>
> Always use printVarSummary when you are having problems with arrays; and
> look at dimensions, coordinates, attributes, and such.
>
> Also, in the future, please show the *first* error received, and be sure
> to show exactly which line it occurs on. Usually the first error causes
> more problems later, and it is harder to sort out. I am not sure what
> happened this time, but that does not matter now. Good luck.
>
>
> On Mon, Jan 11, 2021 at 2:51 PM Vanúcia Schumacher via ncl-talk <
> ncl-talk at mailman.ucar.edu> wrote:
>
> Hi users,
>
> I need to add the time information (preference in hh and mn) extracted
> from an ascii file to a new variable (new_data from script), but I am not
> having success.
> Attached is the script I am trying using the function cd_inv_calendar and
> the error obtained:
> fatal:No coordinate variable exists for dimension (time) in variable
> (new_data)
>
> I appreciate some help to attach this information to my new variable
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210113/31e2d458/attachment.html>
More information about the ncl-talk
mailing list