[ncl-talk] Adding time coordinate
Vanúcia Schumacher
vanucia-schumacher at hotmail.com
Thu Jan 14 05:00:54 MST 2021
This worked!
Thank you so much for advancing and learning
The NCL team is incredible!
________________________________
De: Dave Allured - NOAA Affiliate <dave.allured at noaa.gov>
Enviado: quarta-feira, 13 de janeiro de 2021 19:08
Para: Vanúcia Schumacher <vanucia-schumacher at hotmail.com>
Cc: ncl-talk at ucar.edu <ncl-talk at ucar.edu>
Assunto: Re: [ncl-talk] Adding time coordinate
Good, now the desired file layout is clear. Let's start with the script version that you sent this morning.
First compute time of day for each input obs, as number of minutes from start of day. You could also use hours or seconds here if you want, the code is about the same, however you want to do it. Add this right after you extract hours, minutes, seconds from the input data. You will not need to call cd_inv_calendar; this time of day is simple because all obs are on the same day.
time_of_day = (hh * 60) + mm + (sc / 60.0)
Now add two more grids in memory to compute the averaged time of day. Do this after you create the new_data array. Also add the same lat and lon coordinates to both, and set both new grids to zero.
time_sum = new((/nlat,mlon/),"double")
nobs = new ((/nlat,mlon/),"integer")
Now add these two lines after "lon := lon(ii)". The purpose is to exclude obs outside of the selected bounding box.
VAL2 := VAL(ii)
time_of_day2 := time_of_day(ii)
Inside the loop, change the first line to use VAL2 as shown here. Also add two more lines which sum up the time of day and take care of multiple obs at the same grid point.
do i=0,N-1
new_data({lat(i)},{lon(i)}) = new_data({lat(i)},{lon(i)}) + (/VAL2(i)/)
time_sum({lat(i)},{lon(i)}) = time_sum({lat(i)},{lon(i)}) + time_of_day2(i)
nobs({lat(i)},{lon(i)}) = nobs({lat(i)},{lon(i)}) + 1
end do
After the loop, compute the final time of day average values. The first line protects against divide by zero, and the second line preserves the attached coordinates:
nobs2 = where (nobs.eq.0, nobs at _FillValue, nobs)
avg_time_of_day = time_sum
avg_time_of_day = time_sum / nobs2
Now write avg_time_of_day to your output file, and you should have the time grid that you want. The units is "minutes since start of day". Use printVarSummary, print, and printMinMax as needed, to check intermediate results. Try to debug, but check back here if you get stuck.
On Wed, Jan 13, 2021 at 11:53 AM Vanúcia Schumacher <vanucia-schumacher at hotmail.com<mailto:vanucia-schumacher at hotmail.com>> wrote:
Yes, exactly.
When multiple obs at the same grid point, the obs are summed, and the time of day can be averaged.
A single grid with all obs combined for the whole day, and a second grid with time of day corresponding to each cell of lat and lon.
This is what I'm trying to get
________________________________
Enviado: quarta-feira, 13 de janeiro de 2021 15:42
Para: Vanúcia Schumacher <vanucia-schumacher at hotmail.com<mailto:vanucia-schumacher at hotmail.com>>
So you want two grids total? A single grid with all obs combined for the whole day, and a second grid with time of day?
When there are multiple obs at the same grid point, the obs are summed, and the time of day is averaged?
On Wed, Jan 13, 2021 at 10:23 AM Vanúcia Schumacher <vanucia-schumacher at hotmail.com<mailto:vanucia-schumacher at hotmail.com>> wrote:
About the data, is correct:
yyyy-mm-dd hh:mn:sc (fractional part of the second) lat lon value
2015-01-25 00:04:34 ;239963527; -18.3521115; -59.866315; 18
2015-01-25 00:04:34 ;568370796; -15.2493556 -58.7402566; -18
There are several cases in which the hour and minute will be the same, but the lat and lon information changes, although often close.
How could I attach as extra information of hh-mn, average value in the grid, with the same size of lat and lon without error
________________________________
De: Dennis Shea <shea at ucar.edu<mailto:shea at ucar.edu>>
Enviado: quarta-feira, 13 de janeiro de 2021 13:57
What to say .....
Here are two lines at beginning of the file.
yyyy-mm-dd
hh:mn:sc
????????? lat lon value
2015-01-25 00:04:34;239963527;-18.3521115;-59.866315;18
2015-01-25 00:04:34;568370796;-15.2493556;-58.7402566;-18
As you can see the yyyy-mm-dd-hh-mb-sc are identical. Hence, as noted in a previous email the calculated times are identical. Hence, 'time' in not monotonic.
I ***speculate*** that the ????????? is actually a fractional part of the second.
%> ncl vanucia.dup_time_example.ncl | less
Is that correct?
It is the user's responsibility to know about and explore the data.
On Wed, Jan 13, 2021 at 6:44 AM Vanúcia Schumacher <vanucia-schumacher at hotmail.com<mailto:vanucia-schumacher at hotmail.com>> wrote:
I detailed the attached script better with comments on each step.
The script's job is to take the data from the ascii file and create a regular grid with 1 km of spatial resolution.
The variable (new_data) varies with values above 1 at some grid points.
However, what I need to include in this script and I am not succeeding, is to include the hh and mn information for each variable in the grid point, when there are no missing values. If I have a value above 1 at the grid point, I want to know the hour and minute of that information.
I made the corrections that Dennis suggested (attachment)
I appreciate the progress
Thanks
________________________________
De: Dennis Shea <shea at ucar.edu<mailto:shea at ucar.edu>>
Enviado: quarta-feira, 13 de janeiro de 2021 02:14
I agree with DaveA.
I , also, do not understand your code. You must do a better job of explaining the objective.
[1] Do you just want to convert the text file to a netCDF file?
[2] Some sort of grid???
You define a grid style variable "new_data"
new_data = new((/nlat,mlon,ntimes/),"float")
new_data!0 = "lat2"
new_data!1 = "lon2"
new_data!2 = "time"
new_data&lat2 = lat2
new_data&lon2 = lon2
new_data&time = time
To me, it looks like you are trying to directly place the raw values into the 'new_data' array.
In fact, because each time is unique, there will be ntime=dimsizes(time)=6024 grids with one value per grid.
;;;;new_data({lat(i)},{lon(i)},{time(i)}) = new_data({LAT(i)},{LON(i)},{time(i)})+(/var(i)/)
---
Also, the way you are accessing the various fields is wrong.
NCL is confusing the - delimeter embedded with thh yyyy-mm-dd with a numeric minus sign.
2015-02-19 00:01:27;733939380;-18.4340369;-56.829563;-28
^ ^
===================================================
datas := asciiread(pthasc(j),-1,"string")
nline = dimsizes(datas)
print("nline="+nline)
; NCL code is confusing the - delimeter with a minus sign
; Change delimeter embedded with yyyy-mm-dd to yyyy mm dd
; There may be better ways but (1) converting string to character
; (2) converting - character to : character
; (3) change : character to : string
datac := tochar(datas) ; (nline,:)
datac(:,4) = tochar(":")
datac(:,7) = tochar(":")
datas = tostring(datac) ; overwrite original
;;print(datas)
nfld = str_fields_count(datas(0), delim) ; look at 1st line only
print("nfld="+nfld)
yyyy := toint(str_get_field(datas, 1, delim) )
mm := toint(str_get_field(datas, 2, delim) )
dd := toint(str_get_field(datas, 3, delim) )
hh := toint(str_get_field(datas, 4, delim) )
mn := toint(str_get_field(datas, 5, delim) )
sc := toint(str_get_field(datas, 6, delim) )
time := cd_inv_calendar(yyyy,mm,dd,hh,mn,sc,tunits, 0)
time!0 = "time"
ntimes = dimsizes(time)
printVarSummary(time)
LAT := todouble( str_get_field(datas, 8, delin) ) ; fields count starts at 1
LON := todouble( str_get_field(datas, 9, delim) )
VAL := todouble( str_get_field(datas,10, delim) )
print("time="+time+" yyyy="+yyyy+" mm="+mm+" dd="+dd+" hh="+hh \
+" mn="+mn+" sc="+sc+" LAT="+LAT+" LON="+LON+" VAL="+VAL)
On Tue, Jan 12, 2021 at 7:58 PM Dave Allured - NOAA Affiliate via ncl-talk <ncl-talk at mailman.ucar.edu<mailto:ncl-talk at mailman.ucar.edu>> wrote:
Sure, one netcdf file per day, 1 km grid. But simple application of the cd_inv_calendar function only gets you a long 1-D array of exact times. It is not sufficient to determine the structure of the desired result file.
This still leaves the question of how you want to carry the time-of-day information inside that one file. The related questions are, do you want a 2-D or 3-D array in the file, and what are the times of day on the third dimension. If the array is 3-D, then that means multiple grids within a single day. There are many examples of multiple grids per day in other data sets, and they are almost always regularly spaced over time, such as one grid per hour.
It is even possible to have a single data grid, and a companion grid that has exact time of day at each grid point. This would be unusual, but not difficult. Also, for this you would need to decide what to do about multiple obs at the same grid point.
So forget the details of NCL for a minute. Please describe specifically the grid or grids that you would like, and their spacing across time of day. This will work any way you like, as long as you can fully describe the shape of the desired end product.
On Tue, Jan 12, 2021 at 4:17 PM Vanúcia Schumacher <vanucia-schumacher at hotmail.com<mailto:vanucia-schumacher at hotmail.com>> wrote:
So, in fact what I need is just the hour and minute information, but the easiest way I found was to use the cd_inv_calendar function.
How do you suggest that I can attach this information to the new_data variable?
The purpose of the script is to spatialize the input data to a regular 1km grid.
The input data is just one example of several data in which I use to generate a netcdf file per day with this script.
________________________________
De: Dave Allured - NOAA Affiliate <dave.allured at noaa.gov<mailto:dave.allured at noaa.gov>>
Enviado: terça-feira, 12 de janeiro de 2021 16:39
Okay. It looks like your original dates and times are not regularly spaced, and you are trying to make a new set of time coordinates by setting seconds = 0. This results in more irregular time values with some duplicate values, which are not legal for coordinate subscripting. This is the error "Non-monotonic coordinate value".
It looks like your input file spans all 24 hours in one day. Can you please explain exactly what you want the new time coordinates to look like? Are they regularly spaced or irregular? Start and end times?
On Tue, Jan 12, 2021 at 4:51 AM Vanúcia Schumacher <vanucia-schumacher at hotmail.com<mailto:vanucia-schumacher at hotmail.com>> wrote:
Using your sugestion:
ntimes = dimsizes(time)
new_data = new((/nlat,mlon,ntimes/),"float")
I got this error:
fatal:NclOneDValGetClosestIndex: Non-monotonic coordinate value being used, can't complete coordinate subscript
fatal:Could not obtain coordinate indexes, unable to perform subscript
fatal:["Execute.c":8640]:Execute: Error occurred at or near line 89 in file test.ncl
The line 89: new_data({lat(i)},{lon(i)},{time(i)}) = new_data({lat(i)},{lon(i)},{time(i)})+(/var(i)/)
Using this:
ntimes = dimsizes(time(1))
new_data = new((/nlat,mlon,ntimes/),"float")
I got this error:
fatal:Coordinate variables must be the same dimension as their dimension
fatal:No coordinate variable exists for dimension (time) in variable (new_data)
fatal:["Execute.c":8637]:Execute: Error occurred at or near line 86 in file test.ncl
The line 86: new_data&time = time
Attached is the script and a test data.
Thanks
________________________________
De: Dave Allured - NOAA Affiliate <dave.allured at noaa.gov<mailto:dave.allured at noaa.gov>>
Enviado: segunda-feira, 11 de janeiro de 2021 22:55
Please show the complete text for line 75, the complete error message, and printVarSummary for the array new_data, just before line 75 is executed. Also please attach the current version of your script. Thank you.
On Mon, Jan 11, 2021 at 5:39 PM Vanúcia Schumacher <vanucia-schumacher at hotmail.com<mailto:vanucia-schumacher at hotmail.com>> wrote:
Thanks for the help, but the error persists in the line 75
PrintVarSummary:
Variable: nlat
Type: integer
Total Size: 4 bytes
1 values
Number of Dimensions: 1
Dimensions and sizes: [1]
Coordinates:
Variable: mlon
Type: integer
Total Size: 4 bytes
1 values
Number of Dimensions: 1
Dimensions and sizes: [1]
Coordinates:
Variable: ntimes
Type: integer
Total Size: 4 bytes
1 values
Number of Dimensions: 1
Dimensions and sizes: [1]
Coordinates:
(0) 6024
De: Dave Allured - NOAA Affiliate <dave.allured at noaa.gov<mailto:dave.allured at noaa.gov>>
Enviado: segunda-feira, 11 de janeiro de 2021 20:27
Para: Vanúcia Schumacher <vanucia-schumacher at hotmail.com<mailto:vanucia-schumacher at hotmail.com>>
Cc: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu> <ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
Assunto: Re: [ncl-talk] Adding time coordinate
Your script looks pretty good so far. This does not look right:
new_data = new((/nlat,mlon,toint(time(1))/),"float")
Try this:
ntimes = dimsizes (time)
new_data = new((/nlat,mlon,ntimes/),"float")
Always use printVarSummary when you are having problems with arrays; and look at dimensions, coordinates, attributes, and such.
Also, in the future, please show the *first* error received, and be sure to show exactly which line it occurs on. Usually the first error causes more problems later, and it is harder to sort out. I am not sure what happened this time, but that does not matter now. Good luck.
On Mon, Jan 11, 2021 at 2:51 PM Vanúcia Schumacher via ncl-talk <ncl-talk at mailman.ucar.edu<mailto:ncl-talk at mailman.ucar.edu>> wrote:
Hi users,
I need to add the time information (preference in hh and mn) extracted from an ascii file to a new variable (new_data from script), but I am not having success.
Attached is the script I am trying using the function cd_inv_calendar and the error obtained:
fatal:No coordinate variable exists for dimension (time) in variable (new_data)
I appreciate some help to attach this information to my new variable
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210114/da0c6c38/attachment.html>
More information about the ncl-talk
mailing list