[ncl-talk] A "bug" in reading comma separated files?

[Mary Haley]

Hi Barry,

This is not a bug, but I do think it's an unfortunate reality of the way
"str_get_field" works.

What you want to use is str_split_csv instead. You read your lines as an
arrays of strings using asciiread (as you're already doing), and then use
str_split_csv which will split those lines into a 2D array based on a
delimiter.  It will correctly treat fields with two commas right next to
each other as missing values. You can then parse out the columns you need,
and use tofloat or toint to convert them from strings to floats or ints.

I've attached a sample script based on your data file.

Here's a snippet:

  asc_file = "weather_it_is_telav_20180831_052522.csv"
  delim = ","
  lines = asciiread(asc_file,-1,"string")
  csv_lines = str_split_csv(lines(1:),delim,0)

  Station                         = csv_lines(:,0)    ; read column 1

  ObservationTimeUtc              = csv_lines(:,1)    ; read column 2

  DewPointC                       = tofloat(csv_lines(:,2))    ; etc

Here's the full output:

Station                         = TELAV,TELAV
ObservationTimeUtc              = 8/31/2018 5:05:12 AM,8/31/2018 5:00:06 AM
DewPointC                       = 22.0824,22.2439
HeatIndexC                      = 29.8813,29.959
Humidity                        = 72.4898,73.2047
PressureSeaLevelMBar            = 1011.68,1011.64
PressureSeaLevelMBarRatePerHour = 0.230187,0.21248
RainMillimetersDaily            = 0,0
RainMillimetersRatePerHour      = 0,0
TemperatureC                    = 27.4375,27.4375
VisibilityKilometers            = 9.96921e+36,9.96921e+36
WindChillC                      = 27.4375,27.4375
WindSpeedKph                    = 3.52784,3.52784
WindDirectionDegrees            = 0,304
WindSpeedKphAvg                 = 4.58619,2.82227
WindDirectionDegreesAvg         = 339,300
WindGustKphHourly               = 9.87795,9.52517
WindGustTimeUtcHourly           = missing,missing
WindGustKphDaily                = 15.8753,15.8753
WindGustTimeUtcDaily            = missing,missing
WetBulbTemperatureC             = 23.6,23.7
FeelsLikeTemperatureC           = 29.8813,29.959
RainRateMaxMmPerHour            = 0,0
TemperatureHighC                = 27.4656,27.4375
TemperatureHighUtc              = 8/31/2018 5:01:00 AM,8/31/2018 5:00:00 AM
TemperatureLowC                 = 26.4351,26.4351
TemperatureLowUtc               = 8/31/2018 1:57:00 AM,8/31/2018 1:57:00 AM




On Fri, Aug 31, 2018 at 5:27 AM, Barry Lynn <barry.h.lynn at gmail.com> wrote:

> Please run this program in the following manner:
>
> ncl ./parse_tau_obs.ncl year1=18 month1=08 day1=31
>
>
> What you'll find is that the program skips over the double comma. The data
> that missing between the commas is the "Visibility." But, instead of noting
> visibility as missing it just moves on to the next variable, writing the
> next variable (WindChill) to Visibility.
>
> 0) TemperatureC = "TemperatureC"
>
> (1) TemperatureC = 27.4374727668845
>
> (2) TemperatureC = 27.4374727668845
>
> (0) VisibilityKilometers = "VisibilityKilometers"
>
> (1) VisibilityKilometers = 27.4374727668845
>
> (2) VisibilityKilometers = 27.4374727668845
>
> (0) WindChillC = "WindChillC"
>
> (1) WindChillC = 3.52784036047634
>
> (2) WindChillC = 3.52784036047634
>
> Thank you for your suggestions.
>
> Barry
> --
> Barry H. Lynn, Ph.D
> Senior Associate Scientist, Lecturer,
> The Institute of the Earth Science,
> The Hebrew University of Jerusalem,
> Givat Ram, Jerusalem 91904, Israel
> Tel: 972 547 231 170
> Fax: (972)-25662581
>
> C.E.O, Weather It Is, LTD
> Weather and Climate Focus
> http://weather-it-is.com
> Jerusalem, Israel
> Local: 02 930 9525
> Cell: 054 7 231 170
> Int-IS: x972 2 930 9525
> Hello:
>
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180831/6981a3d4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: read_weather.ncl
Type: application/octet-stream
Size: 4016 bytes
Desc: not available
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180831/6981a3d4/attachment.obj>