[ncl-talk] A "bug" in reading comma separated files?

[Barry Lynn]

Hello:

Just to add a bit:

This page confirms what Mary wrote:

https://www.ncl.ucar.edu/Document/Functions/Built-in/str_get_field.shtml

"If one or more delimiters appear side-by-side in a string, then they are
treated as if they were just one delimiter. See the examples below for a
way to handle consecutive delimiters."

and offers a way around this (although I am not sure why one would need to
do this if you have str_get_csv

To make sure that the empty fields between the consecutive delimiters get
treated as real values, use *str_sub_str*
<https://www.ncl.ucar.edu/Document/Functions/Built-in/str_sub_str.shtml> to
insert a number between these delimiters.

On Fri, Aug 31, 2018 at 6:32 PM Mary Haley <haley at ucar.edu> wrote:

> Hi Barry,
>
> This is not a bug, but I do think it's an unfortunate reality of the way
> "str_get_field" works.
>
> What you want to use is str_split_csv instead. You read your lines as an
> arrays of strings using asciiread (as you're already doing), and then use
> str_split_csv which will split those lines into a 2D array based on a
> delimiter.  It will correctly treat fields with two commas right next to
> each other as missing values. You can then parse out the columns you need,
> and use tofloat or toint to convert them from strings to floats or ints.
>
> I've attached a sample script based on your data file.
>
> Here's a snippet:
>
>   asc_file = "weather_it_is_telav_20180831_052522.csv"
>   delim = ","
>   lines = asciiread(asc_file,-1,"string")
>   csv_lines = str_split_csv(lines(1:),delim,0)
>
>   Station                         = csv_lines(:,0)    ; read column 1
>
>   ObservationTimeUtc              = csv_lines(:,1)    ; read column 2
>
>   DewPointC                       = tofloat(csv_lines(:,2))    ; etc
>
> Here's the full output:
>
> Station                         = TELAV,TELAV
> ObservationTimeUtc              = 8/31/2018 5:05:12 AM,8/31/2018 5:00:06 AM
> DewPointC                       = 22.0824,22.2439
> HeatIndexC                      = 29.8813,29.959
> Humidity                        = 72.4898,73.2047
> PressureSeaLevelMBar            = 1011.68,1011.64
> PressureSeaLevelMBarRatePerHour = 0.230187,0.21248
> RainMillimetersDaily            = 0,0
> RainMillimetersRatePerHour      = 0,0
> TemperatureC                    = 27.4375,27.4375
> VisibilityKilometers            = 9.96921e+36,9.96921e+36
> WindChillC                      = 27.4375,27.4375
> WindSpeedKph                    = 3.52784,3.52784
> WindDirectionDegrees            = 0,304
> WindSpeedKphAvg                 = 4.58619,2.82227
> WindDirectionDegreesAvg         = 339,300
> WindGustKphHourly               = 9.87795,9.52517
> WindGustTimeUtcHourly           = missing,missing
> WindGustKphDaily                = 15.8753,15.8753
> WindGustTimeUtcDaily            = missing,missing
> WetBulbTemperatureC             = 23.6,23.7
> FeelsLikeTemperatureC           = 29.8813,29.959
> RainRateMaxMmPerHour            = 0,0
> TemperatureHighC                = 27.4656,27.4375
> TemperatureHighUtc              = 8/31/2018 5:01:00 AM,8/31/2018 5:00:00 AM
> TemperatureLowC                 = 26.4351,26.4351
> TemperatureLowUtc               = 8/31/2018 1:57:00 AM,8/31/2018 1:57:00 AM
>
>
>
>
> On Fri, Aug 31, 2018 at 5:27 AM, Barry Lynn <barry.h.lynn at gmail.com>
> wrote:
>
>> Please run this program in the following manner:
>>
>> ncl ./parse_tau_obs.ncl year1=18 month1=08 day1=31
>>
>>
>> What you'll find is that the program skips over the double comma. The
>> data that missing between the commas is the "Visibility." But, instead of
>> noting visibility as missing it just moves on to the next variable, writing
>> the next variable (WindChill) to Visibility.
>>
>> 0) TemperatureC = "TemperatureC"
>>
>> (1) TemperatureC = 27.4374727668845
>>
>> (2) TemperatureC = 27.4374727668845
>>
>> (0) VisibilityKilometers = "VisibilityKilometers"
>>
>> (1) VisibilityKilometers = 27.4374727668845
>>
>> (2) VisibilityKilometers = 27.4374727668845
>>
>> (0) WindChillC = "WindChillC"
>>
>> (1) WindChillC = 3.52784036047634
>>
>> (2) WindChillC = 3.52784036047634
>>
>> Thank you for your suggestions.
>>
>> Barry
>> --
>> Barry H. Lynn, Ph.D
>> Senior Associate Scientist, Lecturer,
>> The Institute of the Earth Science,
>> The Hebrew University of Jerusalem,
>> Givat Ram, Jerusalem 91904, Israel
>> Tel: 972 547 231 170
>> Fax: (972)-25662581
>>
>> C.E.O, Weather It Is, LTD
>> Weather and Climate Focus
>> http://weather-it-is.com
>> Jerusalem, Israel
>> Local: 02 930 9525
>> Cell: 054 7 231 170
>> Int-IS: x972 2 930 9525
>> Hello:
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>

-- 
Barry H. Lynn, Ph.D
Senior Associate Scientist, Lecturer,
The Institute of the Earth Science,
The Hebrew University of Jerusalem,
Givat Ram, Jerusalem 91904, Israel
Tel: 972 547 231 170
Fax: (972)-25662581

C.E.O, Weather It Is, LTD
Weather and Climate Focus
http://weather-it-is.com
Jerusalem, Israel
Local: 02 930 9525
Cell: 054 7 231 170
Int-IS: x972 2 930 9525
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180901/7023a655/attachment.html>