[Met_help] [rt.rap.ucar.edu #86677] History for Issue with ASCII2NC
John Halley Gotway via RT
met_help at ucar.edu
Wed Aug 29 09:25:39 MDT 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I'm using ASCII2NC to reformat my ASCAT ASCII data to NetCDF, to be further
used with POINT-STAT. I was using MET6 until mid to late June, when we
switched to MET7.
The command I use is simple:
ASCII2NC <input_file> <output_file> -v 2
Most of the time this tool works fine. However, there have quite a number
of instances where the NetCDF has extra lat/lon/ASCAT_obs that were not in
the ASCAT ASCII file.
For example,
For the time of 20180711_230000, the variables in the *mpr.txt file were:
OBS_LAT = 2.08
OBS_LON = 0.01
FCST = 7.84393
OBS = 69.13
I uploaded these files to incoming/irap/met_help/maccracken:
201807120023_mgdrlite.txt (the original ASCAT ASCII file, created by
reading the ASCAT binary file, and writing it out in MET format)
uv_wind_pb_2018071200_23.nc (The file created from MET ASCII2NC)
hdr_arr_dump.txt
obs_arr_dump.txt (Files created by using ncdump -v, of the header
(lat/lon, and observation value)
point_stat_mask_050000L_20180711_230000V_mpr.txt (The final matched pair
file produced from POINT_STAT)
Is this some type of bug in the way the data is written into the NetCDF
file? Originally, I thought it was the QC flags, but, it's clearly not,
since the lat/lon and observation was not in the file to begin with.
If this is a bug, is it corrected in MET7? Is there anything I can do
about the old data? What do you suggest?
Thanks!
Roz
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Issue with ASCII2NC
From: John Halley Gotway
Time: Mon Aug 20 19:21:07 2018
Roz,
I see that you found an apparent discrepancy in the point observations
produced by ascii2nc and point_stat. Thanks for sending your sample
data.
First, I ran the mgdrlite file through ascii2nc from both met-6.0 and
met-7.0. And the debug messages indicate the same number of
observations
and header locations:
/usr/local/met-6.0/bin/ascii2nc 201807120023_mgdrlite.txt
201807120023_mgdrlite_met-6.0.nc
DEBUG 2: Finished processing 100387 observations for 100372 headers.
/usr/local/met-7.0/bin/ascii2nc 201807120023_mgdrlite.txt
201807120023_mgdrlite_met-7.0.nc
DEBUG 2: Finished processing 100387 observations for 100372 headers.
So at least we can say that the output is consistent across versions.
Second, I ran the following Rscript to dump the NetCDF output file
back to
11-column ascii format for ascii2nc:
Rscript /usr/local/met-7.0/share/met/Rscripts/pntnc2ascii.R
201807120023_mgdrlite_met-7.0.nc > 201807120023_mgdrlite_met-7.0.txt
Third, I grepped for 69.13 and found the obs value you referenced:
grep ' 69.13 ' 201807120023_mgdrlite_met-7.0.txt
ASCAT SID 20180711_230900 2.08 0.01 0 32 0 10 N/A 69.13
>From this I deduce that Point-Stat isn't "creating" any observations.
It's
present in the output from ascii2nc. So let's check the input to
ascii2nc:
Fourth, grep for that obs in the input file:
cat 201807120023_mgdrlite.txt | nl | egrep ' 69.13$'
748 ASCAT SID 20180711_230900 2.08 0.01 N/A 32 N/A 10 N/A 69.13
Looks like that obs exists on line #748 of the input file. So I don't
see
any problem here at all.
Am I missing something, or do you agree that that obs shows up in the
Point-Stat MPR output file since it exists in the input to ascii2nc?
Thanks,
John
On Mon, Aug 20, 2018 at 2:17 PM Rosalyn MacCracken - NOAA Affiliate
via RT <
met_help at ucar.edu> wrote:
>
> Mon Aug 20 14:06:29 2018: Request 86677 was acted upon.
> Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> Queue: met_help
> Subject: Issue with ASCII2NC
> Owner: Nobody
> Requestors: rosalyn.maccracken at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86677 >
>
>
> Hi,
>
> I'm using ASCII2NC to reformat my ASCAT ASCII data to NetCDF, to be
further
> used with POINT-STAT. I was using MET6 until mid to late June, when
we
> switched to MET7.
>
> The command I use is simple:
> ASCII2NC <input_file> <output_file> -v 2
>
> Most of the time this tool works fine. However, there have quite a
number
> of instances where the NetCDF has extra lat/lon/ASCAT_obs that were
not in
> the ASCAT ASCII file.
>
> For example,
> For the time of 20180711_230000, the variables in the *mpr.txt file
were:
> OBS_LAT = 2.08
> OBS_LON = 0.01
> FCST = 7.84393
> OBS = 69.13
>
> I uploaded these files to incoming/irap/met_help/maccracken:
>
> 201807120023_mgdrlite.txt (the original ASCAT ASCII file, created
by
> reading the ASCAT binary file, and writing it out in MET format)
>
> uv_wind_pb_2018071200_23.nc (The file created from MET ASCII2NC)
> hdr_arr_dump.txt
> obs_arr_dump.txt (Files created by using ncdump -v, of the header
> (lat/lon, and observation value)
>
> point_stat_mask_050000L_20180711_230000V_mpr.txt (The final matched
pair
> file produced from POINT_STAT)
>
> Is this some type of bug in the way the data is written into the
NetCDF
> file? Originally, I thought it was the QC flags, but, it's clearly
not,
> since the lat/lon and observation was not in the file to begin with.
>
> If this is a bug, is it corrected in MET7? Is there anything I can
do
> about the old data? What do you suggest?
>
> Thanks!
>
> Roz
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Issue with ASCII2NC
From: Rosalyn MacCracken - NOAA Affiliate
Time: Tue Aug 21 06:44:53 2018
Hi John,
Ok, I must have missed that observation, since I was focused on
230000, and
not 230900. I can't check on that, because my VPN isn't working, and
I
can't get to my server, since there is something wrong at the Data
Center.
So, what I was worried about was that it wasn't in the original ASCAT
binary file, but, it shows up in the netcdf, and then in the MPR file.
As soon as I can get on my machine, I can make sure that I sent the
correct
file, and that we are looking at the same thing. I'll be back in
touch
hopefully sometime soon this morning.
Roz
On Mon, Aug 20, 2018 at 9:21 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Roz,
>
> I see that you found an apparent discrepancy in the point
observations
> produced by ascii2nc and point_stat. Thanks for sending your sample
data.
>
> First, I ran the mgdrlite file through ascii2nc from both met-6.0
and
> met-7.0. And the debug messages indicate the same number of
observations
> and header locations:
>
> /usr/local/met-6.0/bin/ascii2nc 201807120023_mgdrlite.txt
> 201807120023_mgdrlite_met-6.0.nc
>
> DEBUG 2: Finished processing 100387 observations for 100372 headers.
>
>
> /usr/local/met-7.0/bin/ascii2nc 201807120023_mgdrlite.txt
> 201807120023_mgdrlite_met-7.0.nc
>
> DEBUG 2: Finished processing 100387 observations for 100372 headers.
>
>
> So at least we can say that the output is consistent across
versions.
>
>
> Second, I ran the following Rscript to dump the NetCDF output file
back to
> 11-column ascii format for ascii2nc:
>
>
> Rscript /usr/local/met-7.0/share/met/Rscripts/pntnc2ascii.R
> 201807120023_mgdrlite_met-7.0.nc > 201807120023_mgdrlite_met-7.0.txt
>
>
> Third, I grepped for 69.13 and found the obs value you referenced:
>
> grep ' 69.13 ' 201807120023_mgdrlite_met-7.0.txt
>
>
> ASCAT SID 20180711_230900 2.08 0.01 0 32 0 10 N/A 69.13
>
>
> From this I deduce that Point-Stat isn't "creating" any
observations. It's
> present in the output from ascii2nc. So let's check the input to
ascii2nc:
>
>
> Fourth, grep for that obs in the input file:
>
> cat 201807120023_mgdrlite.txt | nl | egrep ' 69.13$'
>
> 748 ASCAT SID 20180711_230900 2.08 0.01 N/A 32 N/A 10 N/A 69.13
>
>
> Looks like that obs exists on line #748 of the input file. So I
don't see
> any problem here at all.
>
>
> Am I missing something, or do you agree that that obs shows up in
the
> Point-Stat MPR output file since it exists in the input to ascii2nc?
>
>
> Thanks,
> John
>
> On Mon, Aug 20, 2018 at 2:17 PM Rosalyn MacCracken - NOAA Affiliate
via RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Aug 20 14:06:29 2018: Request 86677 was acted upon.
> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> > Queue: met_help
> > Subject: Issue with ASCII2NC
> > Owner: Nobody
> > Requestors: rosalyn.maccracken at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86677 >
> >
> >
> > Hi,
> >
> > I'm using ASCII2NC to reformat my ASCAT ASCII data to NetCDF, to
be
> further
> > used with POINT-STAT. I was using MET6 until mid to late June,
when we
> > switched to MET7.
> >
> > The command I use is simple:
> > ASCII2NC <input_file> <output_file> -v 2
> >
> > Most of the time this tool works fine. However, there have quite
a
> number
> > of instances where the NetCDF has extra lat/lon/ASCAT_obs that
were not
> in
> > the ASCAT ASCII file.
> >
> > For example,
> > For the time of 20180711_230000, the variables in the *mpr.txt
file were:
> > OBS_LAT = 2.08
> > OBS_LON = 0.01
> > FCST = 7.84393
> > OBS = 69.13
> >
> > I uploaded these files to incoming/irap/met_help/maccracken:
> >
> > 201807120023_mgdrlite.txt (the original ASCAT ASCII file, created
by
> > reading the ASCAT binary file, and writing it out in MET format)
> >
> > uv_wind_pb_2018071200_23.nc (The file created from MET ASCII2NC)
> > hdr_arr_dump.txt
> > obs_arr_dump.txt (Files created by using ncdump -v, of the header
> > (lat/lon, and observation value)
> >
> > point_stat_mask_050000L_20180711_230000V_mpr.txt (The final
matched
> pair
> > file produced from POINT_STAT)
> >
> > Is this some type of bug in the way the data is written into the
NetCDF
> > file? Originally, I thought it was the QC flags, but, it's
clearly not,
> > since the lat/lon and observation was not in the file to begin
with.
> >
> > If this is a bug, is it corrected in MET7? Is there anything I
can do
> > about the old data? What do you suggest?
> >
> > Thanks!
> >
> > Roz
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Issue with ASCII2NC
From: Rosalyn MacCracken - NOAA Affiliate
Time: Tue Aug 28 13:39:18 2018
Hi John,
Yup, I'm wrong. I finally had a chance to get on to my test machine,
and
sure enough, it's in that file 201807120023.txt. I wasn't looking at
the
correct time. I was looking for the ob at 230000, not at 230900. So,
sorry that you wasted you time double checking that ob for me. I
thought I
had solved my mystery.
So, one last question before you can close this ticket. If for some
reason, these observations aren't associated with a QC flag,
indicating it
shouldn't be used, (QC=1 indicates it shouldn't be used), I can set an
upper limit of the wind speed for observation in the point-stat file,
so,
that the observation shouldn't be used over a certain wind speed,
correct?
Or, does that defeat the purpose of matching up obs to determine how
precise they are? I need to check on the flags for this file first,
to see
if it's let through QC.
Roz
On Tue, Aug 21, 2018 at 8:44 AM, Rosalyn MacCracken - NOAA Affiliate <
rosalyn.maccracken at noaa.gov> wrote:
> Hi John,
>
> Ok, I must have missed that observation, since I was focused on
230000,
> and not 230900. I can't check on that, because my VPN isn't
working, and I
> can't get to my server, since there is something wrong at the Data
Center.
> So, what I was worried about was that it wasn't in the original
ASCAT
> binary file, but, it shows up in the netcdf, and then in the MPR
file.
>
> As soon as I can get on my machine, I can make sure that I sent the
> correct file, and that we are looking at the same thing. I'll be
back in
> touch hopefully sometime soon this morning.
>
> Roz
>
> On Mon, Aug 20, 2018 at 9:21 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Roz,
>>
>> I see that you found an apparent discrepancy in the point
observations
>> produced by ascii2nc and point_stat. Thanks for sending your
sample data.
>>
>> First, I ran the mgdrlite file through ascii2nc from both met-6.0
and
>> met-7.0. And the debug messages indicate the same number of
observations
>> and header locations:
>>
>> /usr/local/met-6.0/bin/ascii2nc 201807120023_mgdrlite.txt
>> 201807120023_mgdrlite_met-6.0.nc
>>
>> DEBUG 2: Finished processing 100387 observations for 100372
headers.
>>
>>
>> /usr/local/met-7.0/bin/ascii2nc 201807120023_mgdrlite.txt
>> 201807120023_mgdrlite_met-7.0.nc
>>
>> DEBUG 2: Finished processing 100387 observations for 100372
headers.
>>
>>
>> So at least we can say that the output is consistent across
versions.
>>
>>
>> Second, I ran the following Rscript to dump the NetCDF output file
back to
>> 11-column ascii format for ascii2nc:
>>
>>
>> Rscript /usr/local/met-7.0/share/met/Rscripts/pntnc2ascii.R
>> 201807120023_mgdrlite_met-7.0.nc > 201807120023_mgdrlite_met-
7.0.txt
>>
>>
>> Third, I grepped for 69.13 and found the obs value you referenced:
>>
>> grep ' 69.13 ' 201807120023_mgdrlite_met-7.0.txt
>>
>>
>> ASCAT SID 20180711_230900 2.08 0.01 0 32 0 10 N/A 69.13
>>
>>
>> From this I deduce that Point-Stat isn't "creating" any
observations.
>> It's
>> present in the output from ascii2nc. So let's check the input to
>> ascii2nc:
>>
>>
>> Fourth, grep for that obs in the input file:
>>
>> cat 201807120023_mgdrlite.txt | nl | egrep ' 69.13$'
>>
>> 748 ASCAT SID 20180711_230900 2.08 0.01 N/A 32 N/A 10 N/A 69.13
>>
>>
>> Looks like that obs exists on line #748 of the input file. So I
don't see
>> any problem here at all.
>>
>>
>> Am I missing something, or do you agree that that obs shows up in
the
>> Point-Stat MPR output file since it exists in the input to
ascii2nc?
>>
>>
>> Thanks,
>> John
>>
>> On Mon, Aug 20, 2018 at 2:17 PM Rosalyn MacCracken - NOAA Affiliate
via
>> RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > Mon Aug 20 14:06:29 2018: Request 86677 was acted upon.
>> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
>> > Queue: met_help
>> > Subject: Issue with ASCII2NC
>> > Owner: Nobody
>> > Requestors: rosalyn.maccracken at noaa.gov
>> > Status: new
>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86677 >
>> >
>> >
>> > Hi,
>> >
>> > I'm using ASCII2NC to reformat my ASCAT ASCII data to NetCDF, to
be
>> further
>> > used with POINT-STAT. I was using MET6 until mid to late June,
when we
>> > switched to MET7.
>> >
>> > The command I use is simple:
>> > ASCII2NC <input_file> <output_file> -v 2
>> >
>> > Most of the time this tool works fine. However, there have quite
a
>> number
>> > of instances where the NetCDF has extra lat/lon/ASCAT_obs that
were not
>> in
>> > the ASCAT ASCII file.
>> >
>> > For example,
>> > For the time of 20180711_230000, the variables in the *mpr.txt
file
>> were:
>> > OBS_LAT = 2.08
>> > OBS_LON = 0.01
>> > FCST = 7.84393
>> > OBS = 69.13
>> >
>> > I uploaded these files to incoming/irap/met_help/maccracken:
>> >
>> > 201807120023_mgdrlite.txt (the original ASCAT ASCII file,
created by
>> > reading the ASCAT binary file, and writing it out in MET format)
>> >
>> > uv_wind_pb_2018071200_23.nc (The file created from MET ASCII2NC)
>> > hdr_arr_dump.txt
>> > obs_arr_dump.txt (Files created by using ncdump -v, of the
header
>> > (lat/lon, and observation value)
>> >
>> > point_stat_mask_050000L_20180711_230000V_mpr.txt (The final
matched
>> pair
>> > file produced from POINT_STAT)
>> >
>> > Is this some type of bug in the way the data is written into the
NetCDF
>> > file? Originally, I thought it was the QC flags, but, it's
clearly not,
>> > since the lat/lon and observation was not in the file to begin
with.
>> >
>> > If this is a bug, is it corrected in MET7? Is there anything I
can do
>> > about the old data? What do you suggest?
>> >
>> > Thanks!
>> >
>> > Roz
>> >
>> > --
>> > Rosalyn MacCracken
>> > Support Scientist
>> >
>> > Ocean Applications Branch
>> > NOAA/NWS Ocean Prediction Center
>> > NCWCP
>> > 5830 University Research Ct
>> > College Park, MD 20740-3818
>> >
>> > (p) 301-683-1551
>> > rosalyn.maccracken at noaa.gov
>> >
>> >
>>
>>
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Issue with ASCII2NC
From: Rosalyn MacCracken - NOAA Affiliate
Time: Tue Aug 28 15:34:15 2018
Hi John,
I did find the file that didn't contain that 69.13 wspd observation.
It
was created from archived ASCAT data, that we have in our archive.
What I
don't know is if these set of files are missing some observations that
the
original files contained. For example, in random directories
(designated
by day), only contain the 3 minute files for 23z, and on other days,
they
contain all the 3 minute files available for that day (00z - 23z).
So, I'm
not sure if the file with those bad observations are just missing out
of
the directory, or, if all the data is there, then why aren't those bad
observations in the files.
I'm not sure. If you want me to share that file with you, I can. I'm
not
really sure how to explain this...
Roz
On Tue, Aug 28, 2018 at 3:39 PM, Rosalyn MacCracken - NOAA Affiliate <
rosalyn.maccracken at noaa.gov> wrote:
> Hi John,
>
> Yup, I'm wrong. I finally had a chance to get on to my test
machine, and
> sure enough, it's in that file 201807120023.txt. I wasn't looking
at the
> correct time. I was looking for the ob at 230000, not at 230900.
So,
> sorry that you wasted you time double checking that ob for me. I
thought I
> had solved my mystery.
>
> So, one last question before you can close this ticket. If for some
> reason, these observations aren't associated with a QC flag,
indicating it
> shouldn't be used, (QC=1 indicates it shouldn't be used), I can set
an
> upper limit of the wind speed for observation in the point-stat
file, so,
> that the observation shouldn't be used over a certain wind speed,
correct?
> Or, does that defeat the purpose of matching up obs to determine how
> precise they are? I need to check on the flags for this file first,
to see
> if it's let through QC.
>
> Roz
>
> On Tue, Aug 21, 2018 at 8:44 AM, Rosalyn MacCracken - NOAA Affiliate
<
> rosalyn.maccracken at noaa.gov> wrote:
>
>> Hi John,
>>
>> Ok, I must have missed that observation, since I was focused on
230000,
>> and not 230900. I can't check on that, because my VPN isn't
working, and I
>> can't get to my server, since there is something wrong at the Data
Center.
>> So, what I was worried about was that it wasn't in the original
ASCAT
>> binary file, but, it shows up in the netcdf, and then in the MPR
file.
>>
>> As soon as I can get on my machine, I can make sure that I sent the
>> correct file, and that we are looking at the same thing. I'll be
back in
>> touch hopefully sometime soon this morning.
>>
>> Roz
>>
>> On Mon, Aug 20, 2018 at 9:21 PM, John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>>> Roz,
>>>
>>> I see that you found an apparent discrepancy in the point
observations
>>> produced by ascii2nc and point_stat. Thanks for sending your
sample
>>> data.
>>>
>>> First, I ran the mgdrlite file through ascii2nc from both met-6.0
and
>>> met-7.0. And the debug messages indicate the same number of
observations
>>> and header locations:
>>>
>>> /usr/local/met-6.0/bin/ascii2nc 201807120023_mgdrlite.txt
>>> 201807120023_mgdrlite_met-6.0.nc
>>>
>>> DEBUG 2: Finished processing 100387 observations for 100372
headers.
>>>
>>>
>>> /usr/local/met-7.0/bin/ascii2nc 201807120023_mgdrlite.txt
>>> 201807120023_mgdrlite_met-7.0.nc
>>>
>>> DEBUG 2: Finished processing 100387 observations for 100372
headers.
>>>
>>>
>>> So at least we can say that the output is consistent across
versions.
>>>
>>>
>>> Second, I ran the following Rscript to dump the NetCDF output file
back
>>> to
>>> 11-column ascii format for ascii2nc:
>>>
>>>
>>> Rscript /usr/local/met-7.0/share/met/Rscripts/pntnc2ascii.R
>>> 201807120023_mgdrlite_met-7.0.nc > 201807120023_mgdrlite_met-
7.0.txt
>>>
>>>
>>> Third, I grepped for 69.13 and found the obs value you referenced:
>>>
>>> grep ' 69.13 ' 201807120023_mgdrlite_met-7.0.txt
>>>
>>>
>>> ASCAT SID 20180711_230900 2.08 0.01 0 32 0 10 N/A 69.13
>>>
>>>
>>> From this I deduce that Point-Stat isn't "creating" any
observations.
>>> It's
>>> present in the output from ascii2nc. So let's check the input to
>>> ascii2nc:
>>>
>>>
>>> Fourth, grep for that obs in the input file:
>>>
>>> cat 201807120023_mgdrlite.txt | nl | egrep ' 69.13$'
>>>
>>> 748 ASCAT SID 20180711_230900 2.08 0.01 N/A 32 N/A 10 N/A 69.13
>>>
>>>
>>> Looks like that obs exists on line #748 of the input file. So I
don't
>>> see
>>> any problem here at all.
>>>
>>>
>>> Am I missing something, or do you agree that that obs shows up in
the
>>> Point-Stat MPR output file since it exists in the input to
ascii2nc?
>>>
>>>
>>> Thanks,
>>> John
>>>
>>> On Mon, Aug 20, 2018 at 2:17 PM Rosalyn MacCracken - NOAA
Affiliate via
>>> RT <
>>> met_help at ucar.edu> wrote:
>>>
>>> >
>>> > Mon Aug 20 14:06:29 2018: Request 86677 was acted upon.
>>> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
>>> > Queue: met_help
>>> > Subject: Issue with ASCII2NC
>>> > Owner: Nobody
>>> > Requestors: rosalyn.maccracken at noaa.gov
>>> > Status: new
>>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86677
>>> >
>>> >
>>> >
>>> > Hi,
>>> >
>>> > I'm using ASCII2NC to reformat my ASCAT ASCII data to NetCDF, to
be
>>> further
>>> > used with POINT-STAT. I was using MET6 until mid to late June,
when we
>>> > switched to MET7.
>>> >
>>> > The command I use is simple:
>>> > ASCII2NC <input_file> <output_file> -v 2
>>> >
>>> > Most of the time this tool works fine. However, there have
quite a
>>> number
>>> > of instances where the NetCDF has extra lat/lon/ASCAT_obs that
were
>>> not in
>>> > the ASCAT ASCII file.
>>> >
>>> > For example,
>>> > For the time of 20180711_230000, the variables in the *mpr.txt
file
>>> were:
>>> > OBS_LAT = 2.08
>>> > OBS_LON = 0.01
>>> > FCST = 7.84393
>>> > OBS = 69.13
>>> >
>>> > I uploaded these files to incoming/irap/met_help/maccracken:
>>> >
>>> > 201807120023_mgdrlite.txt (the original ASCAT ASCII file,
created by
>>> > reading the ASCAT binary file, and writing it out in MET format)
>>> >
>>> > uv_wind_pb_2018071200_23.nc (The file created from MET
ASCII2NC)
>>> > hdr_arr_dump.txt
>>> > obs_arr_dump.txt (Files created by using ncdump -v, of the
header
>>> > (lat/lon, and observation value)
>>> >
>>> > point_stat_mask_050000L_20180711_230000V_mpr.txt (The final
matched
>>> pair
>>> > file produced from POINT_STAT)
>>> >
>>> > Is this some type of bug in the way the data is written into the
NetCDF
>>> > file? Originally, I thought it was the QC flags, but, it's
clearly
>>> not,
>>> > since the lat/lon and observation was not in the file to begin
with.
>>> >
>>> > If this is a bug, is it corrected in MET7? Is there anything I
can do
>>> > about the old data? What do you suggest?
>>> >
>>> > Thanks!
>>> >
>>> > Roz
>>> >
>>> > --
>>> > Rosalyn MacCracken
>>> > Support Scientist
>>> >
>>> > Ocean Applications Branch
>>> > NOAA/NWS Ocean Prediction Center
>>> > NCWCP
>>> > 5830 University Research Ct
>>> > College Park, MD 20740-3818
>>> >
>>> > (p) 301-683-1551
>>> > rosalyn.maccracken at noaa.gov
>>> >
>>> >
>>>
>>>
>>
>>
>> --
>> Rosalyn MacCracken
>> Support Scientist
>>
>> Ocean Applications Branch
>> NOAA/NWS Ocean Prediction Center
>> NCWCP
>> 5830 University Research Ct
>> College Park, MD 20740-3818
>>
>> (p) 301-683-1551
>> rosalyn.maccracken at noaa.gov
>>
>
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Issue with ASCII2NC
From: John Halley Gotway
Time: Tue Aug 28 17:09:28 2018
Roz,
I'm having a difficult time following the details of the data flow you
describe. The theory was that ascii2nc was somehow creating spurious
observations in it's processing of the ASCAT data.
If you're able to isolate that problem and send an example input file
for
ascii2nc to illustrate this behavior, I'd be happy to debug. But it
sounds
like there's a lot of moving parts and the problem might reside in the
data
processing logic.
Let me know if you're able to isolate it in a single call to ascii2nc
that
I can run in the debugger.
Thanks,
John
On Tue, Aug 28, 2018 at 3:34 PM Rosalyn MacCracken - NOAA Affiliate
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86677 >
>
> Hi John,
>
> I did find the file that didn't contain that 69.13 wspd observation.
It
> was created from archived ASCAT data, that we have in our archive.
What I
> don't know is if these set of files are missing some observations
that the
> original files contained. For example, in random directories
(designated
> by day), only contain the 3 minute files for 23z, and on other days,
they
> contain all the 3 minute files available for that day (00z - 23z).
So, I'm
> not sure if the file with those bad observations are just missing
out of
> the directory, or, if all the data is there, then why aren't those
bad
> observations in the files.
>
> I'm not sure. If you want me to share that file with you, I can.
I'm not
> really sure how to explain this...
>
> Roz
>
> On Tue, Aug 28, 2018 at 3:39 PM, Rosalyn MacCracken - NOAA Affiliate
<
> rosalyn.maccracken at noaa.gov> wrote:
>
> > Hi John,
> >
> > Yup, I'm wrong. I finally had a chance to get on to my test
machine, and
> > sure enough, it's in that file 201807120023.txt. I wasn't looking
at the
> > correct time. I was looking for the ob at 230000, not at 230900.
So,
> > sorry that you wasted you time double checking that ob for me. I
> thought I
> > had solved my mystery.
> >
> > So, one last question before you can close this ticket. If for
some
> > reason, these observations aren't associated with a QC flag,
indicating
> it
> > shouldn't be used, (QC=1 indicates it shouldn't be used), I can
set an
> > upper limit of the wind speed for observation in the point-stat
file, so,
> > that the observation shouldn't be used over a certain wind speed,
> correct?
> > Or, does that defeat the purpose of matching up obs to determine
how
> > precise they are? I need to check on the flags for this file
first, to
> see
> > if it's let through QC.
> >
> > Roz
> >
> > On Tue, Aug 21, 2018 at 8:44 AM, Rosalyn MacCracken - NOAA
Affiliate <
> > rosalyn.maccracken at noaa.gov> wrote:
> >
> >> Hi John,
> >>
> >> Ok, I must have missed that observation, since I was focused on
230000,
> >> and not 230900. I can't check on that, because my VPN isn't
working,
> and I
> >> can't get to my server, since there is something wrong at the
Data
> Center.
> >> So, what I was worried about was that it wasn't in the original
ASCAT
> >> binary file, but, it shows up in the netcdf, and then in the MPR
file.
> >>
> >> As soon as I can get on my machine, I can make sure that I sent
the
> >> correct file, and that we are looking at the same thing. I'll be
back
> in
> >> touch hopefully sometime soon this morning.
> >>
> >> Roz
> >>
> >> On Mon, Aug 20, 2018 at 9:21 PM, John Halley Gotway via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >>> Roz,
> >>>
> >>> I see that you found an apparent discrepancy in the point
observations
> >>> produced by ascii2nc and point_stat. Thanks for sending your
sample
> >>> data.
> >>>
> >>> First, I ran the mgdrlite file through ascii2nc from both met-
6.0 and
> >>> met-7.0. And the debug messages indicate the same number of
> observations
> >>> and header locations:
> >>>
> >>> /usr/local/met-6.0/bin/ascii2nc 201807120023_mgdrlite.txt
> >>> 201807120023_mgdrlite_met-6.0.nc
> >>>
> >>> DEBUG 2: Finished processing 100387 observations for 100372
headers.
> >>>
> >>>
> >>> /usr/local/met-7.0/bin/ascii2nc 201807120023_mgdrlite.txt
> >>> 201807120023_mgdrlite_met-7.0.nc
> >>>
> >>> DEBUG 2: Finished processing 100387 observations for 100372
headers.
> >>>
> >>>
> >>> So at least we can say that the output is consistent across
versions.
> >>>
> >>>
> >>> Second, I ran the following Rscript to dump the NetCDF output
file back
> >>> to
> >>> 11-column ascii format for ascii2nc:
> >>>
> >>>
> >>> Rscript /usr/local/met-7.0/share/met/Rscripts/pntnc2ascii.R
> >>> 201807120023_mgdrlite_met-7.0.nc > 201807120023_mgdrlite_met-
7.0.txt
> >>>
> >>>
> >>> Third, I grepped for 69.13 and found the obs value you
referenced:
> >>>
> >>> grep ' 69.13 ' 201807120023_mgdrlite_met-7.0.txt
> >>>
> >>>
> >>> ASCAT SID 20180711_230900 2.08 0.01 0 32 0 10 N/A 69.13
> >>>
> >>>
> >>> From this I deduce that Point-Stat isn't "creating" any
observations.
> >>> It's
> >>> present in the output from ascii2nc. So let's check the input
to
> >>> ascii2nc:
> >>>
> >>>
> >>> Fourth, grep for that obs in the input file:
> >>>
> >>> cat 201807120023_mgdrlite.txt | nl | egrep ' 69.13$'
> >>>
> >>> 748 ASCAT SID 20180711_230900 2.08 0.01 N/A 32 N/A 10 N/A
69.13
> >>>
> >>>
> >>> Looks like that obs exists on line #748 of the input file. So I
don't
> >>> see
> >>> any problem here at all.
> >>>
> >>>
> >>> Am I missing something, or do you agree that that obs shows up
in the
> >>> Point-Stat MPR output file since it exists in the input to
ascii2nc?
> >>>
> >>>
> >>> Thanks,
> >>> John
> >>>
> >>> On Mon, Aug 20, 2018 at 2:17 PM Rosalyn MacCracken - NOAA
Affiliate via
> >>> RT <
> >>> met_help at ucar.edu> wrote:
> >>>
> >>> >
> >>> > Mon Aug 20 14:06:29 2018: Request 86677 was acted upon.
> >>> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> >>> > Queue: met_help
> >>> > Subject: Issue with ASCII2NC
> >>> > Owner: Nobody
> >>> > Requestors: rosalyn.maccracken at noaa.gov
> >>> > Status: new
> >>> > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86677
> >>> >
> >>> >
> >>> >
> >>> > Hi,
> >>> >
> >>> > I'm using ASCII2NC to reformat my ASCAT ASCII data to NetCDF,
to be
> >>> further
> >>> > used with POINT-STAT. I was using MET6 until mid to late
June, when
> we
> >>> > switched to MET7.
> >>> >
> >>> > The command I use is simple:
> >>> > ASCII2NC <input_file> <output_file> -v 2
> >>> >
> >>> > Most of the time this tool works fine. However, there have
quite a
> >>> number
> >>> > of instances where the NetCDF has extra lat/lon/ASCAT_obs that
were
> >>> not in
> >>> > the ASCAT ASCII file.
> >>> >
> >>> > For example,
> >>> > For the time of 20180711_230000, the variables in the *mpr.txt
file
> >>> were:
> >>> > OBS_LAT = 2.08
> >>> > OBS_LON = 0.01
> >>> > FCST = 7.84393
> >>> > OBS = 69.13
> >>> >
> >>> > I uploaded these files to incoming/irap/met_help/maccracken:
> >>> >
> >>> > 201807120023_mgdrlite.txt (the original ASCAT ASCII file,
created by
> >>> > reading the ASCAT binary file, and writing it out in MET
format)
> >>> >
> >>> > uv_wind_pb_2018071200_23.nc (The file created from MET
ASCII2NC)
> >>> > hdr_arr_dump.txt
> >>> > obs_arr_dump.txt (Files created by using ncdump -v, of the
header
> >>> > (lat/lon, and observation value)
> >>> >
> >>> > point_stat_mask_050000L_20180711_230000V_mpr.txt (The final
matched
> >>> pair
> >>> > file produced from POINT_STAT)
> >>> >
> >>> > Is this some type of bug in the way the data is written into
the
> NetCDF
> >>> > file? Originally, I thought it was the QC flags, but, it's
clearly
> >>> not,
> >>> > since the lat/lon and observation was not in the file to begin
with.
> >>> >
> >>> > If this is a bug, is it corrected in MET7? Is there anything
I can
> do
> >>> > about the old data? What do you suggest?
> >>> >
> >>> > Thanks!
> >>> >
> >>> > Roz
> >>> >
> >>> > --
> >>> > Rosalyn MacCracken
> >>> > Support Scientist
> >>> >
> >>> > Ocean Applications Branch
> >>> > NOAA/NWS Ocean Prediction Center
> >>> > NCWCP
> >>> > 5830 University Research Ct
> >>> > College Park, MD 20740-3818
> >>> >
> >>> > (p) 301-683-1551
> >>> > rosalyn.maccracken at noaa.gov
> >>> >
> >>> >
> >>>
> >>>
> >>
> >>
> >> --
> >> Rosalyn MacCracken
> >> Support Scientist
> >>
> >> Ocean Applications Branch
> >> NOAA/NWS Ocean Prediction Center
> >> NCWCP
> >> 5830 University Research Ct
> >> College Park, MD 20740-3818
> >>
> >> (p) 301-683-1551
> >> rosalyn.maccracken at noaa.gov
> >>
> >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
>
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Issue with ASCII2NC
From: Rosalyn MacCracken - NOAA Affiliate
Time: Wed Aug 29 07:40:48 2018
Hi John,
Sorry, I'm just frustrated because I can't recreate the error. I'm
thinking that this isn't an ascii2nc error after all, and it's just an
error with the ASCAT data that is making it past the QC flag. So, I'm
going to say, just forget the issue and close the ticket. I think
that
I'll have to find someone to tell me more about the behavior of ASCAT
and
why it is estimating very high wind speeds.
Roz
------------------------------------------------
More information about the Met_help
mailing list