[Met_help] [rt.rap.ucar.edu #67028] History for remove single observation
John Halley Gotway via RT
met_help at ucar.edu
Mon May 12 12:06:06 MDT 2014
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I was wondering if there was a simple way to remove one
station from an observation dataset. For example, when using point stat I
have one station with bad obs, but is not being filtered out with its
current qc flag. In point stat I could specify all the stations I want to
keep with "sid", but that is very tedious because I just want to remove one
station.
So, is there any way to remove one station, either in the
NETCDF observation file or via point stat?
Thanks,
Travis
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #67028] remove single observation
From: John Halley Gotway
Time: Fri May 09 12:24:24 2014
Travis,
There is no "slick" way of removing a single observation from the
analysis in Point-Stat. We have discussed enhancing the masking logic
in the past but haven't made any decisions on if/how to
proceed. So this is good feedback to hear that providing a way of
easily excluding stations would be helpful.
For now, I'd suggest editing the NetCDF observation file to remove
that station. You could do it in 2 steps:
(1) Run an Rscript to dump the NetCDF observation file to ascii,
excluding the station you want to throw out.
(2) Run the resulting ascii file through ascii2nc to prepare the data
for use in Point-Stat.
Here's an example using data generated by the test scripts included
with the tarball:
Rscript METv4.1/scripts/Rscripts/pntnc2ascii.R
METv4.1/out/pb2nc/sample_pb.nc | egrep -v "KDEN|\[|NULL" >
sample_pb_minus_KDEN.txt
METv4.1/bin/ascii2nc sample_pb_minus_KDEN.txt
sample_pb_minus_KDEN.nc
Here, I'm just excluding observations for Denver (KDEN), and the other
stuff is to get rid of the last few lines of output from that Rscript
that get printed from R. If you don't strip those lines
off, ascii2nc will complain.
Hope that helps get you going. And I'll think more about how to
provide this functionality more directly in point_stat.
Thanks,
John
On 05/09/2014 11:28 AM, Travis Wilson via RT wrote:
>
> Fri May 09 11:28:47 2014: Request 67028 was acted upon.
> Transaction: Ticket created by Wilson0028 at ucla.edu
> Queue: met_help
> Subject: remove single observation
> Owner: Nobody
> Requestors: Wilson0028 at ucla.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67028 >
>
>
> Hi,
> I was wondering if there was a simple way to remove
one
> station from an observation dataset. For example, when using point
stat I
> have one station with bad obs, but is not being filtered out with
its
> current qc flag. In point stat I could specify all the stations I
want to
> keep with "sid", but that is very tedious because I just want to
remove one
> station.
>
> So, is there any way to remove one station, either
in the
> NETCDF observation file or via point stat?
>
> Thanks,
> Travis
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #67028] remove single observation
From: Travis Wilson
Time: Fri May 09 13:58:31 2014
Hi John,
That command works great on the sample dataset in MET. However, it
is not working on the netcdf files I created with madis2nc. I get an
error when running ascii2nc
ERROR :
ERROR : timestring_to_sec(const char *) -> can't parse time string
"3600"
ERROR :
The error is specific to column 8 in the following lines
ADPSFC KUIL 20110105_044600 47.95 -124.55 62 33 -9999 -9999 V
-2.026499999999999968026
ADPSFC KUIL 20110105_044600 47.95 -124.55 62 34 -9999 -9999 V
-0.357329999999999980975
ADPSFC KUIL 20110105_044600 47.95 -124.55 62 61 3600 -9999 C
0.000000000000000000000
If I remove line 3 that contains the 3600 it works, else it will fail.
This must be a bug???? I have the latest patches on METv4.1.
Anyways, I really appreciate you helping me out. By the way, I will
be in Boulder come June for the WRF workshop. I'll be sure to stop by
your office to say hi (This is UCLA Travis, DTC visitor).
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, May 09, 2014 11:24 AM
To: Wilson0028 at ucla.edu
Subject: Re: [rt.rap.ucar.edu #67028] remove single observation
Travis,
There is no "slick" way of removing a single observation from the
analysis in Point-Stat. We have discussed enhancing the masking logic
in the past but haven't made any decisions on if/how to proceed. So
this is good feedback to hear that providing a way of easily excluding
stations would be helpful.
For now, I'd suggest editing the NetCDF observation file to remove
that station. You could do it in 2 steps:
(1) Run an Rscript to dump the NetCDF observation file to ascii,
excluding the station you want to throw out.
(2) Run the resulting ascii file through ascii2nc to prepare the data
for use in Point-Stat.
Here's an example using data generated by the test scripts included
with the tarball:
Rscript METv4.1/scripts/Rscripts/pntnc2ascii.R
METv4.1/out/pb2nc/sample_pb.nc | egrep -v "KDEN|\[|NULL" >
sample_pb_minus_KDEN.txt
METv4.1/bin/ascii2nc sample_pb_minus_KDEN.txt
sample_pb_minus_KDEN.nc
Here, I'm just excluding observations for Denver (KDEN), and the other
stuff is to get rid of the last few lines of output from that Rscript
that get printed from R. If you don't strip those lines off, ascii2nc
will complain.
Hope that helps get you going. And I'll think more about how to
provide this functionality more directly in point_stat.
Thanks,
John
On 05/09/2014 11:28 AM, Travis Wilson via RT wrote:
>
> Fri May 09 11:28:47 2014: Request 67028 was acted upon.
> Transaction: Ticket created by Wilson0028 at ucla.edu
> Queue: met_help
> Subject: remove single observation
> Owner: Nobody
> Requestors: Wilson0028 at ucla.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67028
> >
>
>
> Hi,
> I was wondering if there was a simple way to remove
> one station from an observation dataset. For example, when using
> point stat I have one station with bad obs, but is not being
filtered
> out with its current qc flag. In point stat I could specify all the
> stations I want to keep with "sid", but that is very tedious because
I
> just want to remove one station.
>
> So, is there any way to remove one station, either
in
> the NETCDF observation file or via point stat?
>
> Thanks,
> Travis
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #67028] remove single observation
From: John Halley Gotway
Time: Fri May 09 14:08:59 2014
Travis,
The problem is in the Rscript. The "level" for precipitation
observations should be the accumulation interval in HHMMSS format. In
the NetCDF file, it's being stored as a number of seconds, and that
Rscript is just writing out the number of seconds instead of a
formatted HHMMSS string. This didn't come up in the MET test data
because that doesn't contain any observations of precip - so no
accumulation intervals to deal with.
I'll work on updating that Rscript to write out a formatted HHMMSS
string for accumulation intervals and send you a new version to try.
Thanks,
John
On 05/09/2014 01:58 PM, Travis Wilson via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67028 >
>
> Hi John,
> That command works great on the sample dataset in MET. However, it
is not working on the netcdf files I created with madis2nc. I get an
error when running ascii2nc
>
> ERROR :
> ERROR : timestring_to_sec(const char *) -> can't parse time string
"3600"
> ERROR :
>
> The error is specific to column 8 in the following lines
>
> ADPSFC KUIL 20110105_044600 47.95 -124.55 62 33 -9999 -9999 V
-2.026499999999999968026
> ADPSFC KUIL 20110105_044600 47.95 -124.55 62 34 -9999 -9999 V
-0.357329999999999980975
> ADPSFC KUIL 20110105_044600 47.95 -124.55 62 61 3600 -9999 C
0.000000000000000000000
>
> If I remove line 3 that contains the 3600 it works, else it will
fail. This must be a bug???? I have the latest patches on METv4.1.
>
> Anyways, I really appreciate you helping me out. By the way, I will
be in Boulder come June for the WRF workshop. I'll be sure to stop by
your office to say hi (This is UCLA Travis, DTC visitor).
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, May 09, 2014 11:24 AM
> To: Wilson0028 at ucla.edu
> Subject: Re: [rt.rap.ucar.edu #67028] remove single observation
>
> Travis,
>
> There is no "slick" way of removing a single observation from the
analysis in Point-Stat. We have discussed enhancing the masking logic
in the past but haven't made any decisions on if/how to proceed. So
this is good feedback to hear that providing a way of easily excluding
stations would be helpful.
>
> For now, I'd suggest editing the NetCDF observation file to remove
that station. You could do it in 2 steps:
> (1) Run an Rscript to dump the NetCDF observation file to ascii,
excluding the station you want to throw out.
> (2) Run the resulting ascii file through ascii2nc to prepare the
data for use in Point-Stat.
>
> Here's an example using data generated by the test scripts included
with the tarball:
> Rscript METv4.1/scripts/Rscripts/pntnc2ascii.R
METv4.1/out/pb2nc/sample_pb.nc | egrep -v "KDEN|\[|NULL" >
sample_pb_minus_KDEN.txt
> METv4.1/bin/ascii2nc sample_pb_minus_KDEN.txt
sample_pb_minus_KDEN.nc
>
> Here, I'm just excluding observations for Denver (KDEN), and the
other stuff is to get rid of the last few lines of output from that
Rscript that get printed from R. If you don't strip those lines off,
ascii2nc will complain.
>
> Hope that helps get you going. And I'll think more about how to
provide this functionality more directly in point_stat.
>
> Thanks,
> John
>
>
>
> On 05/09/2014 11:28 AM, Travis Wilson via RT wrote:
>>
>> Fri May 09 11:28:47 2014: Request 67028 was acted upon.
>> Transaction: Ticket created by Wilson0028 at ucla.edu
>> Queue: met_help
>> Subject: remove single observation
>> Owner: Nobody
>> Requestors: Wilson0028 at ucla.edu
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67028
>>>
>>
>>
>> Hi,
>> I was wondering if there was a simple way to
remove
>> one station from an observation dataset. For example, when using
>> point stat I have one station with bad obs, but is not being
filtered
>> out with its current qc flag. In point stat I could specify all
the
>> stations I want to keep with "sid", but that is very tedious
because I
>> just want to remove one station.
>>
>> So, is there any way to remove one station,
either in
>> the NETCDF observation file or via point stat?
>>
>> Thanks,
>> Travis
>>
>
>
>
------------------------------------------------
Subject: remove single observation
From: John Halley Gotway
Time: Fri May 09 14:39:03 2014
Travis,
OK, please try the attached version of that Rscript. Here are the
lines I added:
# format level column to HHMMSS for accumulation intervals
ind_accum = ( dfSid$gc == 61 | dfSid$gc == 62 | dfSid$gc == 63 ) & (
dfSid$lvl != -9999 );
dfSid$lvl[ind_accum] = format(.POSIXct(dfSid$lvl[ind_accum],
tz="GMT"), "%H%M%S");
Please let me know if you run into any more problems.
Thanks,
John
On 05/09/2014 01:58 PM, Travis Wilson via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67028 >
>
> Hi John,
> That command works great on the sample dataset in MET. However, it
is not working on the netcdf files I created with madis2nc. I get an
error when running ascii2nc
>
> ERROR :
> ERROR : timestring_to_sec(const char *) -> can't parse time string
"3600"
> ERROR :
>
> The error is specific to column 8 in the following lines
>
> ADPSFC KUIL 20110105_044600 47.95 -124.55 62 33 -9999 -9999 V
-2.026499999999999968026
> ADPSFC KUIL 20110105_044600 47.95 -124.55 62 34 -9999 -9999 V
-0.357329999999999980975
> ADPSFC KUIL 20110105_044600 47.95 -124.55 62 61 3600 -9999 C
0.000000000000000000000
>
> If I remove line 3 that contains the 3600 it works, else it will
fail. This must be a bug???? I have the latest patches on METv4.1.
>
> Anyways, I really appreciate you helping me out. By the way, I will
be in Boulder come June for the WRF workshop. I'll be sure to stop by
your office to say hi (This is UCLA Travis, DTC visitor).
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, May 09, 2014 11:24 AM
> To: Wilson0028 at ucla.edu
> Subject: Re: [rt.rap.ucar.edu #67028] remove single observation
>
> Travis,
>
> There is no "slick" way of removing a single observation from the
analysis in Point-Stat. We have discussed enhancing the masking logic
in the past but haven't made any decisions on if/how to proceed. So
this is good feedback to hear that providing a way of easily excluding
stations would be helpful.
>
> For now, I'd suggest editing the NetCDF observation file to remove
that station. You could do it in 2 steps:
> (1) Run an Rscript to dump the NetCDF observation file to ascii,
excluding the station you want to throw out.
> (2) Run the resulting ascii file through ascii2nc to prepare the
data for use in Point-Stat.
>
> Here's an example using data generated by the test scripts included
with the tarball:
> Rscript METv4.1/scripts/Rscripts/pntnc2ascii.R
METv4.1/out/pb2nc/sample_pb.nc | egrep -v "KDEN|\[|NULL" >
sample_pb_minus_KDEN.txt
> METv4.1/bin/ascii2nc sample_pb_minus_KDEN.txt
sample_pb_minus_KDEN.nc
>
> Here, I'm just excluding observations for Denver (KDEN), and the
other stuff is to get rid of the last few lines of output from that
Rscript that get printed from R. If you don't strip those lines off,
ascii2nc will complain.
>
> Hope that helps get you going. And I'll think more about how to
provide this functionality more directly in point_stat.
>
> Thanks,
> John
>
>
>
> On 05/09/2014 11:28 AM, Travis Wilson via RT wrote:
>>
>> Fri May 09 11:28:47 2014: Request 67028 was acted upon.
>> Transaction: Ticket created by Wilson0028 at ucla.edu
>> Queue: met_help
>> Subject: remove single observation
>> Owner: Nobody
>> Requestors: Wilson0028 at ucla.edu
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67028
>>>
>>
>>
>> Hi,
>> I was wondering if there was a simple way to
remove
>> one station from an observation dataset. For example, when using
>> point stat I have one station with bad obs, but is not being
filtered
>> out with its current qc flag. In point stat I could specify all
the
>> stations I want to keep with "sid", but that is very tedious
because I
>> just want to remove one station.
>>
>> So, is there any way to remove one station,
either in
>> the NETCDF observation file or via point stat?
>>
>> Thanks,
>> Travis
>>
>
>
>
------------------------------------------------
Subject: remove single observation
From: John Halley Gotway
Time: Fri May 09 14:39:03 2014
library(ncdf);
usage = function(){
cat("\nUsage: pntnc2ascii\n",
" [-sid name]\n",
" [-msg_type name]\n",
" [-code value]\n",
" nc_file\n\n",
" where \"-sid name\" prints only observations for that
station id (optional).\n",
" \"-msg_typ name \" prints only observations for
that message type (optional).\n",
" \"-code value\" prints only observations for that
GRIB code (optional).\n",
" \"nc_file\" is the netCDF point observation file
(required).\n\n",
sep="");
}
# parse and verify command line arguments
strSid = ""; strMsg = ""; strGc = "";
listArgs = commandArgs(TRUE);
if( 1 > length(listArgs) ){ usage(); q(); }
if( 1 < length(listArgs) ){
for( i in seq(1, length(listArgs) - 1, by=2) ){
if ( "-sid" == listArgs[i] ){
strSid = listArgs[i+1];
} else if( "-msg_typ" == listArgs[i] ){
strMsg = listArgs[i+1];
} else if( "-code" == listArgs[i] ){
strGc = listArgs[i+1];
} else {
cat("ERROR: unrecognized input argument '", listArgs[i], "'\n",
sep="");
usage(); q(status=1);
}
}
}
# open the input NetCDF file
strPntNc = listArgs[ length(listArgs) ];
ncPnt = open.ncdf(c(strPntNc), write=F);
# build the header data frame
dfHdr = data.frame(
typ = get.var.ncdf(ncPnt, ncPnt$var[["hdr_typ"]]),
sid = get.var.ncdf(ncPnt, ncPnt$var[["hdr_sid"]]),
vld = get.var.ncdf(ncPnt, ncPnt$var[["hdr_vld"]]),
lat = get.var.ncdf(ncPnt, ncPnt$var[["hdr_arr"]])[1,],
lon = get.var.ncdf(ncPnt, ncPnt$var[["hdr_arr"]])[2,],
elv = get.var.ncdf(ncPnt, ncPnt$var[["hdr_arr"]])[3,]
);
dfHdr$hdr_id = seq(0,nrow(dfHdr)-1);
# replace empty SID strings with NA
dfHdr$sid[dfHdr$sid == ""] = NA;
# check for the obs_qty variable
if( 0 < length(ncPnt$var[["obs_qty"]]) ) {
ObsQty = get.var.ncdf(ncPnt, ncPnt$var[["obs_qty"]]);
} else {
ObsQty = rep(-9999, ncPnt$dim[["nobs"]]$len);
}
# build the observation data frame
dfObs = data.frame(
hdr_id = get.var.ncdf(ncPnt, ncPnt$var[["obs_arr"]])[1,],
gc = get.var.ncdf(ncPnt, ncPnt$var[["obs_arr"]])[2,],
lvl = get.var.ncdf(ncPnt, ncPnt$var[["obs_arr"]])[3,],
hgt = get.var.ncdf(ncPnt, ncPnt$var[["obs_arr"]])[4,],
qty = ObsQty,
ob = signif(get.var.ncdf(ncPnt, ncPnt$var[["obs_arr"]])[5,], 5)
);
# apply the filtering criteria and merge the headers and obs
if( strSid != "" ){ dfHdr = dfHdr[dfHdr$sid == strSid,]; }
if( strMsg != "" ){ dfHdr = dfHdr[dfHdr$typ == strMsg,]; }
if( strGc != "" ){ dfObs = dfObs[dfObs$gc == strGc, ]; }
dfSid = merge(dfHdr, dfObs);
# format level column to HHMMSS for accumulation intervals
ind_accum = ( dfSid$gc == 61 | dfSid$gc == 62 | dfSid$gc == 63 ) & (
dfSid$lvl != -9999 );
dfSid$lvl[ind_accum] = format(.POSIXct(dfSid$lvl[ind_accum],
tz="GMT"), "%H%M%S");
# print the formatted observations
options(scipen=20, digits=6);
apply(dfSid[,2:12], 1, function(d){ cat(d,"\n") })
# better formatting, but very slow...
#for(i in 1:nrow(dfSid)){
# cat( sprintf("%-8s%-8s%-15s%11.5f%11.5f%6d%5d%10.2f%14.5f%12.5f\n",
# dfSid[i,]$typ,
# dfSid[i,]$sid,
# dfSid[i,]$vld,
# dfSid[i,]$lat,
# dfSid[i,]$lon,
# dfSid[i,]$elv,
# dfSid[i,]$gc,
# dfSid[i,]$lvl,
# dfSid[i,]$hgt,
# dfSid[i,]$ob
# ), sep="");
#}
# close the input NetCDF file
close.ncdf(ncPnt);
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #67028] remove single observation
From: Travis Wilson
Time: Fri May 09 15:31:46 2014
Thanks John, it works great. You are the man!
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, May 09, 2014 1:39 PM
To: Wilson0028 at ucla.edu
Subject: Re: [rt.rap.ucar.edu #67028] remove single observation
Travis,
OK, please try the attached version of that Rscript. Here are the
lines I added:
# format level column to HHMMSS for accumulation intervals ind_accum =
( dfSid$gc == 61 | dfSid$gc == 62 | dfSid$gc == 63 ) & ( dfSid$lvl !=
-9999 ); dfSid$lvl[ind_accum] = format(.POSIXct(dfSid$lvl[ind_accum],
tz="GMT"), "%H%M%S");
Please let me know if you run into any more problems.
Thanks,
John
On 05/09/2014 01:58 PM, Travis Wilson via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67028 >
>
> Hi John,
> That command works great on the sample dataset in MET. However, it
> is not working on the netcdf files I created with madis2nc. I get
an
> error when running ascii2nc
>
> ERROR :
> ERROR : timestring_to_sec(const char *) -> can't parse time string
"3600"
> ERROR :
>
> The error is specific to column 8 in the following lines
>
> ADPSFC KUIL 20110105_044600 47.95 -124.55 62 33 -9999 -9999 V
-2.026499999999999968026
> ADPSFC KUIL 20110105_044600 47.95 -124.55 62 34 -9999 -9999 V
-0.357329999999999980975
> ADPSFC KUIL 20110105_044600 47.95 -124.55 62 61 3600 -9999 C
0.000000000000000000000
>
> If I remove line 3 that contains the 3600 it works, else it will
fail. This must be a bug???? I have the latest patches on METv4.1.
>
> Anyways, I really appreciate you helping me out. By the way, I will
be in Boulder come June for the WRF workshop. I'll be sure to stop by
your office to say hi (This is UCLA Travis, DTC visitor).
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, May 09, 2014 11:24 AM
> To: Wilson0028 at ucla.edu
> Subject: Re: [rt.rap.ucar.edu #67028] remove single observation
>
> Travis,
>
> There is no "slick" way of removing a single observation from the
analysis in Point-Stat. We have discussed enhancing the masking logic
in the past but haven't made any decisions on if/how to proceed. So
this is good feedback to hear that providing a way of easily excluding
stations would be helpful.
>
> For now, I'd suggest editing the NetCDF observation file to remove
that station. You could do it in 2 steps:
> (1) Run an Rscript to dump the NetCDF observation file to ascii,
excluding the station you want to throw out.
> (2) Run the resulting ascii file through ascii2nc to prepare the
data for use in Point-Stat.
>
> Here's an example using data generated by the test scripts included
with the tarball:
> Rscript METv4.1/scripts/Rscripts/pntnc2ascii.R
METv4.1/out/pb2nc/sample_pb.nc | egrep -v "KDEN|\[|NULL" >
sample_pb_minus_KDEN.txt
> METv4.1/bin/ascii2nc sample_pb_minus_KDEN.txt
> sample_pb_minus_KDEN.nc
>
> Here, I'm just excluding observations for Denver (KDEN), and the
other stuff is to get rid of the last few lines of output from that
Rscript that get printed from R. If you don't strip those lines off,
ascii2nc will complain.
>
> Hope that helps get you going. And I'll think more about how to
provide this functionality more directly in point_stat.
>
> Thanks,
> John
>
>
>
> On 05/09/2014 11:28 AM, Travis Wilson via RT wrote:
>>
>> Fri May 09 11:28:47 2014: Request 67028 was acted upon.
>> Transaction: Ticket created by Wilson0028 at ucla.edu
>> Queue: met_help
>> Subject: remove single observation
>> Owner: Nobody
>> Requestors: Wilson0028 at ucla.edu
>> Status: new
>> Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67028
>>>
>>
>>
>> Hi,
>> I was wondering if there was a simple way to
remove
>> one station from an observation dataset. For example, when using
>> point stat I have one station with bad obs, but is not being
filtered
>> out with its current qc flag. In point stat I could specify all
the
>> stations I want to keep with "sid", but that is very tedious
because
>> I just want to remove one station.
>>
>> So, is there any way to remove one station,
either
>> in the NETCDF observation file or via point stat?
>>
>> Thanks,
>> Travis
>>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list