[Met_help] [rt.rap.ucar.edu #99278] History for Help on processing observation data with little-r format
John Halley Gotway via RT
met_help at ucar.edu
Mon Jul 12 11:23:32 MDT 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
Would you please help me? I'm stuck on processing the obs data using "MET
ASCII2NC".
I need 12 hour obs data to verify WRF and MPAS 12hr forecasts. I tried
twice:
1. First time I used the global little-r format data, it failed. see below
for the error message. This data puts 6 hour obs data in one file and
located in /glade/collections/rda/data/ds351.0/little_r/2015/OBS:*
2. The 2nd time I used hourly local little-r format data which got from the
global data using WRFDA (The data are in
/glade/u/home/taosun/work/data/2015case/gts/).
It works fine but still has problems when using "PointStat" with this
processed obs data. see log files
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210329101301
and
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210329102722
The ASCII2NC2.conf and ASCII2NC3.conf are in /gpfs/fs1/work/zhuming/METplus4
One More question: How to set time for processing 12 hour obs data?
Thanks Thanks!
Zhuming
1.
/gpfs/fs1/work/zhuming/METplus4>master_metplus.py -c ./ASCII2NC2.conf
./metplus_system.conf
Running METplus 4.0.0-beta3
Starting METplus v4.0.0-beta3
03/26 12:35:27.961 metplus (config_metplus.py:121) INFO: Starting METplus
configuration setup.
03/26 12:35:27.963 metplus (config_metplus.py:249) INFO:
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_system.conf:
Parsed this file
03/26 12:35:27.964 metplus (config_metplus.py:249) INFO:
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_data.conf:
Parsed this file
03/26 12:35:27.964 metplus (config_metplus.py:249) INFO:
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_runtime.conf:
Parsed this file
03/26 12:35:27.964 metplus (config_metplus.py:249) INFO:
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_logging.conf:
Parsed this file
03/26 12:35:27.965 metplus (config_metplus.py:249) INFO: ./ASCII2NC2.conf:
Parsed this file
03/26 12:35:27.966 metplus (config_metplus.py:249) INFO:
./metplus_system.conf: Parsed this file
03/26 12:35:27.972 metplus (config_metplus.py:438) INFO: Adding:
config.LOG_TIMESTAMP='20210326123527'
03/26 12:35:27.972 metplus (config_metplus.py:444) INFO: Replace [config]
LOG_METPLUS with
'/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210326123527'
03/26 12:35:27.976 metplus (config_metplus.py:316) INFO: Setting [dir]
METPLUS_BASE to /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-beta3
03/26 12:35:27.976 metplus (config_metplus.py:316) INFO: Setting [dir]
PARM_BASE to /gpfs/fs1/work/zhuming/METplus4/parm
03/26 12:35:27.976 metplus (config_metplus.py:323) INFO: METPLUS_CONF:
/glade/scratch/zhuming/metplus4/metplus_final.conf written here.
03/26 12:35:27.977 metplus (config_metplus.py:141) INFO: Completed METplus
configuration setup.
03/26 12:35:27.977 metplus (met_util.py:101) INFO: Running METplus
v4.0.0-beta3 called with command:
/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-beta3/ush/master_metplus.py -c
./ASCII2NC2.conf ./metplus_system.conf
03/26 12:35:27.977 metplus (met_util.py:103) INFO: Log file:
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210326123527
03/26 12:35:28.864 metplus (met_util.py:1022) INFO:
****************************************
03/26 12:35:28.864 metplus (met_util.py:1023) INFO: * Running METplus
03/26 12:35:28.864 metplus (met_util.py:1027) INFO: * at valid time:
2015-07-15 00:00
03/26 12:35:28.864 metplus (met_util.py:1028) INFO:
****************************************
03/26 12:35:28.867 metplus (command_runner.py:90) INFO: COMMAND:
/glade/p/ral/jntp/MET/MET_releases/10.0.0-beta3/bin/ascii2nc
/glade/collections/rda/data/ds351.0/little_r/2015/OBS:2015071500
/glade/scratch/zhuming/metplus4/ascii2nc2/obs.2015071500.nc -format
"little_r" -config
/gpfs/fs1/work/zhuming/METplus4/parm/met_config/Ascii2NcConfig_wrapped -v 2
03/26 12:35:29.433 metplus (command_builder.py:166) ERROR:
(command_builder.py:1151) MET command returned a non-zero return
code:/glade/p/ral/jntp/MET/MET_releases/10.0.0-beta3/bin/ascii2nc
/glade/collections/rda/data/ds351.0/little_r/2015/OBS:2015071500
/glade/scratch/zhuming/metplus4/ascii2nc2/obs.2015071500.nc -format
"little_r" -config
/gpfs/fs1/work/zhuming/METplus4/parm/met_config/Ascii2NcConfig_wrapped -v 2
03/26 12:35:29Z run-METplus-metplus: ERROR: (command_builder.py:1151) MET
command returned a non-zero return
code:/glade/p/ral/jntp/MET/MET_releases/10.0.0-beta3/bin/ascii2nc
/glade/collections/rda/data/ds351.0/little_r/2015/OBS:2015071500
/glade/scratch/zhuming/metplus4/ascii2nc2/obs.2015071500.nc -format
"little_r" -config
/gpfs/fs1/work/zhuming/METplus4/parm/met_config/Ascii2NcConfig_wrapped -v 2
03/26 12:35:29.433 metplus (command_builder.py:1153) INFO: Check the
logfile for more information on why it failed:
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210326123527
03/26 12:35:29.434 metplus.ASCII2NC (met_util.py:214) ERROR: ASCII2NC had 1
error.
03/26 12:35:29Z run-METplus-metplus.ASCII2NC: ERROR: ASCII2NC had 1 error.
03/26 12:35:29.434 metplus (met_util.py:883) INFO:
/glade/scratch/zhuming/metplus4/metplus_final.conf: write metplus.conf here
03/26 12:35:29.436 metplus (met_util.py:251) ERROR: METplus has finished
running but had 1 error.
03/26 12:35:29Z run-METplus-metplus: ERROR: METplus has finished running
but had 1 error.
03/26 12:35:29.436 metplus (met_util.py:252) INFO: Check the log file for
more information:
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210326123527
cheyenne5:/gpfs/fs1/work/zhuming/METplus4>
2.
cheyenne5:/gpfs/fs1/work/zhuming/METplus4>master_metplus.py -c
./ASCII2NC3.conf ./metplus_system.conf
Running METplus 4.0.0-beta3
Starting METplus v4.0.0-beta3
03/29 10:13:01.465 metplus (config_metplus.py:121) INFO: Starting METplus
configuration setup.
03/29 10:13:01.475 metplus (config_metplus.py:249) INFO:
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_system.conf:
Parsed this file
03/29 10:13:01.475 metplus (config_metplus.py:249) INFO:
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_data.conf:
Parsed this file
03/29 10:13:01.476 metplus (config_metplus.py:249) INFO:
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_runtime.conf:
Parsed this file
03/29 10:13:01.490 metplus (config_metplus.py:249) INFO:
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_logging.conf:
Parsed this file
03/29 10:13:01.514 metplus (config_metplus.py:249) INFO: ./ASCII2NC3.conf:
Parsed this file
03/29 10:13:01.515 metplus (config_metplus.py:249) INFO:
./metplus_system.conf: Parsed this file
03/29 10:13:01.521 metplus (config_metplus.py:438) INFO: Adding:
config.LOG_TIMESTAMP='20210329101301'
03/29 10:13:01.521 metplus (config_metplus.py:444) INFO: Replace [config]
LOG_METPLUS with
'/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210329101301'
03/29 10:13:01.524 metplus (config_metplus.py:316) INFO: Setting [dir]
METPLUS_BASE to /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-beta3
03/29 10:13:01.524 metplus (config_metplus.py:316) INFO: Setting [dir]
PARM_BASE to /gpfs/fs1/work/zhuming/METplus4/parm
03/29 10:13:01.524 metplus (config_metplus.py:323) INFO: METPLUS_CONF:
/glade/scratch/zhuming/metplus4/metplus_final.conf written here.
03/29 10:13:01.526 metplus (config_metplus.py:141) INFO: Completed METplus
configuration setup.
03/29 10:13:01.526 metplus (met_util.py:101) INFO: Running METplus
v4.0.0-beta3 called with command:
/glade/p/ral/jntp/MET/METplus/METplus-4.0.0-beta3/ush/master_metplus.py -c
./ASCII2NC3.conf ./metplus_system.conf
03/29 10:13:01.526 metplus (met_util.py:103) INFO: Log file:
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210329101301
03/29 10:13:04.746 metplus (met_util.py:1022) INFO:
****************************************
03/29 10:13:04.747 metplus (met_util.py:1023) INFO: * Running METplus
03/29 10:13:04.747 metplus (met_util.py:1027) INFO: * at valid time:
2015-07-15 00:00
03/29 10:13:04.747 metplus (met_util.py:1028) INFO:
****************************************
03/29 10:13:04.758 metplus (command_runner.py:90) INFO: COMMAND:
/glade/p/ral/jntp/MET/MET_releases/10.0.0-beta3/bin/ascii2nc
/glade/u/home/taosun/work/data/2015case/gts/obs.2015071500
/glade/scratch/zhuming/metplus4/ascii2nc3/obs.2015071500.nc -format
"little_r" -config
/gpfs/fs1/work/zhuming/METplus4/parm/met_config/Ascii2NcConfig_wrapped -v 2
03/29 10:14:17.268 metplus (met_util.py:883) INFO:
/glade/scratch/zhuming/metplus4/metplus_final.conf: write metplus.conf here
03/29 10:14:17.274 metplus (met_util.py:244) INFO: Check the log file for
more information:
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210329101301
03/29 10:14:17.301 metplus (met_util.py:245) INFO: METplus has successfully
finished running.
cheyenne5:/gpfs/fs1/work/zhuming/METplus4>
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Help on processing observation data with little-r format
From: John Halley Gotway
Time: Mon Mar 29 12:17:50 2021
Zhuming,
Sorry for the delay in making progress on this issue. I was out of the
office last week was coordinating a bit with George on the issues
you've
run into.
I'll log on to cheyenne now to see why there are issues with ascii2nc
processing the little-r formatted data.
Thanks,
John
On Mon, Mar 29, 2021 at 10:37 AM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> Mon Mar 29 10:36:26 2021: Request 99278 was acted upon.
> Transaction: Ticket created by zhuming at ucar.edu
> Queue: met_help
> Subject: Help on processing observation data with little-r
format
> Owner: Nobody
> Requestors: zhuming at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99278 >
>
>
> Hello,
>
> Would you please help me? I'm stuck on processing the obs data using
"MET
> ASCII2NC".
>
> I need 12 hour obs data to verify WRF and MPAS 12hr forecasts. I
tried
> twice:
>
> 1. First time I used the global little-r format data, it failed.
see below
> for the error message. This data puts 6 hour obs data in one file
and
> located in /glade/collections/rda/data/ds351.0/little_r/2015/OBS:*
>
> 2. The 2nd time I used hourly local little-r format data which got
from the
> global data using WRFDA (The data are in
> /glade/u/home/taosun/work/data/2015case/gts/).
> It works fine but still has problems when using "PointStat" with
this
> processed obs data. see log files
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210329101301
> and
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210329102722
>
> The ASCII2NC2.conf and ASCII2NC3.conf are in
> /gpfs/fs1/work/zhuming/METplus4
>
> One More question: How to set time for processing 12 hour obs data?
>
> Thanks Thanks!
> Zhuming
>
>
> 1.
> /gpfs/fs1/work/zhuming/METplus4>master_metplus.py -c
./ASCII2NC2.conf
> ./metplus_system.conf
>
> Running METplus 4.0.0-beta3
>
> Starting METplus v4.0.0-beta3
>
> 03/26 12:35:27.961 metplus (config_metplus.py:121) INFO: Starting
METplus
> configuration setup.
>
> 03/26 12:35:27.963 metplus (config_metplus.py:249) INFO:
>
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_system.conf:
> Parsed this file
>
> 03/26 12:35:27.964 metplus (config_metplus.py:249) INFO:
>
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_data.conf:
> Parsed this file
>
> 03/26 12:35:27.964 metplus (config_metplus.py:249) INFO:
>
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_runtime.conf:
> Parsed this file
>
> 03/26 12:35:27.964 metplus (config_metplus.py:249) INFO:
>
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_logging.conf:
> Parsed this file
>
> 03/26 12:35:27.965 metplus (config_metplus.py:249) INFO:
./ASCII2NC2.conf:
> Parsed this file
>
> 03/26 12:35:27.966 metplus (config_metplus.py:249) INFO:
> ./metplus_system.conf: Parsed this file
>
> 03/26 12:35:27.972 metplus (config_metplus.py:438) INFO: Adding:
> config.LOG_TIMESTAMP='20210326123527'
>
> 03/26 12:35:27.972 metplus (config_metplus.py:444) INFO: Replace
[config]
> LOG_METPLUS with
>
'/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210326123527'
>
> 03/26 12:35:27.976 metplus (config_metplus.py:316) INFO: Setting
[dir]
> METPLUS_BASE to /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-beta3
>
> 03/26 12:35:27.976 metplus (config_metplus.py:316) INFO: Setting
[dir]
> PARM_BASE to /gpfs/fs1/work/zhuming/METplus4/parm
>
> 03/26 12:35:27.976 metplus (config_metplus.py:323) INFO:
METPLUS_CONF:
> /glade/scratch/zhuming/metplus4/metplus_final.conf written here.
>
> 03/26 12:35:27.977 metplus (config_metplus.py:141) INFO: Completed
METplus
> configuration setup.
>
> 03/26 12:35:27.977 metplus (met_util.py:101) INFO: Running METplus
> v4.0.0-beta3 called with command:
> /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta3/ush/master_metplus.py -c
> ./ASCII2NC2.conf ./metplus_system.conf
>
> 03/26 12:35:27.977 metplus (met_util.py:103) INFO: Log file:
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210326123527
>
> 03/26 12:35:28.864 metplus (met_util.py:1022) INFO:
> ****************************************
>
> 03/26 12:35:28.864 metplus (met_util.py:1023) INFO: * Running
METplus
>
> 03/26 12:35:28.864 metplus (met_util.py:1027) INFO: * at valid
time:
> 2015-07-15 00:00
>
> 03/26 12:35:28.864 metplus (met_util.py:1028) INFO:
> ****************************************
>
> 03/26 12:35:28.867 metplus (command_runner.py:90) INFO: COMMAND:
> /glade/p/ral/jntp/MET/MET_releases/10.0.0-beta3/bin/ascii2nc
> /glade/collections/rda/data/ds351.0/little_r/2015/OBS:2015071500
> /glade/scratch/zhuming/metplus4/ascii2nc2/obs.2015071500.nc -format
> "little_r" -config
>
/gpfs/fs1/work/zhuming/METplus4/parm/met_config/Ascii2NcConfig_wrapped
-v 2
>
> 03/26 12:35:29.433 metplus (command_builder.py:166) ERROR:
> (command_builder.py:1151) MET command returned a non-zero return
> code:/glade/p/ral/jntp/MET/MET_releases/10.0.0-beta3/bin/ascii2nc
> /glade/collections/rda/data/ds351.0/little_r/2015/OBS:2015071500
> /glade/scratch/zhuming/metplus4/ascii2nc2/obs.2015071500.nc -format
> "little_r" -config
>
/gpfs/fs1/work/zhuming/METplus4/parm/met_config/Ascii2NcConfig_wrapped
-v 2
>
> 03/26 12:35:29Z run-METplus-metplus: ERROR:
(command_builder.py:1151) MET
> command returned a non-zero return
> code:/glade/p/ral/jntp/MET/MET_releases/10.0.0-beta3/bin/ascii2nc
> /glade/collections/rda/data/ds351.0/little_r/2015/OBS:2015071500
> /glade/scratch/zhuming/metplus4/ascii2nc2/obs.2015071500.nc -format
> "little_r" -config
>
/gpfs/fs1/work/zhuming/METplus4/parm/met_config/Ascii2NcConfig_wrapped
-v 2
>
> 03/26 12:35:29.433 metplus (command_builder.py:1153) INFO: Check the
> logfile for more information on why it failed:
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210326123527
>
> 03/26 12:35:29.434 metplus.ASCII2NC (met_util.py:214) ERROR:
ASCII2NC had 1
> error.
>
> 03/26 12:35:29Z run-METplus-metplus.ASCII2NC: ERROR: ASCII2NC had 1
error.
>
> 03/26 12:35:29.434 metplus (met_util.py:883) INFO:
> /glade/scratch/zhuming/metplus4/metplus_final.conf: write
metplus.conf here
>
> 03/26 12:35:29.436 metplus (met_util.py:251) ERROR: METplus has
finished
> running but had 1 error.
>
> 03/26 12:35:29Z run-METplus-metplus: ERROR: METplus has finished
running
> but had 1 error.
>
> 03/26 12:35:29.436 metplus (met_util.py:252) INFO: Check the log
file for
> more information:
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210326123527
>
> cheyenne5:/gpfs/fs1/work/zhuming/METplus4>
>
>
>
>
> 2.
>
>
> cheyenne5:/gpfs/fs1/work/zhuming/METplus4>master_metplus.py -c
> ./ASCII2NC3.conf ./metplus_system.conf
>
> Running METplus 4.0.0-beta3
>
> Starting METplus v4.0.0-beta3
>
> 03/29 10:13:01.465 metplus (config_metplus.py:121) INFO: Starting
METplus
> configuration setup.
>
> 03/29 10:13:01.475 metplus (config_metplus.py:249) INFO:
>
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_system.conf:
> Parsed this file
>
> 03/29 10:13:01.475 metplus (config_metplus.py:249) INFO:
>
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_data.conf:
> Parsed this file
>
> 03/29 10:13:01.476 metplus (config_metplus.py:249) INFO:
>
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_runtime.conf:
> Parsed this file
>
> 03/29 10:13:01.490 metplus (config_metplus.py:249) INFO:
>
/gpfs/fs1/work/zhuming/METplus4/parm/metplus_config/metplus_logging.conf:
> Parsed this file
>
> 03/29 10:13:01.514 metplus (config_metplus.py:249) INFO:
./ASCII2NC3.conf:
> Parsed this file
>
> 03/29 10:13:01.515 metplus (config_metplus.py:249) INFO:
> ./metplus_system.conf: Parsed this file
>
> 03/29 10:13:01.521 metplus (config_metplus.py:438) INFO: Adding:
> config.LOG_TIMESTAMP='20210329101301'
>
> 03/29 10:13:01.521 metplus (config_metplus.py:444) INFO: Replace
[config]
> LOG_METPLUS with
>
'/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210329101301'
>
> 03/29 10:13:01.524 metplus (config_metplus.py:316) INFO: Setting
[dir]
> METPLUS_BASE to /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-beta3
>
> 03/29 10:13:01.524 metplus (config_metplus.py:316) INFO: Setting
[dir]
> PARM_BASE to /gpfs/fs1/work/zhuming/METplus4/parm
>
> 03/29 10:13:01.524 metplus (config_metplus.py:323) INFO:
METPLUS_CONF:
> /glade/scratch/zhuming/metplus4/metplus_final.conf written here.
>
> 03/29 10:13:01.526 metplus (config_metplus.py:141) INFO: Completed
METplus
> configuration setup.
>
> 03/29 10:13:01.526 metplus (met_util.py:101) INFO: Running METplus
> v4.0.0-beta3 called with command:
> /glade/p/ral/jntp/MET/METplus/METplus-4.0.0-
beta3/ush/master_metplus.py -c
> ./ASCII2NC3.conf ./metplus_system.conf
>
> 03/29 10:13:01.526 metplus (met_util.py:103) INFO: Log file:
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210329101301
>
> 03/29 10:13:04.746 metplus (met_util.py:1022) INFO:
> ****************************************
>
> 03/29 10:13:04.747 metplus (met_util.py:1023) INFO: * Running
METplus
>
> 03/29 10:13:04.747 metplus (met_util.py:1027) INFO: * at valid
time:
> 2015-07-15 00:00
>
> 03/29 10:13:04.747 metplus (met_util.py:1028) INFO:
> ****************************************
>
> 03/29 10:13:04.758 metplus (command_runner.py:90) INFO: COMMAND:
> /glade/p/ral/jntp/MET/MET_releases/10.0.0-beta3/bin/ascii2nc
> /glade/u/home/taosun/work/data/2015case/gts/obs.2015071500
> /glade/scratch/zhuming/metplus4/ascii2nc3/obs.2015071500.nc -format
> "little_r" -config
>
/gpfs/fs1/work/zhuming/METplus4/parm/met_config/Ascii2NcConfig_wrapped
-v 2
>
> 03/29 10:14:17.268 metplus (met_util.py:883) INFO:
> /glade/scratch/zhuming/metplus4/metplus_final.conf: write
metplus.conf here
>
> 03/29 10:14:17.274 metplus (met_util.py:244) INFO: Check the log
file for
> more information:
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210329101301
>
> 03/29 10:14:17.301 metplus (met_util.py:245) INFO: METplus has
successfully
> finished running.
>
> cheyenne5:/gpfs/fs1/work/zhuming/METplus4>
>
>
------------------------------------------------
Subject: Help on processing observation data with little-r format
From: John Halley Gotway
Time: Mon Mar 29 13:40:10 2021
Zhuming,
OK, making a little progress. Looking at the files in:
/glade/collections/rda/data/ds351.0/little_r/2015
I tested with "OBS:2015080700" out of the many, many obs in that file
4 of
them end with the following line:
0 0 0
While all the rest of them end with this line:
1 0 0
Here are the lines I'm talking about (see attached):
egrep -B 2 " 0 0 0" OBS\:2015080700 > bad_obs
When I manually remove those 4, ascii2nc runs to completion. So this
is a
variation in the data that ascii2nc is not expecting.
I don't know very much about the little_r format. Do you know what
these 4
report types are? Looks like they contain bad lat/lon values anyway
(0,0)
or (9.92,0).
I assume we just need to make ascii2nc smart enough to skip over these
reports?
Thanks,
John
On Mon, Mar 29, 2021 at 12:17 PM The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Mon Mar 29 12:17:50 2021: Request 99278 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> Queue: met_help
> Subject: Help on processing observation data with little-r
format
> Owner: johnhg
> Requestors: zhuming at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99278 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: Help on processing observation data with little-r format
From: John Halley Gotway
Time: Tue Mar 30 10:22:27 2021
FYI, I wrote up this GitHub issue describing the problem:
https://github.com/dtcenter/MET/issues/1737
And I have a fix for it to enable ascii2nc to read these little_r
files and
correctly skip over these bad records where the timestamp is bad.
These
changes will be included in the next release of MET, version 10.0.0.
John
On Mon, Mar 29, 2021 at 1:39 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Zhuming,
>
> OK, making a little progress. Looking at the files in:
> /glade/collections/rda/data/ds351.0/little_r/2015
>
> I tested with "OBS:2015080700" out of the many, many obs in that
file 4 of
> them end with the following line:
> 0 0 0
> While all the rest of them end with this line:
> 1 0 0
>
> Here are the lines I'm talking about (see attached):
> egrep -B 2 " 0 0 0" OBS\:2015080700 > bad_obs
>
> When I manually remove those 4, ascii2nc runs to completion. So this
is a
> variation in the data that ascii2nc is not expecting.
> I don't know very much about the little_r format. Do you know what
these 4
> report types are? Looks like they contain bad lat/lon values anyway
(0,0)
> or (9.92,0).
>
> I assume we just need to make ascii2nc smart enough to skip over
these
> reports?
>
> Thanks,
> John
>
> On Mon, Mar 29, 2021 at 12:17 PM The RT System itself via RT <
> met_help at ucar.edu> wrote:
>
>>
>> Mon Mar 29 12:17:50 2021: Request 99278 was acted upon.
>> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>> Queue: met_help
>> Subject: Help on processing observation data with little-r
format
>> Owner: johnhg
>> Requestors: zhuming at ucar.edu
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99278 >
>>
>>
>> This transaction appears to have no content
>>
>
------------------------------------------------
Subject: Help on processing observation data with little-r format
From: zhuming at ucar.edu
Time: Tue Mar 30 18:54:55 2021
Hi John,
First thanks for your time and your help.
1. Could you let me know how to set up time for 12 hour obs using
"ASCII2NC". such as:
VALID_TIME_FMT = %Y%m%d%H
VALID_BEG = 2015071500
VALID_END = 2015071500
VALID_INCREMENT = 1M
LEAD_SEQ = 0
2. I tried removing obs with end 0 0 0 as you suggested using
little-r
format, "ASCII2NC" worked fine.
When I use this processed data in "PointStat":
/gpfs/fs1/work/zhuming/METplus4>master_metplus.py -c PointStat11.conf
./metplus_system.conf
METplus has successfully finished running, but I checked the results
in /glade/scratch/zhuming/metplus4/point_stat11, I can only see the
head in
the files. See the log file of
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210330164037
or
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210330171205
3. I also tried prebufr data stored in
/glade/collections/rda/data/ds337.0/prepnr/
/gpfs/fs1/work/zhuming/METplus4>master_metplus.py -c PointStat22.conf
./metplus_system.conf
see log file of
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210330181351.
I can see some matched pairs upair, but not on the surface.
I'm totally confused!
Thanks!
Zhuming
On Mon, Mar 29, 2021 at 1:40 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Zhuming,
>
> OK, making a little progress. Looking at the files in:
> /glade/collections/rda/data/ds351.0/little_r/2015
>
> I tested with "OBS:2015080700" out of the many, many obs in that
file 4 of
> them end with the following line:
> 0 0 0
> While all the rest of them end with this line:
> 1 0 0
>
> Here are the lines I'm talking about (see attached):
> egrep -B 2 " 0 0 0" OBS\:2015080700 > bad_obs
>
> When I manually remove those 4, ascii2nc runs to completion. So this
is a
> variation in the data that ascii2nc is not expecting.
> I don't know very much about the little_r format. Do you know what
these 4
> report types are? Looks like they contain bad lat/lon values anyway
(0,0)
> or (9.92,0).
>
> I assume we just need to make ascii2nc smart enough to skip over
these
> reports?
>
> Thanks,
> John
>
> On Mon, Mar 29, 2021 at 12:17 PM The RT System itself via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Mar 29 12:17:50 2021: Request 99278 was acted upon.
> > Transaction: Given to johnhg (John Halley Gotway) by RT_System
> > Queue: met_help
> > Subject: Help on processing observation data with little-r
format
> > Owner: johnhg
> > Requestors: zhuming at ucar.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99278 >
> >
> >
> > This transaction appears to have no content
> >
>
>
------------------------------------------------
Subject: Help on processing observation data with little-r format
From: John Halley Gotway
Time: Mon Apr 19 11:30:16 2021
Hi Zhuming,
I'm on the met_help desk on Mondays and was looking through existing
tickets. I see that I have 3 tickets from you that remain open.
- RMSE using MET/METplus point-stat tool
- Verification on MPAS forecast using MET
- Help on processing observation data with little-r format
But I do know that you were also working with George McCabe on setting
up/running through METplus. With several open tickets, it's difficult
for me to know how to help. Can you point me in the right direction
and let me know what open questions you have about the MET tools?
Thanks,
John
------------------------------------------------
Subject: Help on processing observation data with little-r format
From: zhuming at ucar.edu
Time: Mon Apr 19 20:41:54 2021
Hi John,
Sorry for so many help emails.
Generally I started a help email on Monday, and immediately I got an
answer on Mon or Tue (thanks lot). But After I replied for further
help, no
one answered me again. So on next Monday, I started a help email
again. One
day I was replied with a message like " I need to reply with the
ticket
number" . Everything seems to go smoothly after that.
You may keep the ticket "about little-r obs data'', and delete the
other
two. I prefer to use the little-r format, but I switched to using
prepbufr
data because I still can't make it work.
1. Processing data using "ASCII2NC" works fine, after removing obs
with end
0 0 0 as you suggested using little-r format. See
log file: /glade/scratch/zhuming/metplus4/logs/master_metplus.lo
g.20210419195816
config file: /glade/work/zhuming/METplus4/ASCII2NC.conf
2. When I use this processed data in "PointStat", METplus has
successfully
finished running, but I checked the results I can only see the head in
the
files. See
config file: /glade/work/zhuming/METplus4/PointStat.conf
log file:
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210419202330
DEBUG 2: Processing T_2m(0,*,*) versus TMP/Z2, for observation type
ADPSFC,
over region FULL, for interpolation method BILIN(4), using 0 matched
pairs.
DEBUG 3: Number of matched pairs = 0
DEBUG 3: Observations processed = 2652900
DEBUG 3: Rejected: station id = 0
DEBUG 3: Rejected: obs var name = 2170559
DEBUG 3: Rejected: valid time = 239164
DEBUG 3: Rejected: bad obs value = 0
DEBUG 3: Rejected: off the grid = 238346
DEBUG 3: Rejected: topography = 0
DEBUG 3: Rejected: level mismatch = 4831
DEBUG 3: Rejected: quality marker = 0
DEBUG 3: Rejected: message type = 0
DEBUG 3: Rejected: masking region = 0
DEBUG 3: Rejected: bad fcst value = 0
DEBUG 3: Rejected: bad climo mean = 0
DEBUG 3: Rejected: bad climo stdev = 0
DEBUG 3: Rejected: duplicates = 0
DEBUG 2:
Thanks!
Zhuming
On Mon, Apr 19, 2021 at 11:30 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Hi Zhuming,
>
> I'm on the met_help desk on Mondays and was looking through existing
> tickets. I see that I have 3 tickets from you that remain open.
>
> - RMSE using MET/METplus point-stat tool
> - Verification on MPAS forecast using MET
> - Help on processing observation data with little-r format
>
> But I do know that you were also working with George McCabe on
setting
> up/running through METplus. With several open tickets, it's
difficult for
> me to know how to help. Can you point me in the right direction and
let me
> know what open questions you have about the MET tools?
>
> Thanks,
> John
>
------------------------------------------------
Subject: Help on processing observation data with little-r format
From: John Halley Gotway
Time: Wed Apr 21 12:47:10 2021
Zhuming,
Great, thanks for sending these log messages. I made a copy of your
current
setup in:
/glade/p/ral/jntp/MET/MET_Help/zhuming_data_20210421
But I too could get 0 matched pairs. So I tried running MET's
plot_point_obs tool like this:
/glade/p/ral/jntp/MET/MET_releases/10.0.0-beta3/bin/plot_point_obs
/glade/scratch/zhuming/metplus4/ascii2nc/obs.2015071500.nc
obs.2015071500.ps
-gc 11 -msg_typ ADPSFC
That should plot the location of temperature observations for the
ADPSFC
message type, but there are none:
DEBUG 2: Finished plotting 0 locations.
So we need to see what actually IS present in that observation file...
I do see GRIB codes of 11, for temperature... so that's good:
ncdump -v obs_gc /glade/scratch/zhuming/metplus4/ascii2nc/
obs.2015071500.nc
But checking the message types, there are no instances of ADPSFC:
ncdump -v hdr_typ_table /glade/scratch/zhuming/metplus4/ascii2nc/
obs.2015071500.nc
* hdr_typ_table = "AIRCFT", "SATWND", "ADPUPA" ;*
If there are not any observations of TMP for ADPSFC, we won't get any
matched pairs for them.
So please take a step back and look at the data you're passing to
ascii2nc.
John
On Mon, Apr 19, 2021 at 8:42 PM zhuming at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99278 >
>
> Hi John,
>
> Sorry for so many help emails.
> Generally I started a help email on Monday, and immediately I got
an
> answer on Mon or Tue (thanks lot). But After I replied for further
help, no
> one answered me again. So on next Monday, I started a help email
again. One
> day I was replied with a message like " I need to reply with the
ticket
> number" . Everything seems to go smoothly after that.
>
> You may keep the ticket "about little-r obs data'', and delete the
other
> two. I prefer to use the little-r format, but I switched to using
prepbufr
> data because I still can't make it work.
>
> 1. Processing data using "ASCII2NC" works fine, after removing obs
with end
> 0 0 0 as you suggested using little-r format. See
> log file: /glade/scratch/zhuming/metplus4/logs/master_metplus.lo
> g.20210419195816
> config file: /glade/work/zhuming/METplus4/ASCII2NC.conf
>
> 2. When I use this processed data in "PointStat", METplus has
successfully
> finished running, but I checked the results I can only see the head
in the
> files. See
> config file: /glade/work/zhuming/METplus4/PointStat.conf
> log file:
>
/glade/scratch/zhuming/metplus4/logs/master_metplus.log.20210419202330
>
>
> DEBUG 2: Processing T_2m(0,*,*) versus TMP/Z2, for observation type
ADPSFC,
> over region FULL, for interpolation method BILIN(4), using 0 matched
pairs.
>
> DEBUG 3: Number of matched pairs = 0
>
> DEBUG 3: Observations processed = 2652900
>
> DEBUG 3: Rejected: station id = 0
>
> DEBUG 3: Rejected: obs var name = 2170559
>
> DEBUG 3: Rejected: valid time = 239164
>
> DEBUG 3: Rejected: bad obs value = 0
>
> DEBUG 3: Rejected: off the grid = 238346
>
> DEBUG 3: Rejected: topography = 0
>
> DEBUG 3: Rejected: level mismatch = 4831
>
> DEBUG 3: Rejected: quality marker = 0
>
> DEBUG 3: Rejected: message type = 0
>
> DEBUG 3: Rejected: masking region = 0
>
> DEBUG 3: Rejected: bad fcst value = 0
>
> DEBUG 3: Rejected: bad climo mean = 0
>
> DEBUG 3: Rejected: bad climo stdev = 0
>
> DEBUG 3: Rejected: duplicates = 0
>
> DEBUG 2:
>
>
> Thanks!
> Zhuming
>
>
> On Mon, Apr 19, 2021 at 11:30 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Hi Zhuming,
> >
> > I'm on the met_help desk on Mondays and was looking through
existing
> > tickets. I see that I have 3 tickets from you that remain open.
> >
> > - RMSE using MET/METplus point-stat tool
> > - Verification on MPAS forecast using MET
> > - Help on processing observation data with little-r format
> >
> > But I do know that you were also working with George McCabe on
setting
> > up/running through METplus. With several open tickets, it's
difficult for
> > me to know how to help. Can you point me in the right direction
and let
> me
> > know what open questions you have about the MET tools?
> >
> > Thanks,
> > John
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list