[Dart-dev] [6554] DART/trunk/observations/NCEP/prep_bufr/prep_bufr.html: updated to remove references to the MSS (now HPSS).
nancy at ucar.edu
nancy at ucar.edu
Tue Oct 29 11:58:44 MDT 2013
Revision: 6554
Author: nancy
Date: 2013-10-29 11:58:44 -0600 (Tue, 29 Oct 2013)
Log Message:
-----------
updated to remove references to the MSS (now HPSS). also added
text to help with cosblocking files on yellowstone.
Modified Paths:
--------------
DART/trunk/observations/NCEP/prep_bufr/prep_bufr.html
-------------- next part --------------
Modified: DART/trunk/observations/NCEP/prep_bufr/prep_bufr.html
===================================================================
--- DART/trunk/observations/NCEP/prep_bufr/prep_bufr.html 2013-10-29 16:55:27 UTC (rev 6553)
+++ DART/trunk/observations/NCEP/prep_bufr/prep_bufr.html 2013-10-29 17:58:44 UTC (rev 6554)
@@ -36,9 +36,9 @@
<H2>Overview</H2>
<P>
-Translating NCEP BUFR files into DART obs_seq.out files (input
+Translating NCEP PREPBUFR files into DART obs_seq.out files (input
file to filter) is a 2 stage process. The first stage uses NCEP
-software to translate the BUFR file into an "intermediate" text file.
+software to translate the PREPBUFR file into an intermediate text file.
This is described in this document. The second step is to translate
the intermediate files into obs_seq.out files, which is done by
create_real_obs, as described in
@@ -75,14 +75,15 @@
is available on your system.
</P>
<P>
-If you have raw unblocked BUFR files you will need to convert them
-to blocked format (what prepbufr expects as input). You will need
-to uncomment the blk/ublk section in the script and build that
+If you have raw unblocked PREPBUFR files you will need to convert them
+to blocked format (what prepbufr expects as input). The blk/ublk
+section of the build script compiles the <em class=code>cword.x</em>
converter program.
</P>
<P>
-If you are running on an Intel (little-endian) based machine, uncomment
-the grabbufr section in the script to build that converter program.
+If you are running on an Intel (little-endian) based machine you will
+need the <em class=code>grabbufr</em> byte swapping program that is
+also built by this script.
</P>
<H4 class=indent1> One-shot execution</H4>
@@ -96,7 +97,7 @@
<UL>
<LI>building the executables.
<LI>running the blocker if needed (generally not if you have
- downloaded the blocked format BUFR files).
+ downloaded the blocked format PREPBUFR files).
<LI>running the binary format converter if you are on an Intel
(little-endian) machine.
<LI>linking the input file to a fixed input filename
@@ -118,7 +119,7 @@
</P>
-<H3 class=indent1> Installation of the NCEP BUFR decoding program</H3>
+<H3 class=indent1> Installation of the NCEP PREPBUFR decoding program</H3>
<P>
This package is currently organized into files under the
@@ -129,27 +130,27 @@
lib NCEP BUFR library source
install.sh A script to install the NCEP PREPBUFR decoder and the NCEP BUFR library.
exe Executables of the decoder and converter.
-data Where the NCEP BUFR files (prepqm****) could be loaded into
+data Where the NCEP PREPBUFR files (prepqm****) could be loaded into
from the NCAR Mass Store (the script assumes this is the default location).
work Where we run the script to do the decoding.
-convert_bufr Source code (grabbufr) to convert the binary big-endian BUFR files to
+convert_bufr Source code (grabbufr) to convert the binary big-endian PREPBUFR files to
little-endian files, and a script to compile the program.
blk_ublk Source code (cwordsh) to convert between blocked and unblocked format.
-docs Some background information about NCEP BUFR observations.
+docs Some background information about NCEP PREPBUFR observations.
</PRE>
<H4 class=indent1> The decoding program: src/prepbufr.f</H4>
<P>
-The program prepbufr.f is used to decode the NCEP reanalysis BUFR data into
+The program prepbufr.f is used to decode the NCEP reanalysis PREPBUFR data into
intermediate text files. This program was originally developed by NCEP.
It has been modified to output surface pressure, dry temperature, specific
humidity, and wind components (U/V) of conventional radiosonde, aircraft
reports, and satellite cloud motion derived wind. There are additional
-observation types on the BUFR files, but using them they would require
+observation types on the PREPBUFR files, but using them they would require
significant modifications of prepbufr and require detailed knowledge
-of the NCEP BUFR files. The NCEP quality control indexes for these
+of the NCEP PREPBUFR files. The NCEP quality control indexes for these
observations based on NCEP forecasts are also output and used in DART
observation sequence files. The NCEP PREPBUFR decoding program is written
in Fortran 77 and has been successfully compiled on Linux computers using
@@ -172,7 +173,7 @@
set the CPLAT variable in the install.sh script to match the compilers
available on your system. CPLAT = linux is the default.
Execute the install.sh script to complete the compilations for the main
-decoding program and the NCEP BUFR library.
+decoding program, the NCEP BUFR library, and the conversion utilities.
</P>
<P>
@@ -194,17 +195,19 @@
<P>
For platforms with little-endian binary file format (e.g. Intel, AMD®,
and non-MIPS SGI processors) the program grabbufr.f is used to convert the
-big-endian format NCEP BUFR data into little-endian format. The grabbufr.f
+big-endian format NCEP PREPBUFR data into little-endian format. The grabbufr.f
code is written in Fortran 90, and has been compiled can be compiled with
the pgf90 compiler on a Linux system, with gfortran on an Intel based
-Mac, and the ifort compiler on other Linux machines. Instructions for
-building it can be found in convert_bufr/README, but basically go to the
+Mac, and the ifort compiler on other Linux machines. More detailed
+instructions for
+building it can be found in convert_bufr/README, but the base install
+script should build this by default. In case of problems, cd into the
convert_bufr subdirectory, edit convert_bufr.csh to set your compiler,
and run it to compile the converter code (grabbufr).
</P>
<P>
-This program reads the whole BUFR file into memory, and needs to know
+This program reads the whole PREPBUFR file into memory, and needs to know
the size of the file (in bytes). Unfortunately, the system call STAT()
returns this size as one number in an array, and the index into that array
differs depending on the system and sometimes the word size (32 vs 64)
@@ -228,104 +231,104 @@
use links to temporarily give the files the names needed.
</P>
-<H4 class=indent1> The blocking program blk_ublk/cwordsh</H4>
+<H4 class=indent1> The blocking program blk_ublk/cword.x</H4>
<P>
The prepbufr.x program expects to read a blocked input file, which
is generally what is available for download. However, if you have an
unblocked file that you need to convert, there is a conversion program.
-Change directories into the blk_ublk subdirectory and edit the cwordsh.make
-file and edit it to match your system. Execute it and it will build
-a cwordsh.x executable, which can be run with the shell-script wrapper
-program cwordsh.
+The install.sh script will try to build this by default, but in case of
+problems you can build it separately.
+Change directories into the blk_ublk subdirectory and read the
+README_cwordsh file for more help. The cwordsh shell-script wrapper
+shows how to run the executable cwordsh.x executable.
</P>
<P>
-The shell scripts in this directory actually create both a .c and .f
-file (which did not exist before), then compile them, then remove them. If you
-have problems with the compile, comment out the remove at the end of the make
-script so you can see the files it is actually trying to compile.
-</P>
-
-<P>
Note that if you can get the blocked file formats to begin with, this
program is not needed.
</P>
-<H3 class=indent1> Getting the NCEP Reanalysis BUFR format data from NCAR MSS. </H3>
+<H3 class=indent1> Getting the NCEP Reanalysis PREPBUFR format data from NCAR HPSS. </H3>
<P>
-The NCEP BUFR files (prepqmYYMMDDHH) can be found within the NCEP reanalysis
-dataset, ds090.0, on NCAR Mass Store System (MSS).
+The NCEP PREPBUFR files (prepqmYYMMDDHH) can be found within the NCEP reanalysis
+dataset, ds090.0, on NCAR Mass Store System (HPSS).
</P>
<P>
-<!-- kdr; How would I launch another window for the data web site?
- How to change bullets to '>' prompts for commands? -->
To find the files:
</P>
<UL>
-<LI>go to the <A HREF="http://dss.ucar.edu/pub/reanalysis/" target="_blank">
+<LI>go to the <A HREF="http://rda.ucar.edu/datasets/ds090.0/" target="_blank">
NCAR/NCEP reanalysis archive.</A></LI>
-<LI>Click "Data Archive Summary" </LI>
-<LI>Click "Complete Tar list of Table of Contents of All VSN's" </LI>
-<LI>Look at the years you want and click on the ds090.0 next to the
- year mark.</LI>
-<LI>Find the data set you want by searching for prepqmYYMMDDHH, with
- YY, MM, DD, HH = the year, month, day, and hour you want.</LI>
+<LI>Click on the "Inventories" tab.</LI>
+<LI>Select the year you are interested in.</LI>
+<LI>Search for files with the string "prepqm" in the name.</LI>
+<LI>Depending on the year the format of the filenames change,
+but they should contain the year, usually as 2 digits, the month,
+and then either the start/stop day for weekly files, or the
+letters A and B for semi-monthly files.</LI>
</UL>
<P>
-The prepqm file will be grouped with others in a weekly (monthly after
-2003) tar file. Each tar file has a unique file number like "A#####".
-For example, for January of 2003, the 4 MSS TAR files are: A21899, A21900,
-A21901, A21902. Before 2004 these are unblocked files. The decoding program
-requires blocked version of the files. The corresponding blocked files
-are A21899.blk, A21900.blk, A21901.blk, and A21902.blk. These names don't
-appear in the web page, but are found on the mass store. Each is about
-400 Mb. After 2003 the files without ".blk" are blocked and can be used.
-They are about 1.7 Gb. Also, after September, 2003, these files include
+Depending on the year you select, the prepqm files can be weekly,
+monthly, or semi-monthly. Each tar file has a unique dataset
+number of the form "A#####".
+For example, for January of 2003, the 4 HPSS TAR files are: A21899, A21900,
+A21901, A21902.
+After September 2003, these files include
AIRCRAFT data (airplane readings taken at cruising elevation) but not
ACARS data (airplane readings taken during takeoff and landing). There
are different datasets which include ACARS data but their use is restricted
-and you must contact the DSS group to get access.
+and you must contact the RDA group to get access.
</P>
<P>
-Change directories into the prep_bufr/data subdirectory.
-From a machine with access to the MSS use
+If you are running on a machine with direct access to the NCAR HPSS,
+then change directories into the prep_bufr/data subdirectory and run:
+<br /> <br />
+<em class=input>> hsi get /DSS/A##### rawfile</em><br />
<br />
-<em class=input>msrcp mss:/DSS/A##### data.tar</em><br />or<br />
-<em class=input>msrcp mss:/DSS/A#####.blk data.tar</em><br />
-where ##### is the data set number, to read the NCEP BUFR data tar files.
-The files which include ACARS data have a read
-password; to specify that to the 'msrcp' command, add the '-rpwd <i>pass</i>'
-argument (where <i>pass</i> is a 1-8 character password).
-<br /><br />
-You must then run the 'cosconvert' program to convert the file into a
-readable tar file.
+where ##### is the data set number you want.
</P>
-<ul>
-<li>On bluefire: /contrib/bin/cosconvert -b data.tar
-<li>On other platforms, download the appropriate version from:
-http://www.dss.ucar.edu/libraries/io/cos_blocking/utils/
-</ul>
<P>
-Then
-<br /><br />
-<em class=input>tar -xvf data.tar </em>
-<br /><br />
-will yield individual 6-hourly
-NCEP BUFR data files for the observations in the +/- 3-hour time windows
+These files may be readable tar files, or they may require
+running the <em class=file>cosconvert</em> program first.
+See if the <em class=file>tar</em> command can read them:
+<br /> <br />
+<em class=input>> tar -tvf rawfile</em><br />
+<br />
+If you get a good table of contents then simply rename the file
+and untar it:
+<br /> <br />
+<em class=input>> mv rawfile data.tar</em><br />
+<em class=input>> tar -xvf data.tar</em><br />
+<br />
+However, if you get an error from the tar command you
+will need to run the <em class=file>cosconvert</em> program to convert the file into a
+readable tar file. On the NCAR machine <em class=machine>yellowstone</em>, run:
+<br /> <br />
+<em class=input>> /glade/u/home/rdadata/bin/cosconvert -b rawfile data.tar</em><br />
+<br />
+On other platforms, download the appropriate version from:
+<a href="http://rda.ucar.edu/libraries/io/cos_blocking/utils/" target="_blank">
+http://rda.ucar.edu/libraries/io/cos_blocking/utils/</a> .
+Build and run the converter and
+then you should have a tar file you can unpack.
+</P>
+<P>
+The output of tar should yield individual 6-hourly
+NCEP PREPBUFR data files for the observations in the +/- 3-hour time windows
of 00Z, 06Z, 12Z, and 18Z of each day. Note that DART obs_seq files are
organized such that a 24 hour file with 4 observation times would contain
observations from 3:01Z to 3:00Z of the next day, centered on 6Z, 12Z, 18Z
-and "24Z". In addition, there are some observations at 3:00Z on the BUFR
+and "24Z". In addition, there are some observations at 3:00Z on the PREPBUFR
file labelled with 06Z. Then, in order to make a full day intermediate file
-incorporating all the required obs from the "next" day, you'll need the BUFR
+incorporating all the required obs from the "next" day, you'll need the PREPBUFR
files through 6Z of the day after the last day of interest. For example,
to generate the observation sequence for Jan 1, 2003, the decoded NCEP
-BUFR text files for Jan 1 and 2, 2003 are needed, and hence the BUFR files
+PREPBUFR text files for Jan 1 and 2, 2003 are needed, and hence the PREPBUFR files
</P>
<UL>
<LI> prepqm03010106 </LI>
@@ -338,22 +341,22 @@
are needed.
</P>
-<H3 class=indent1> Running the NCEP BUFR decoding program</H3>
+<H3 class=indent1> Running the NCEP PREPBUFR decoding program</H3>
<P>
In prep_bufr/work/prepbufr.csh set the appropriate values of the year,
month, first day, and last day of the period you desire, and the variable
"convert" to control conversion from big- to little-endian. Confirm that
-the raw BUFR files are in ../data, or that prepbufr.csh has been changed
+the raw PREPBUFR files are in ../data, or that prepbufr.csh has been changed
to find them. Execute prepbufr.csh in the work directory. It has code
for running in the LSF batch environment, but not PBS.
</P>
<P>
-Currently, this script generates decoded BUFR text data each 24 hours
+Currently, this script generates decoded PREPBUFR text data each 24 hours
which contains the observations within the time window of -3:01 hours
to +3:00Z within each six-hour synoptic time. These daily output text
-files are named as temp_obs.yyyymmdd. These text BUFR data files can
+files are named as temp_obs.yyyymmdd. These text PREPBUFR data files can
then be read by
DART/observations/NCEP/ascii_to_obs/work/<A HREF="../create_real_obs.html">create_real_obs </A>
to generate the DART daily observation sequence files.
@@ -361,8 +364,8 @@
<P>
There is an alternate section in the script which creates a decoded
-BUFR text data file each 6 hours (so they are 1-for-1 with the original
-BUFR files). Edit the script prepbufr.csh and look for the commented
+PREPBUFR text data file each 6 hours (so they are 1-for-1 with the original
+PREPBUFR files). Edit the script prepbufr.csh and look for the commented
out code which outputs 4 individual files per day. Note that if you
chose this option, you will have to make corresponding changes in
the create_obs_seq.csh script in step 2.
@@ -500,7 +503,7 @@
<A NAME="FilesUsed"></A>
<div class="top">[<a href="#">top</a>]</div><hr />
<H2>FILES</H2>
-<UL><LI>input file(s); NCEP BUFR observation files named using ObsBase
+<UL><LI>input file(s); NCEP PREPBUFR observation files named using ObsBase
with the "yymmddhh" date tag on the end. Input to grabbufr if big- to
little-endian is to be done. Input to prepbufr if not.
<LI>intermediate (binary) prepqm.little; output from grabbufr,
@@ -516,7 +519,7 @@
<A NAME="References"></A>
<div class="top">[<a href="#">top</a>]</div><hr />
<H2>REFERENCES</H2>
-DART/observations/NCEP/prep_bufr/docs/* (NCEP text files describing the BUFR files)
+DART/observations/NCEP/prep_bufr/docs/* (NCEP text files describing the PREPBUFR files)
<!--==================================================================-->
<!-- Describe all the error conditions and codes. -->
@@ -526,7 +529,7 @@
<div class="top">[<a href="#">top</a>]</div><hr />
<H2>ERROR CODES and CONDITIONS</H2>
<P>
-this space intentionally left blank
+Various, see the source code, doc directory, and README files for more help.
</P>
<H2>KNOWN BUGS</H2>
@@ -542,9 +545,16 @@
<div class="top">[<a href="#">top</a>]</div><hr />
<H2>FUTURE PLANS</H2>
<P>
-Further development to get observations directly from original
-(undecoded) NCEP BUFR files.
+This converter could be combined with the DART library code
+to go directly from PREPBUFR files to DART obs_sequence files.
</P>
+<P>
+The converter should make an output file with a fully qualified date
+in the name, so constructing files that start at 21Z and
+continue to 3Z the next day are not as difficult to process.
+There are currently two separate versions of the converter only
+because one deals with the date wrap. This shouldn't be needed.
+</P>
<!--==================================================================-->
<!-- Legalese & Metadata -->
More information about the Dart-dev
mailing list