[TCDP-flight-news] FLIGHT+ Dataset updates: Work is underway to extend the dataset to 2016-2018

[Jonathan Vigh]

Greetings Registered FLIGHT+ Dataset User,

I'd like to share several updates and other information about the 
FLIGHT+ Dataset.
**
*
*
*Dataset extension to 2016 - 2018*

Through funding support from the NOAA Hurricane Forecast Improvement 
Project (HFIP) through a grant entitled "New Frameworks for Predicting 
Extreme Rapid Intensfication" (award number: NA18NWS4680058), work is 
underway to extend the FLIGHT+ Dataset to include the 2016 - 2018 
seasons. The target for completion of this dataset extension is February 
25, 2020, however I wanted to check with the user community to see if 
there are any storms in particular that should be prioritized. I will 
focus first on the high-impact storms (Harvey, Irma, Maria in 2017).  
Some data will start to populate to the web site as this work 
progresses, but keep in mind that any data files for the period 
2014-2018 should be considered as "beta" releases until their status is 
updated as "final" in the data provenance Wiki. Notices will be sent to 
this list when a new batch of storms has been finalized.

*Current version *

If you are using any versions lower than v1.3, please download the 
updated dataset. A number of important bugs and issues have been 
corrected in v1.3. For a full version history, see:
https://verif.rap.ucar.edu/tcdata/flight/sources/

Briefly, there are still two issues that are only partially resolved. 
These are described below.


    Version 1.3

*Version 1.3*is the current version of FLIGHT+. A full description for 
what has changed in this version is not yet available, but briefly this 
version provides partial fixes for two major issues:

*1. Bad time metadata in NOAA NetCDF data files*: A key piece of 
metadata that FLIGHT+ uses construct the time coordinate for these data 
files was off by exactly five hours in all NOAA NetCDF Data files from 
2005-2012. This issue resulted in very few radial legs being parsed from 
any of these 160 flights (and any radial legs that were parsed were 
likely parsed backward). All AFRES flights during this period were 
unaffected. The FLIGHT+ code has been updated to detect incorrect time 
stamps and to correct the time, if necessary. As a result of this fix, 
approximately 1000+ radial legs have been added to the v1.3 dataset. It 
is possible that the processing of NOAA data files from 2012 onward 
still have issues. If so, then radial legs may be missing from these 
flights for the more recent years. This issue will be investigated 
further and hopefully fixed in future Version 1.4.

*2. Improvements in the radial binning to account for low precision 
lat/lon data:*Some of the early source data files only provide the 
lat/lon of the plane location in 0.1 deg increments even though the data 
frequency had been increased to 1-second. As a result, the calculation 
of the distance from plane to the center of the cyclone can appear to be 
constant for 10-15 seconds, then jumps to a new value. When the motion 
of the cyclone itself is removed, the distance to the center may slowly 
increase or decrease by slight amounts, then undergo a jump (depending 
on whether the motion of the storm is along with the motion of the plane 
or against it). This affects the radial binning of the flight level data 
into the radial legs. In previous versions of FLIGHT+, all values in a 
particular radial leg were sorted by distance to center. Although an 
effort was likely made to filter and interpolate the distance to center 
to fix the jumps, it has been determined that the method of doing so was 
inadequate. As a result, data from +/- 0.6 km of a given point may have 
been interspersed, resulting in a decrease in the effective grid spacing 
of the radial data from 100 m to approximately 1 km. This likely would 
have mattered most for any users attempting to compute radial 
derivatives of these data. In v1.3, the algorithm for detecting the 
effects of this low precision on the distance to center has been 
improved, and any points in which the distance to the center is changing 
against the direction of the jumps are now filtered out and linear 
interpolation is used to fill these in (which is reasonable since the 
plane is flying at a fairly constant speed). The issue still remains 
however for jumps that occur in the direction of the storm motion. This 
issue will be investigated further and hopefully fixed in future Version 
1.4.

*New prototype azimuthal mean data product and roadmap for future 
FLIGHT+ development*

Another update is that the FLIGHT+ Development Team (which now includes 
Dr. Eric Hendricks) has recently prototyped a new capability to compute 
the azimuthal mean radial profiles as a new Level 4 data product. Once 
the data extension work is complete, our attention will turn to rolling 
out the azimuthal mean data product for the entire dataset (1997 - 
2018). Once that is complete, we will begin work on the next version of 
the dataset, which will resolve the remaining known issues and extend 
the dataset to 2019. The link above gives the current roadmap for 
anticipated future development of FLIGHT+.

*Reminder on the FLIGHT+ Data User Agreement and how to properly cite 
FLIGHT+*

If you are in the planning to use FLIGHT+ in a publication, please 
remember to get in touch with us and fully follow the FLIGHT+ Data User 
Agreement and license that you agreed to. The user agreement and license 
can be viewed here:

https://verif.rap.ucar.edu/tcdata/flight/dataset/data_user_agreement_and_license.php

We've noticed a couple publications that didn't cite FLIGHT+ properly. 
It is vital to include an accurate citation to the FLIGHT+ Dataset in 
any scientific or technical publication so that we can justify continued 
funding support and updates. The proper citation for v1.3 is:

Vigh, J. L., N. M. Dorst, C. L. Williams, D. P. Stern, E. W. Uhlhorn, B. 
W. Klotz, J. Martinez, H. E. Willoughby, F. D. Marks, Jr., D. R. Chavas, 
2018: FLIGHT+: The Extended Flight Level Dataset for Tropical Cyclones
           (Version 1.3). Tropical Cyclone Data Project, National Center 
for Atmospheric Research, Research Applications Laboratory, Boulder, 
Colorado.
           [Available online at: http://dx.doi.org/10.5065/D6WS8R93.] 
Accessed* dd mmm yyyy.
where * dd mmm yyyy is the date which you last accessed the dataset 
(e.g., 23 Dec 2019).

*Feedback*

Feedback is always welcome. Please send any comments, concerns, or 
dataset issues to ral-tcdp at ucar.edu.

Best regards,

Jonathan Vigh and the FLIGHT+ Development Team