[pyngl-talk] how to use the new MRMS grib codetable

LANG, TIMOTHY J. (MSFC-ZP11) timothy.j.lang at nasa.gov
Thu Jul 2 15:41:14 MDT 2015 

I am the author of MMM-Py so please feel free to send all questions or
comments to me (timothy.j.lang at nasa.gov).

The grib read component of it calls a version of wgrib2 compiled with the
MRMS codetables to convert the MRMS grib2 files to netCDF, which are then
ingested via netCDF4-python. Subsectioning via keywords (e.g.,
latrange=[30,40]) is enabled.

Right now MMM-Py is only tested and working with the 3D MRMS radar
reflectivity mosaics. I do have plans to incorporate 2D rainfall products
in the near future.

Cheers,
Timothy Lang
Research AST, Atmospheric Measurements
NASA Marshall Space Flight Center (ZP11)
Huntsville, AL 35812
(256) 961-7861
timothy.j.lang at nasa.gov



On 7/2/15, 4:29 PM, "pyngl-talk-request at ucar.edu"
<pyngl-talk-request at ucar.edu> wrote:

>Send pyngl-talk mailing list submissions to
>	pyngl-talk at ucar.edu
>
>To subscribe or unsubscribe via the World Wide Web, visit
>	http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>or, via email, send a message with subject or body 'help' to
>	pyngl-talk-request at ucar.edu
>
>You can reach the person managing the list at
>	pyngl-talk-owner at ucar.edu
>
>When replying, please edit your Subject line so it is more specific
>than "Re: Contents of pyngl-talk digest..."
>
>
>Today's Topics:
>
>   1. Re: how to use the new MRMS grib codetable (David Brown)
>   2. Re: how to use the new MRMS grib codetable (Root, Benjamin)
>   3. Re: how to use the new MRMS grib codetable (David Brown)
>
>
>----------------------------------------------------------------------
>
>Message: 1
>Date: Thu, 2 Jul 2015 13:51:33 -0600
>From: David Brown <dbrown at ucar.edu>
>Subject: Re: [pyngl-talk] how to use the new MRMS grib codetable
>To: "Root, Benjamin" <broot at aer.com>
>Cc: "pyngl-talk at ucar.edu" <pyngl-talk at ucar.edu>, Nick Guy
>	<nick.guy at uwyo.edu>
>Message-ID:
>	<CAC92F7ftgzNGEXxB3ACzENNx1qhi++E43Sv9XL7DGKhGRZkMug at mail.gmail.com>
>Content-Type: text/plain; charset=UTF-8
>
>Hi Ben,
>Coincidentally, there is someone also asking about MRMS data support
>for NCL. I am not sure at this point whether it is just a matter of
>creating the right tables or whether some source code changes will be
>needed as well. But given the amount of interest I will look into this
>now.
> -dave
>
>On Thu, Jul 2, 2015 at 1:25 PM, Root, Benjamin <broot at aer.com> wrote:
>> ...unless one is having difficulties compiling the new wgrib2 package
>> (grrr...).
>>
>> I guess what I am trying to get at is (1) is there a utility that can
>> help convert a wgrib2 codetable into a form recognizable by the ecmwf
>> grib api and (2) how does one get custom codetables recognized by PyNIO?
>>
>> Ben Root
>>
>> On 07/02/2015 03:12 PM, David Brown wrote:
>>> Hi Ben and Nick,
>>> Thanks Nick for the pointer to MMM-Py. From there I can get to
>>> information about the MRMS data, plus some sample data sets.
>>> The key point, relative to PyNIO from this documentation is the
>>>following:
>>>
>>> "The GRIB2 IDs used by MRMS are not officially sanctioned at this
>>> time.  Instead, MRMS uses a local discipline number (209) and it?s own
>>> set of category and parameter values.  This means that most widely
>>> available GRIB2 decoders will require customization."
>>>
>>> MMM-Py relies on a customized version of wgrib2 to do the actual
>>> reading. To make this data readable by PyNIO it will probably be
>>> easiest to work from the modified wgrib2 code. This may be pretty
>>> simple, but it will take a bit of research.
>>>
>>> So for now, I would agree that MMM-Py is probably the best bet.
>>>  -dave
>>>
>>>
>>>
>>>
>>> On Thu, Jul 2, 2015 at 12:14 PM, Nick Guy <Nick.Guy at uwyo.edu> wrote:
>>>> Ben,
>>>>
>>>> Not sure what your use is, but this package works to read in MRMS and
>>>>is
>>>> pretty nice to work with:
>>>> https://github.com/nasa/MMM-Py
>>>>
>>>> Nick
>>>>
>>>> On Jul 2, 2015, at 12:00 PM, pyngl-talk-request at ucar.edu wrote:
>>>>
>>>> Send pyngl-talk mailing list submissions to
>>>> pyngl-talk at ucar.edu
>>>>
>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>>> or, via email, send a message with subject or body 'help' to
>>>> pyngl-talk-request at ucar.edu
>>>>
>>>> You can reach the person managing the list at
>>>> pyngl-talk-owner at ucar.edu
>>>>
>>>> When replying, please edit your Subject line so it is more specific
>>>> than "Re: Contents of pyngl-talk digest..."
>>>>
>>>>
>>>> Today's Topics:
>>>>
>>>>   1. how to use the new MRMS grib codetable (Root, Benjamin)
>>>>
>>>>
>>>> ----------------------------------------------------------------------
>>>>
>>>> Message: 1
>>>> Date: Thu, 2 Jul 2015 16:43:27 +0000
>>>> From: "Root, Benjamin" <broot at aer.com>
>>>> Subject: [pyngl-talk] how to use the new MRMS grib codetable
>>>> To: "pyngl-talk at ucar.edu" <pyngl-talk at ucar.edu>
>>>> Message-ID:
>>>> <47A91F4E74556440877A9E87A31CE9D20952D420 at VRSKUTMB2.veriskdom.com>
>>>> Content-Type: text/plain; charset="us-ascii"
>>>>
>>>> I am working with the (somewhat) new MRMS datafeed from NCEP. I
>>>>managed
>>>> to find a codetable for it in the wgrib2 source code:
>>>> 
>>>>http://www.ftp.cpc.ncep.noaa.gov/wd51we/wgrib2/user_tables/MRMS_usertab
>>>>le,
>>>> but I am stuck in trying to get PyNIO to utilize it. I am not even
>>>>sure
>>>> if it is in the right format for use in PyNIO.
>>>>
>>>> Any tips would be appreciated.
>>>>
>>>> Cheers!
>>>> Ben Root
>>>>
>>>> ________________________________
>>>>
>>>> This email is intended solely for the recipient. It may contain
>>>>privileged,
>>>> proprietary or confidential information or material. If you are not
>>>>the
>>>> intended recipient, please delete this email and any attachments and
>>>>notify
>>>> the sender of the error.
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> _______________________________________________
>>>> pyngl-talk mailing list
>>>> pyngl-talk at ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>>>
>>>>
>>>> End of pyngl-talk Digest, Vol 125, Issue 2
>>>> ******************************************
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> pyngl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>>>
>>
>>
>> ________________________________
>>
>> This email is intended solely for the recipient. It may contain
>>privileged, proprietary or confidential information or material. If you
>>are not the intended recipient, please delete this email and any
>>attachments and notify the sender of the error.
>>
>
>
>------------------------------
>
>Message: 2
>Date: Thu, 2 Jul 2015 21:19:36 +0000
>From: "Root, Benjamin" <broot at aer.com>
>Subject: Re: [pyngl-talk] how to use the new MRMS grib codetable
>To: David Brown <dbrown at ucar.edu>
>Cc: "pyngl-talk at ucar.edu" <pyngl-talk at ucar.edu>, Nick Guy
>	<nick.guy at uwyo.edu>
>Message-ID:
>	<47A91F4E74556440877A9E87A31CE9D20952D686 at VRSKUTMB2.veriskdom.com>
>Content-Type: text/plain; charset="Windows-1252"
>
>That's great to hear! I would just like to point out one particular
>pitfall that is tripping me up. The MRMS data from NCEP's LDM feed are
>all gzipped (yeah, I know, craziness...). Both PyNIO and pygrib expect a
>filepath object to a grib2 file as opposed to a file-like object. I have
>dug down into PyNIO's Grib2 reader, and I see that this requirement
>seems to be fairly artificial, as the relevant C-code just simply opens
>up a standard C file object.
>
>matplotlib has code that can produce a C file object from a Python
>file-like object that you guys can use. We use it to interface with
>libfreetype. It originally came from Numpy so that that library could
>interface with datafile readers that expect a FILE pointer.
>
>https://github.com/matplotlib/matplotlib/blob/master/src/file_compat.h
>
>I don't know yet if this would work for file-like objects such as
>objects from a TarFile or GzipFile, but it might be a step forward. This
>code works on Windows, Linux, and Macs, and is py3k compatible.
>
>Cheers!
>Ben Root
>
>On 07/02/2015 03:51 PM, David Brown wrote:
>> Hi Ben,
>> Coincidentally, there is someone also asking about MRMS data support
>> for NCL. I am not sure at this point whether it is just a matter of
>> creating the right tables or whether some source code changes will be
>> needed as well. But given the amount of interest I will look into this
>> now.
>>  -dave
>>
>> On Thu, Jul 2, 2015 at 1:25 PM, Root, Benjamin <broot at aer.com> wrote:
>>> ...unless one is having difficulties compiling the new wgrib2 package
>>> (grrr...).
>>>
>>> I guess what I am trying to get at is (1) is there a utility that can
>>> help convert a wgrib2 codetable into a form recognizable by the ecmwf
>>> grib api and (2) how does one get custom codetables recognized by
>>>PyNIO?
>>>
>>> Ben Root
>>>
>>> On 07/02/2015 03:12 PM, David Brown wrote:
>>>> Hi Ben and Nick,
>>>> Thanks Nick for the pointer to MMM-Py. From there I can get to
>>>> information about the MRMS data, plus some sample data sets.
>>>> The key point, relative to PyNIO from this documentation is the
>>>>following:
>>>>
>>>> "The GRIB2 IDs used by MRMS are not officially sanctioned at this
>>>> time.  Instead, MRMS uses a local discipline number (209) and it?s own
>>>> set of category and parameter values.  This means that most widely
>>>> available GRIB2 decoders will require customization."
>>>>
>>>> MMM-Py relies on a customized version of wgrib2 to do the actual
>>>> reading. To make this data readable by PyNIO it will probably be
>>>> easiest to work from the modified wgrib2 code. This may be pretty
>>>> simple, but it will take a bit of research.
>>>>
>>>> So for now, I would agree that MMM-Py is probably the best bet.
>>>>  -dave
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jul 2, 2015 at 12:14 PM, Nick Guy <Nick.Guy at uwyo.edu> wrote:
>>>>> Ben,
>>>>>
>>>>> Not sure what your use is, but this package works to read in MRMS
>>>>>and is
>>>>> pretty nice to work with:
>>>>> https://github.com/nasa/MMM-Py
>>>>>
>>>>> Nick
>>>>>
>>>>> On Jul 2, 2015, at 12:00 PM, pyngl-talk-request at ucar.edu wrote:
>>>>>
>>>>> Send pyngl-talk mailing list submissions to
>>>>> pyngl-talk at ucar.edu
>>>>>
>>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>>>> or, via email, send a message with subject or body 'help' to
>>>>> pyngl-talk-request at ucar.edu
>>>>>
>>>>> You can reach the person managing the list at
>>>>> pyngl-talk-owner at ucar.edu
>>>>>
>>>>> When replying, please edit your Subject line so it is more specific
>>>>> than "Re: Contents of pyngl-talk digest..."
>>>>>
>>>>>
>>>>> Today's Topics:
>>>>>
>>>>>   1. how to use the new MRMS grib codetable (Root, Benjamin)
>>>>>
>>>>>
>>>>> 
>>>>>----------------------------------------------------------------------
>>>>>
>>>>> Message: 1
>>>>> Date: Thu, 2 Jul 2015 16:43:27 +0000
>>>>> From: "Root, Benjamin" <broot at aer.com>
>>>>> Subject: [pyngl-talk] how to use the new MRMS grib codetable
>>>>> To: "pyngl-talk at ucar.edu" <pyngl-talk at ucar.edu>
>>>>> Message-ID:
>>>>> <47A91F4E74556440877A9E87A31CE9D20952D420 at VRSKUTMB2.veriskdom.com>
>>>>> Content-Type: text/plain; charset="us-ascii"
>>>>>
>>>>> I am working with the (somewhat) new MRMS datafeed from NCEP. I
>>>>>managed
>>>>> to find a codetable for it in the wgrib2 source code:
>>>>> 
>>>>>http://www.ftp.cpc.ncep.noaa.gov/wd51we/wgrib2/user_tables/MRMS_userta
>>>>>ble,
>>>>> but I am stuck in trying to get PyNIO to utilize it. I am not even
>>>>>sure
>>>>> if it is in the right format for use in PyNIO.
>>>>>
>>>>> Any tips would be appreciated.
>>>>>
>>>>> Cheers!
>>>>> Ben Root
>>>>>
>>>>> ________________________________
>>>>>
>>>>> This email is intended solely for the recipient. It may contain
>>>>>privileged,
>>>>> proprietary or confidential information or material. If you are not
>>>>>the
>>>>> intended recipient, please delete this email and any attachments and
>>>>>notify
>>>>> the sender of the error.
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> _______________________________________________
>>>>> pyngl-talk mailing list
>>>>> pyngl-talk at ucar.edu
>>>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>>>>
>>>>>
>>>>> End of pyngl-talk Digest, Vol 125, Issue 2
>>>>> ******************************************
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> pyngl-talk mailing list
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>>>>
>>>
>>> ________________________________
>>>
>>> This email is intended solely for the recipient. It may contain
>>>privileged, proprietary or confidential information or material. If you
>>>are not the intended recipient, please delete this email and any
>>>attachments and notify the sender of the error.
>>>
>
>
>
>
>------------------------------
>
>Message: 3
>Date: Thu, 2 Jul 2015 15:29:44 -0600
>From: David Brown <dbrown at ucar.edu>
>Subject: Re: [pyngl-talk] how to use the new MRMS grib codetable
>To: "Root, Benjamin" <broot at aer.com>
>Cc: "pyngl-talk at ucar.edu" <pyngl-talk at ucar.edu>, Nick Guy
>	<nick.guy at uwyo.edu>
>Message-ID:
>	<CAC92F7dJPSWgPWegOmySV_V+D1ge=wNN2NvJDhmqaWOWbFmvoQ at mail.gmail.com>
>Content-Type: text/plain; charset=UTF-8
>
>OK, this is a pretty different issue, but I would definitely like to
>be able to handle file-like objects as you propose. We will definitely
>put this on the to-do list.
> -dave
>
>
>On Thu, Jul 2, 2015 at 3:19 PM, Root, Benjamin <broot at aer.com> wrote:
>> That's great to hear! I would just like to point out one particular
>> pitfall that is tripping me up. The MRMS data from NCEP's LDM feed are
>> all gzipped (yeah, I know, craziness...). Both PyNIO and pygrib expect a
>> filepath object to a grib2 file as opposed to a file-like object. I have
>> dug down into PyNIO's Grib2 reader, and I see that this requirement
>> seems to be fairly artificial, as the relevant C-code just simply opens
>> up a standard C file object.
>>
>> matplotlib has code that can produce a C file object from a Python
>> file-like object that you guys can use. We use it to interface with
>> libfreetype. It originally came from Numpy so that that library could
>> interface with datafile readers that expect a FILE pointer.
>>
>> https://github.com/matplotlib/matplotlib/blob/master/src/file_compat.h
>>
>> I don't know yet if this would work for file-like objects such as
>> objects from a TarFile or GzipFile, but it might be a step forward. This
>> code works on Windows, Linux, and Macs, and is py3k compatible.
>>
>> Cheers!
>> Ben Root
>>
>> On 07/02/2015 03:51 PM, David Brown wrote:
>>> Hi Ben,
>>> Coincidentally, there is someone also asking about MRMS data support
>>> for NCL. I am not sure at this point whether it is just a matter of
>>> creating the right tables or whether some source code changes will be
>>> needed as well. But given the amount of interest I will look into this
>>> now.
>>>  -dave
>>>
>>> On Thu, Jul 2, 2015 at 1:25 PM, Root, Benjamin <broot at aer.com> wrote:
>>>> ...unless one is having difficulties compiling the new wgrib2 package
>>>> (grrr...).
>>>>
>>>> I guess what I am trying to get at is (1) is there a utility that can
>>>> help convert a wgrib2 codetable into a form recognizable by the ecmwf
>>>> grib api and (2) how does one get custom codetables recognized by
>>>>PyNIO?
>>>>
>>>> Ben Root
>>>>
>>>> On 07/02/2015 03:12 PM, David Brown wrote:
>>>>> Hi Ben and Nick,
>>>>> Thanks Nick for the pointer to MMM-Py. From there I can get to
>>>>> information about the MRMS data, plus some sample data sets.
>>>>> The key point, relative to PyNIO from this documentation is the
>>>>>following:
>>>>>
>>>>> "The GRIB2 IDs used by MRMS are not officially sanctioned at this
>>>>> time.  Instead, MRMS uses a local discipline number (209) and it?s
>>>>>own
>>>>> set of category and parameter values.  This means that most widely
>>>>> available GRIB2 decoders will require customization."
>>>>>
>>>>> MMM-Py relies on a customized version of wgrib2 to do the actual
>>>>> reading. To make this data readable by PyNIO it will probably be
>>>>> easiest to work from the modified wgrib2 code. This may be pretty
>>>>> simple, but it will take a bit of research.
>>>>>
>>>>> So for now, I would agree that MMM-Py is probably the best bet.
>>>>>  -dave
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 2, 2015 at 12:14 PM, Nick Guy <Nick.Guy at uwyo.edu> wrote:
>>>>>> Ben,
>>>>>>
>>>>>> Not sure what your use is, but this package works to read in MRMS
>>>>>>and is
>>>>>> pretty nice to work with:
>>>>>> https://github.com/nasa/MMM-Py
>>>>>>
>>>>>> Nick
>>>>>>
>>>>>> On Jul 2, 2015, at 12:00 PM, pyngl-talk-request at ucar.edu wrote:
>>>>>>
>>>>>> Send pyngl-talk mailing list submissions to
>>>>>> pyngl-talk at ucar.edu
>>>>>>
>>>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>>>>> or, via email, send a message with subject or body 'help' to
>>>>>> pyngl-talk-request at ucar.edu
>>>>>>
>>>>>> You can reach the person managing the list at
>>>>>> pyngl-talk-owner at ucar.edu
>>>>>>
>>>>>> When replying, please edit your Subject line so it is more specific
>>>>>> than "Re: Contents of pyngl-talk digest..."
>>>>>>
>>>>>>
>>>>>> Today's Topics:
>>>>>>
>>>>>>   1. how to use the new MRMS grib codetable (Root, Benjamin)
>>>>>>
>>>>>>
>>>>>> 
>>>>>>---------------------------------------------------------------------
>>>>>>-
>>>>>>
>>>>>> Message: 1
>>>>>> Date: Thu, 2 Jul 2015 16:43:27 +0000
>>>>>> From: "Root, Benjamin" <broot at aer.com>
>>>>>> Subject: [pyngl-talk] how to use the new MRMS grib codetable
>>>>>> To: "pyngl-talk at ucar.edu" <pyngl-talk at ucar.edu>
>>>>>> Message-ID:
>>>>>> <47A91F4E74556440877A9E87A31CE9D20952D420 at VRSKUTMB2.veriskdom.com>
>>>>>> Content-Type: text/plain; charset="us-ascii"
>>>>>>
>>>>>> I am working with the (somewhat) new MRMS datafeed from NCEP. I
>>>>>>managed
>>>>>> to find a codetable for it in the wgrib2 source code:
>>>>>> 
>>>>>>http://www.ftp.cpc.ncep.noaa.gov/wd51we/wgrib2/user_tables/MRMS_usert
>>>>>>able,
>>>>>> but I am stuck in trying to get PyNIO to utilize it. I am not even
>>>>>>sure
>>>>>> if it is in the right format for use in PyNIO.
>>>>>>
>>>>>> Any tips would be appreciated.
>>>>>>
>>>>>> Cheers!
>>>>>> Ben Root
>>>>>>
>>>>>> ________________________________
>>>>>>
>>>>>> This email is intended solely for the recipient. It may contain
>>>>>>privileged,
>>>>>> proprietary or confidential information or material. If you are not
>>>>>>the
>>>>>> intended recipient, please delete this email and any attachments
>>>>>>and notify
>>>>>> the sender of the error.
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> pyngl-talk mailing list
>>>>>> pyngl-talk at ucar.edu
>>>>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>>>>>
>>>>>>
>>>>>> End of pyngl-talk Digest, Vol 125, Issue 2
>>>>>> ******************************************
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> pyngl-talk mailing list
>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>>>>>
>>>>
>>>> ________________________________
>>>>
>>>> This email is intended solely for the recipient. It may contain
>>>>privileged, proprietary or confidential information or material. If
>>>>you are not the intended recipient, please delete this email and any
>>>>attachments and notify the sender of the error.
>>>>
>>
>>
>
>
>------------------------------
>
>_______________________________________________
>pyngl-talk mailing list
>pyngl-talk at ucar.edu
>http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>
>
>End of pyngl-talk Digest, Vol 125, Issue 4
>******************************************

More information about the pyngl-talk mailing list