[Met_help] [rt.rap.ucar.edu #90404] History for Python Embedding
John Halley Gotway via RT
met_help at ucar.edu
Wed Jul 10 17:01:14 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Moving this request for help into met_help - Tara
- From Taylor Mandelbaum -
Hey all,
Thanks for the information. I'm trying to wrap my head around the whole procedure using python embedding; it's a bit complex. I had a few questions that maybe you can help me with:
1. Is there a specific reason PostScript files are used as the output?
2. For the attr(s) dict, it seems like a quasi-schema - maybe there's another way (perhaps via a js or json) that could be used and stored separately in a dedicated folder? I think that there's a huge missing piece in the atmospheric science community with regard to proper data ingestion. Pulling schemas from the data engineering world (for extract-transform-load or ETL) and applying them here for users might be valuable.
3. Does PYTHON_NUMPY need to be declared? Is there a way for the code to identify if the output is coming from a py file?
4. Why do MET_PYTHON_CC and MET_PYTHON_LD both need to be declared? Is that so MET can access python's source code for the process?
5. If I'm understanding correctly, MET will find the attr(s) variable and met_data within the python file and thus it doesn't need to be returned in a function?
6. I noticed in the docs that there's a -title flag. Is there documentation on all flags?
Using one of the python script examples:
Plot VIIRS AOD data from a NetCDF file:
plot_data_plane PYTHON_NUMPY viirs_aod.ps
'name="read_VIIRS_AOD_NetCDF.py
npp_eaot_ip_gridded_0.25_20181012.high.nc aot_ip_out";'
1. Can I save it as something other than a .ps? Such as a .png or .pdf?
2. Does the command need to be invoked in the same folder as the .py and data file?
3. Can the python embedding be used on other MET commands other than plot_data_plane? If so, are there tutorials on it? I think the challenge here is that the python files need to be structured in a specific way with the attrs and met_data files explicitly declared, but it might be challenging to figure out how to structure them in the script for different commands.
>From a more fundamental perspective, what benefit does this method, and MET for example, have over building a python CLI tool with a flow such as:
plot_data -filename:'gefs-legacy-00z_rfcst-cal_tmean_20170807_8-14day.bin' -record:10
As there are sufficient python libraries to handle most (if not all) filetypes that encapsulate common meteorological formats?
One of the massive strengths of python is using the community of open source resources to build apps (command line, GUI, web app, etc) that can be created rapidly, debugged rapidly, and easily invoked. I am curious to see if we can simplify MET's processes using python as more of an abstraction wrapper. If I got anything wrong or missed anything, please let me know. I'm still working on learning the way MET works and I could have overlooked something I shouldn't have.
Thanks again!
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Python Embedding
From: Minna Win
Time: Thu May 30 16:03:30 2019
Hi Taylor,
It looks like you have numerous questions about Python embedding in
MET. Randy Bullock is the best person to address your questions. He
is currently out of the office this week, so please allow a few more
business days to get a response. Thanks in advance for your patience.
Regards,
Minna
------------------------------------------------
Subject: Python Embedding
From: Minna Win
Time: Thu May 30 16:04:57 2019
Hi Taylor,
It looks like you have numerous questions about Python embedding in
MET. Randy Bullock is the best person to address your questions. He is
currently out of the office this week, so please allow a few more
business days to get a response. Thanks in advance for your patience.
Regards,
Minna
------------------------------------------------
Subject: Python Embedding
From: Randy Bullock
Time: Tue Jun 04 13:58:40 2019
Hi Taylor -
Let me see if I can take a stab at your questions:
1. Is there a specific reason PostScript files are used as the output?
Answer: Back when MET was first being put together, we decided that we
didn't want to use any proprietary graphics systems like matlab. The
postscript graphics library had already been written (by me) so it was
handy, and we figured that postscript viewers were common in the unix
world. As long as there was one postscript laser printer SOMEWHERE in
the
user's building, then he/she could print out the plots. Also at that
time,
pdf had not yet taken over the electronic document world to the extent
it
has today.
2. For the attr(s) dict, it seems like a quasi-schema - maybe there's
another way (perhaps via a js or json) that could be used and stored
separately in a dedicated folder? I think that there's a huge missing
piece
in the atmospheric science community with regard to proper data
ingestion.
Pulling schemas from the data engineering world (for extract-
transform-load
or ETL) and applying them here for users might be valuable.
Answer: It might be valuable. It might be a lot of work, too. Who
is
going to pay us to do that? You have to remember that if a user asks
us to
implement some improvement in MET such as new statistics or file
formats
(or data schemas, for that matter), then (a) one of our team members
has to
become an expert (or nearly so) in that thing, because a lot of users
will
have questions about it and will expect us to have the answers for
them,
and (b) someone will have to pay us to do it. There are a lot of
things
I'd love to see added to MET, but nobody wants to write us a check.
3. Does PYTHON_NUMPY need to be declared? Is there a way for the code
to
identify if the output is coming from a py file?
Answer: We tried several ways of doing this, but this one seemed the
most
robust.
4. Why do MET_PYTHON_CC and MET_PYTHON_LD both need to be declared? Is
that
so MET can access python's source code for the process?
Answer: Basically, yes.
5. If I'm understanding correctly, MET will find the attr(s) variable
and
met_data within the python file and thus it doesn't need to be
returned in
a function?
Answer: that's correct. When MET runs a python script, it has access
to
the (python) global namespace, and can find the attrs dictionary
there.
6. I noticed in the docs that there's a -title flag. Is there
documentation on all flags?
Answer: I assume you're talking about the plot_data_plane tool. The
title
switch just controls some annotation for the plot. There should be
explanations for all of the command-line switches for all of the MET
tools.
[Note: I looks like you started your numbering over at 1, here.]
1. Can I save it as something other than a .ps? Such as a .png or
.pdf?
Answer: Once you have a .ps file, you can (on unix-like systems) use
the
command "/usr/bin/ps2pdf -dPDFSETTINGS=/prepress" command to convert
it
into a pdf, if you want. Also there are many tools from which you can
create a raster graphics file as a screengrab from a document viewer.
Be
careful wiith these, however, because if you don't know what you're
doing,
you can end up with a result that is of very poor quality.
2. Does the command need to be invoked in the same folder as the .py
and
data file?
Answer: No, but it's usually helpful if the directory where your .py
file
lives is listed in your PTHONPATH environment variable.
3. Can the python embedding be used on other MET commands other than
plot_data_plane? If so, are there tutorials on it? I think the
challenge
here is that the python files need to be structured in a specific way
with
the attrs and met_data files explicitly declared, but it might be
challenging to figure out how to structure them in the script for
different
commands.
Answer: Currently there are just a few MET tools where the python
embedding works. In our next release, the python embedding will be
extended to all of the MET tools where it would make sense to do so.
-----------------
One more thing: I'm not sure I can answer your Python CLI question at
this
time. The whole "MET python embedding" thing is so new that it's
still
somewhat fluid. As we get deeper into this I'm sure we'll find better
ways
to do things.
Hope this helps.
Randy Bullock
On Thu, May 30, 2019 at 3:48 PM Tara Jensen via RT <met_help at ucar.edu>
wrote:
>
> Thu May 30 15:35:52 2019: Request 90404 was acted upon.
> Transaction: Ticket created by jensen
> Queue: met_help
> Subject: Python Embedding
> Owner: Nobody
> Requestors: taylor.mandelbaum at stonybrook.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90404 >
>
>
> Moving this request for help into met_help - Tara
>
> - From Taylor Mandelbaum -
> Hey all,
>
> Thanks for the information. I'm trying to wrap my head around the
whole
> procedure using python embedding; it's a bit complex. I had a few
questions
> that maybe you can help me with:
>
> 1. Is there a specific reason PostScript files are used as the
output?
> 2. For the attr(s) dict, it seems like a quasi-schema - maybe
there's
> another way (perhaps via a js or json) that could be used and stored
> separately in a dedicated folder? I think that there's a huge
missing piece
> in the atmospheric science community with regard to proper data
ingestion.
> Pulling schemas from the data engineering world (for extract-
transform-load
> or ETL) and applying them here for users might be valuable.
> 3. Does PYTHON_NUMPY need to be declared? Is there a way for the
code to
> identify if the output is coming from a py file?
> 4. Why do MET_PYTHON_CC and MET_PYTHON_LD both need to be declared?
Is
> that so MET can access python's source code for the process?
> 5. If I'm understanding correctly, MET will find the attr(s)
variable and
> met_data within the python file and thus it doesn't need to be
returned in
> a function?
> 6. I noticed in the docs that there's a -title flag. Is there
> documentation on all flags?
>
> Using one of the python script examples:
>
> Plot VIIRS AOD data from a NetCDF file:
>
> plot_data_plane PYTHON_NUMPY viirs_aod.ps
> 'name="read_VIIRS_AOD_NetCDF.py
> npp_eaot_ip_gridded_0.25_20181012.high.nc aot_ip_out";'
>
> 1. Can I save it as something other than a .ps? Such as a .png or
.pdf?
> 2. Does the command need to be invoked in the same folder as the .py
and
> data file?
> 3. Can the python embedding be used on other MET commands other than
> plot_data_plane? If so, are there tutorials on it? I think the
challenge
> here is that the python files need to be structured in a specific
way with
> the attrs and met_data files explicitly declared, but it might be
> challenging to figure out how to structure them in the script for
different
> commands.
>
> From a more fundamental perspective, what benefit does this method,
and
> MET for example, have over building a python CLI tool with a flow
such as:
>
> plot_data -filename:'gefs-legacy-00z_rfcst-cal_tmean_20170807_8-
14day.bin'
> -record:10
>
> As there are sufficient python libraries to handle most (if not all)
> filetypes that encapsulate common meteorological formats?
>
> One of the massive strengths of python is using the community of
open
> source resources to build apps (command line, GUI, web app, etc)
that can
> be created rapidly, debugged rapidly, and easily invoked. I am
curious to
> see if we can simplify MET's processes using python as more of an
> abstraction wrapper. If I got anything wrong or missed anything,
please let
> me know. I'm still working on learning the way MET works and I could
have
> overlooked something I shouldn't have.
>
> Thanks again!
>
------------------------------------------------
Subject: Python Embedding
From: Taylor Mandelbaum
Time: Mon Jun 17 19:25:28 2019
Thanks for all of that Randy, apologies for the delayed response. If I
have
any more questions, which I definitely will eventually, I'll reach out
again.
On Tue, Jun 4, 2019 at 3:58 PM Randy Bullock via RT
<met_help at ucar.edu>
wrote:
> Hi Taylor -
>
> Let me see if I can take a stab at your questions:
>
> 1. Is there a specific reason PostScript files are used as the
output?
>
> Answer: Back when MET was first being put together, we decided that
we
> didn't want to use any proprietary graphics systems like matlab.
The
> postscript graphics library had already been written (by me) so it
was
> handy, and we figured that postscript viewers were common in the
unix
> world. As long as there was one postscript laser printer SOMEWHERE
in the
> user's building, then he/she could print out the plots. Also at
that time,
> pdf had not yet taken over the electronic document world to the
extent it
> has today.
>
> 2. For the attr(s) dict, it seems like a quasi-schema - maybe
there's
> another way (perhaps via a js or json) that could be used and stored
> separately in a dedicated folder? I think that there's a huge
missing piece
> in the atmospheric science community with regard to proper data
ingestion.
> Pulling schemas from the data engineering world (for extract-
transform-load
> or ETL) and applying them here for users might be valuable.
>
> Answer: It might be valuable. It might be a lot of work, too. Who
is
> going to pay us to do that? You have to remember that if a user
asks us to
> implement some improvement in MET such as new statistics or file
formats
> (or data schemas, for that matter), then (a) one of our team members
has to
> become an expert (or nearly so) in that thing, because a lot of
users will
> have questions about it and will expect us to have the answers for
them,
> and (b) someone will have to pay us to do it. There are a lot of
things
> I'd love to see added to MET, but nobody wants to write us a check.
>
> 3. Does PYTHON_NUMPY need to be declared? Is there a way for the
code to
> identify if the output is coming from a py file?
>
> Answer: We tried several ways of doing this, but this one seemed
the most
> robust.
>
> 4. Why do MET_PYTHON_CC and MET_PYTHON_LD both need to be declared?
Is that
> so MET can access python's source code for the process?
>
> Answer: Basically, yes.
>
> 5. If I'm understanding correctly, MET will find the attr(s)
variable and
> met_data within the python file and thus it doesn't need to be
returned in
> a function?
>
> Answer: that's correct. When MET runs a python script, it has
access to
> the (python) global namespace, and can find the attrs dictionary
there.
>
> 6. I noticed in the docs that there's a -title flag. Is there
> documentation on all flags?
>
> Answer: I assume you're talking about the plot_data_plane tool.
The title
> switch just controls some annotation for the plot. There should be
> explanations for all of the command-line switches for all of the MET
tools.
>
> [Note: I looks like you started your numbering over at 1, here.]
>
> 1. Can I save it as something other than a .ps? Such as a .png or
.pdf?
>
> Answer: Once you have a .ps file, you can (on unix-like systems)
use the
> command "/usr/bin/ps2pdf -dPDFSETTINGS=/prepress" command to convert
it
> into a pdf, if you want. Also there are many tools from which you
can
> create a raster graphics file as a screengrab from a document
viewer. Be
> careful wiith these, however, because if you don't know what you're
doing,
> you can end up with a result that is of very poor quality.
>
> 2. Does the command need to be invoked in the same folder as the .py
and
> data file?
>
> Answer: No, but it's usually helpful if the directory where your
.py file
> lives is listed in your PTHONPATH environment variable.
>
> 3. Can the python embedding be used on other MET commands other than
> plot_data_plane? If so, are there tutorials on it? I think the
challenge
> here is that the python files need to be structured in a specific
way with
> the attrs and met_data files explicitly declared, but it might be
> challenging to figure out how to structure them in the script for
different
> commands.
>
> Answer: Currently there are just a few MET tools where the python
> embedding works. In our next release, the python embedding will be
> extended to all of the MET tools where it would make sense to do so.
>
> -----------------
>
> One more thing: I'm not sure I can answer your Python CLI question
at this
> time. The whole "MET python embedding" thing is so new that it's
still
> somewhat fluid. As we get deeper into this I'm sure we'll find
better ways
> to do things.
>
> Hope this helps.
>
> Randy Bullock
>
>
> On Thu, May 30, 2019 at 3:48 PM Tara Jensen via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Thu May 30 15:35:52 2019: Request 90404 was acted upon.
> > Transaction: Ticket created by jensen
> > Queue: met_help
> > Subject: Python Embedding
> > Owner: Nobody
> > Requestors: taylor.mandelbaum at stonybrook.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90404 >
> >
> >
> > Moving this request for help into met_help - Tara
> >
> > - From Taylor Mandelbaum -
> > Hey all,
> >
> > Thanks for the information. I'm trying to wrap my head around the
whole
> > procedure using python embedding; it's a bit complex. I had a few
> questions
> > that maybe you can help me with:
> >
> > 1. Is there a specific reason PostScript files are used as the
output?
> > 2. For the attr(s) dict, it seems like a quasi-schema - maybe
there's
> > another way (perhaps via a js or json) that could be used and
stored
> > separately in a dedicated folder? I think that there's a huge
missing
> piece
> > in the atmospheric science community with regard to proper data
> ingestion.
> > Pulling schemas from the data engineering world (for
> extract-transform-load
> > or ETL) and applying them here for users might be valuable.
> > 3. Does PYTHON_NUMPY need to be declared? Is there a way for the
code to
> > identify if the output is coming from a py file?
> > 4. Why do MET_PYTHON_CC and MET_PYTHON_LD both need to be
declared? Is
> > that so MET can access python's source code for the process?
> > 5. If I'm understanding correctly, MET will find the attr(s)
variable and
> > met_data within the python file and thus it doesn't need to be
returned
> in
> > a function?
> > 6. I noticed in the docs that there's a -title flag. Is there
> > documentation on all flags?
> >
> > Using one of the python script examples:
> >
> > Plot VIIRS AOD data from a NetCDF file:
> >
> > plot_data_plane PYTHON_NUMPY viirs_aod.ps
> > 'name="read_VIIRS_AOD_NetCDF.py
> > npp_eaot_ip_gridded_0.25_20181012.high.nc aot_ip_out";'
> >
> > 1. Can I save it as something other than a .ps? Such as a .png or
.pdf?
> > 2. Does the command need to be invoked in the same folder as the
.py and
> > data file?
> > 3. Can the python embedding be used on other MET commands other
than
> > plot_data_plane? If so, are there tutorials on it? I think the
challenge
> > here is that the python files need to be structured in a specific
way
> with
> > the attrs and met_data files explicitly declared, but it might be
> > challenging to figure out how to structure them in the script for
> different
> > commands.
> >
> > From a more fundamental perspective, what benefit does this
method, and
> > MET for example, have over building a python CLI tool with a flow
such
> as:
> >
> > plot_data
> -filename:'gefs-legacy-00z_rfcst-cal_tmean_20170807_8-14day.bin'
> > -record:10
> >
> > As there are sufficient python libraries to handle most (if not
all)
> > filetypes that encapsulate common meteorological formats?
> >
> > One of the massive strengths of python is using the community of
open
> > source resources to build apps (command line, GUI, web app, etc)
that can
> > be created rapidly, debugged rapidly, and easily invoked. I am
curious to
> > see if we can simplify MET's processes using python as more of an
> > abstraction wrapper. If I got anything wrong or missed anything,
please
> let
> > me know. I'm still working on learning the way MET works and I
could have
> > overlooked something I shouldn't have.
> >
> > Thanks again!
> >
>
>
--
*Taylor Mandelbaum*
Research Technician
SoMAS | Stony Brook University
678.468.5789
------------------------------------------------
More information about the Met_help
mailing list