[Met_help] [rt.rap.ucar.edu #87402] History for MET in docker-container on DoD HPC (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Wed Oct 17 08:08:42 MDT 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
CLASSIFICATION: UNCLASSIFIED
Recent development from my HPC resource indicates the possibility of using Singularity to enable the use of containerized applications, but the question arises about whether the container needs to be modified to run in a non-root environment. If not, then the guidance is that the user needs only to download and copy the container to the HPC. Singularity provides the way to deploy and run containers on the HPC. Could you clarify the situation regards the ability of MET V8.0 to run in a non-root environment and, if not, what alternatives there are?
Thanks.
R/
John
CLASSIFICATION: UNCLASSIFIED
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: MET in docker-container on DoD HPC (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Oct 16 13:28:22 2018
John,
That's interesting that you may be able to run Singularity on your HPC
machine.
We taught a short-course at AMS last year and will be doing so again
this
year in Phoenix. It's on the Saturday before AMS begins. We'll be
working
through the steps of running an end-to-end NWP workflow using Docker
containers. The online instructions from last year can be found here:
https://dtcenter.org/met/docker-nwp/tutorial/container_nwp_tutorial
Singularity is different from Docker. However, it's supposed to be
pretty
straight-forward to start with a "Dockerfile", which defines the
Docker
image, and use that to create a Singularity container. In fact, one
of my
co-workers is working on that now.
Here's where you'd want to start. This is the github repo containing
the
Dockerfile for met-8.0:
https://github.com/NCAR/container-dtc-met/tree/v8.0
Hope that helps.
Thanks,
John
On Tue, Oct 16, 2018 at 9:50 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Tue Oct 16 09:49:43 2018: Request 87402 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET in docker-container on DoD HPC (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> Recent development from my HPC resource indicates the possibility of
using
> Singularity to enable the use of containerized applications, but the
> question arises about whether the container needs to be modified to
run in
> a non-root environment. If not, then the guidance is that the user
needs
> only to download and copy the container to the HPC. Singularity
provides
> the way to deploy and run containers on the HPC. Could you clarify
the
> situation regards the ability of MET V8.0 to run in a non-root
environment
> and, if not, what alternatives there are?
>
> Thanks.
>
> R/
> John
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in docker-container on DoD HPC (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Oct 16 13:45:38 2018
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for the info. Singularity isn't running yet, but is anticipated
by next DEC. Just trying to understand how it might be used to enable
the installation of MET in a non-root environment. I would be
interested in learning more about your development work for a
Singularity container since that appears to be the most promising
possibility.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, October 16, 2018 1:28 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in docker-
container on DoD HPC (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
That's interesting that you may be able to run Singularity on your HPC
machine.
We taught a short-course at AMS last year and will be doing so again
this year in Phoenix. It's on the Saturday before AMS begins. We'll
be working through the steps of running an end-to-end NWP workflow
using Docker containers. The online instructions from last year can
be found here:
Caution-https://dtcenter.org/met/docker-
nwp/tutorial/container_nwp_tutorial
Singularity is different from Docker. However, it's supposed to be
pretty straight-forward to start with a "Dockerfile", which defines
the Docker image, and use that to create a Singularity container. In
fact, one of my co-workers is working on that now.
Here's where you'd want to start. This is the github repo containing
the Dockerfile for met-8.0:
Caution-https://github.com/NCAR/container-dtc-met/tree/v8.0
Hope that helps.
Thanks,
John
On Tue, Oct 16, 2018 at 9:50 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Tue Oct 16 09:49:43 2018: Request 87402 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET in docker-container on DoD HPC (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> Recent development from my HPC resource indicates the possibility of
> using Singularity to enable the use of containerized applications,
but
> the question arises about whether the container needs to be modified
> to run in a non-root environment. If not, then the guidance is that
> the user needs only to download and copy the container to the HPC.
> Singularity provides the way to deploy and run containers on the
HPC.
> Could you clarify the situation regards the ability of MET V8.0 to
run
> in a non-root environment and, if not, what alternatives there are?
>
> Thanks.
>
> R/
> John
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in docker-container on DoD HPC (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Oct 16 13:51:59 2018
CLASSIFICATION: UNCLASSIFIED
John -
For installing MET V8.0 in a docker-container on windows, do I have to
install Docker first and then download the docker file at:
https://github.com/NCAR/container-dtc-met/tree/v8.0
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, October 16, 2018 1:28 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in docker-
container on DoD HPC (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
That's interesting that you may be able to run Singularity on your HPC
machine.
We taught a short-course at AMS last year and will be doing so again
this year in Phoenix. It's on the Saturday before AMS begins. We'll
be working through the steps of running an end-to-end NWP workflow
using Docker containers. The online instructions from last year can
be found here:
Caution-https://dtcenter.org/met/docker-
nwp/tutorial/container_nwp_tutorial
Singularity is different from Docker. However, it's supposed to be
pretty straight-forward to start with a "Dockerfile", which defines
the Docker image, and use that to create a Singularity container. In
fact, one of my co-workers is working on that now.
Here's where you'd want to start. This is the github repo containing
the Dockerfile for met-8.0:
Caution-https://github.com/NCAR/container-dtc-met/tree/v8.0
Hope that helps.
Thanks,
John
On Tue, Oct 16, 2018 at 9:50 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Tue Oct 16 09:49:43 2018: Request 87402 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET in docker-container on DoD HPC (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> Recent development from my HPC resource indicates the possibility of
> using Singularity to enable the use of containerized applications,
but
> the question arises about whether the container needs to be modified
> to run in a non-root environment. If not, then the guidance is that
> the user needs only to download and copy the container to the HPC.
> Singularity provides the way to deploy and run containers on the
HPC.
> Could you clarify the situation regards the ability of MET V8.0 to
run
> in a non-root environment and, if not, what alternatives there are?
>
> Thanks.
>
> R/
> John
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in docker-container on DoD HPC (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Oct 16 14:05:31 2018
CLASSIFICATION: UNCLASSIFIED
John - For windows 10, the Docker sw I find using the link provided
at:
https://dtcenter.org/met/users/downloads/docker_container/METv8.0_docker_container.php
is called Docker Desktop and is described at:
https://www.docker.com/products/windows-containers
Is this the sw I need to purchase?
Thanks.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, October 16, 2018 1:28 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in docker-
container on DoD HPC (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
That's interesting that you may be able to run Singularity on your HPC
machine.
We taught a short-course at AMS last year and will be doing so again
this year in Phoenix. It's on the Saturday before AMS begins. We'll
be working through the steps of running an end-to-end NWP workflow
using Docker containers. The online instructions from last year can
be found here:
Caution-https://dtcenter.org/met/docker-
nwp/tutorial/container_nwp_tutorial
Singularity is different from Docker. However, it's supposed to be
pretty straight-forward to start with a "Dockerfile", which defines
the Docker image, and use that to create a Singularity container. In
fact, one of my co-workers is working on that now.
Here's where you'd want to start. This is the github repo containing
the Dockerfile for met-8.0:
Caution-https://github.com/NCAR/container-dtc-met/tree/v8.0
Hope that helps.
Thanks,
John
On Tue, Oct 16, 2018 at 9:50 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Tue Oct 16 09:49:43 2018: Request 87402 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET in docker-container on DoD HPC (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> Recent development from my HPC resource indicates the possibility of
> using Singularity to enable the use of containerized applications,
but
> the question arises about whether the container needs to be modified
> to run in a non-root environment. If not, then the guidance is that
> the user needs only to download and copy the container to the HPC.
> Singularity provides the way to deploy and run containers on the
HPC.
> Could you clarify the situation regards the ability of MET V8.0 to
run
> in a non-root environment and, if not, what alternatives there are?
>
> Thanks.
>
> R/
> John
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET in docker-container on DoD HPC (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Oct 16 14:15:31 2018
John,
I have only run Docker on a Mac laptop myself. Tara Jensen has run it
on
Windows. We've found that running on Windows does work but we did run
into
a few more headaches.
Docker should be a free download. If you're running Windows 10
professional or better than that'll work fine. If not, you'd use the
older
Docker toolbox. You should not need to purchase any software.
The catch is that you'll need root access on your laptop to
install/run
Docker. That may be fine for testing but is problematic on HPC, as
you
know. But I do think that working through that online tutorial is a
great
way to get familiar with containers.
Thanks,
John
On Tue, Oct 16, 2018 at 2:06 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John - For windows 10, the Docker sw I find using the link provided
at:
>
>
https://dtcenter.org/met/users/downloads/docker_container/METv8.0_docker_container.php
>
> is called Docker Desktop and is described at:
> https://www.docker.com/products/windows-containers
>
> Is this the sw I need to purchase?
>
> Thanks.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, October 16, 2018 1:28 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> docker-container on DoD HPC (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> That's interesting that you may be able to run Singularity on your
HPC
> machine.
>
> We taught a short-course at AMS last year and will be doing so again
this
> year in Phoenix. It's on the Saturday before AMS begins. We'll be
working
> through the steps of running an end-to-end NWP workflow using Docker
> containers. The online instructions from last year can be found
here:
> Caution-
> https://dtcenter.org/met/docker-nwp/tutorial/container_nwp_tutorial
>
> Singularity is different from Docker. However, it's supposed to be
pretty
> straight-forward to start with a "Dockerfile", which defines the
Docker
> image, and use that to create a Singularity container. In fact, one
of my
> co-workers is working on that now.
>
> Here's where you'd want to start. This is the github repo
containing the
> Dockerfile for met-8.0:
> Caution-https://github.com/NCAR/container-dtc-met/tree/v8.0
>
> Hope that helps.
>
> Thanks,
> John
>
> On Tue, Oct 16, 2018 at 9:50 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Oct 16 09:49:43 2018: Request 87402 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET in docker-container on DoD HPC (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Recent development from my HPC resource indicates the possibility
of
> > using Singularity to enable the use of containerized applications,
but
> > the question arises about whether the container needs to be
modified
> > to run in a non-root environment. If not, then the guidance is
that
> > the user needs only to download and copy the container to the HPC.
> > Singularity provides the way to deploy and run containers on the
HPC.
> > Could you clarify the situation regards the ability of MET V8.0 to
run
> > in a non-root environment and, if not, what alternatives there
are?
> >
> > Thanks.
> >
> > R/
> > John
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in docker-container on DoD HPC (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Oct 16 15:13:06 2018
CLASSIFICATION: UNCLASSIFIED
John -
I've asked the question of our windows 10 system admin regarding the
requirement for users to have root access in order to run the
containerized MET and I'll find out what their ruling is. Is
Singularity a possibility for running on windows to resolve the issue?
Will MET require modification in order to run with Singularity in a
non-root environment on the HPC?
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, October 16, 2018 2:16 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
docker-container on DoD HPC (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
I have only run Docker on a Mac laptop myself. Tara Jensen has run it
on Windows. We've found that running on Windows does work but we did
run into a few more headaches.
Docker should be a free download. If you're running Windows 10
professional or better than that'll work fine. If not, you'd use the
older Docker toolbox. You should not need to purchase any software.
The catch is that you'll need root access on your laptop to
install/run Docker. That may be fine for testing but is problematic
on HPC, as you know. But I do think that working through that online
tutorial is a great way to get familiar with containers.
Thanks,
John
On Tue, Oct 16, 2018 at 2:06 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John - For windows 10, the Docker sw I find using the link provided
at:
>
> Caution-
https://dtcenter.org/met/users/downloads/docker_container/METv
> 8.0_docker_container.php
>
> is called Docker Desktop and is described at:
> Caution-https://www.docker.com/products/windows-containers
>
> Is this the sw I need to purchase?
>
> Thanks.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Tuesday, October 16, 2018 1:28 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> docker-container on DoD HPC (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> That's interesting that you may be able to run Singularity on your
HPC
> machine.
>
> We taught a short-course at AMS last year and will be doing so again
> this year in Phoenix. It's on the Saturday before AMS begins.
We'll
> be working through the steps of running an end-to-end NWP workflow
> using Docker containers. The online instructions from last year can
be found here:
> Caution-
> Caution-https://dtcenter.org/met/docker-
nwp/tutorial/container_nwp_tut
> orial
>
> Singularity is different from Docker. However, it's supposed to be
> pretty straight-forward to start with a "Dockerfile", which defines
> the Docker image, and use that to create a Singularity container.
In
> fact, one of my co-workers is working on that now.
>
> Here's where you'd want to start. This is the github repo
containing
> the Dockerfile for met-8.0:
> Caution-Caution-https://github.com/NCAR/container-dtc-
met/tree/v8.0
>
> Hope that helps.
>
> Thanks,
> John
>
> On Tue, Oct 16, 2018 at 9:50 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Oct 16 09:49:43 2018: Request 87402 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET in docker-container on DoD HPC (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > 402 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Recent development from my HPC resource indicates the possibility
of
> > using Singularity to enable the use of containerized applications,
> > but the question arises about whether the container needs to be
> > modified to run in a non-root environment. If not, then the
guidance
> > is that the user needs only to download and copy the container to
the HPC.
> > Singularity provides the way to deploy and run containers on the
HPC.
> > Could you clarify the situation regards the ability of MET V8.0 to
> > run in a non-root environment and, if not, what alternatives there
are?
> >
> > Thanks.
> >
> > R/
> > John
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET in docker-container on DoD HPC (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Oct 16 15:48:26 2018
John,
No changes would be required to the MET software itself to run it via
a
Singularity. While Singularity containers can be *run* without root
permission, I thought they still had to be built using root. But I'm
not
sure.
I'd suggest working with your sys admins to get Singularity
installed/accessible on your HPC.
Once you have Singularity available to you on a Linux node, you may be
able
to...
- create a singularity container from an existing Docker image
- use spython to convert MET's Dockerfile to a singularity recipe
file...
and use that recipe to make a singularity container.
If you're new to software containers, I do think starting with Docker
is a
good way to go. It's the industry standard and singularity is
basically
intended to make it possible to run Docker containers on HPC.
Thanks,
John
On Tue, Oct 16, 2018 at 3:13 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I've asked the question of our windows 10 system admin regarding the
> requirement for users to have root access in order to run the
containerized
> MET and I'll find out what their ruling is. Is Singularity a
possibility
> for running on windows to resolve the issue?
>
> Will MET require modification in order to run with Singularity in a
> non-root environment on the HPC?
>
> R/
> John
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, October 16, 2018 2:16 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> docker-container on DoD HPC (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I have only run Docker on a Mac laptop myself. Tara Jensen has run
it on
> Windows. We've found that running on Windows does work but we did
run into
> a few more headaches.
>
> Docker should be a free download. If you're running Windows 10
> professional or better than that'll work fine. If not, you'd use
the older
> Docker toolbox. You should not need to purchase any software.
>
> The catch is that you'll need root access on your laptop to
install/run
> Docker. That may be fine for testing but is problematic on HPC, as
you
> know. But I do think that working through that online tutorial is a
great
> way to get familiar with containers.
>
> Thanks,
> John
>
> On Tue, Oct 16, 2018 at 2:06 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John - For windows 10, the Docker sw I find using the link
provided at:
> >
> > Caution-
https://dtcenter.org/met/users/downloads/docker_container/METv
> > 8.0_docker_container.php
> >
> > is called Docker Desktop and is described at:
> > Caution-https://www.docker.com/products/windows-containers
> >
> > Is this the sw I need to purchase?
> >
> > Thanks.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Tuesday, October 16, 2018 1:28 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> > docker-container on DoD HPC (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > That's interesting that you may be able to run Singularity on your
HPC
> > machine.
> >
> > We taught a short-course at AMS last year and will be doing so
again
> > this year in Phoenix. It's on the Saturday before AMS begins.
We'll
> > be working through the steps of running an end-to-end NWP workflow
> > using Docker containers. The online instructions from last year
can be
> found here:
> > Caution-
> > Caution-https://dtcenter.org/met/docker-
nwp/tutorial/container_nwp_tut
> > orial
> >
> > Singularity is different from Docker. However, it's supposed to
be
> > pretty straight-forward to start with a "Dockerfile", which
defines
> > the Docker image, and use that to create a Singularity container.
In
> > fact, one of my co-workers is working on that now.
> >
> > Here's where you'd want to start. This is the github repo
containing
> > the Dockerfile for met-8.0:
> > Caution-Caution-https://github.com/NCAR/container-dtc-
met/tree/v8.0
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> > On Tue, Oct 16, 2018 at 9:50 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Oct 16 09:49:43 2018: Request 87402 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: MET in docker-container on DoD HPC (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > > 402 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > Recent development from my HPC resource indicates the
possibility of
> > > using Singularity to enable the use of containerized
applications,
> > > but the question arises about whether the container needs to be
> > > modified to run in a non-root environment. If not, then the
guidance
> > > is that the user needs only to download and copy the container
to the
> HPC.
> > > Singularity provides the way to deploy and run containers on the
HPC.
> > > Could you clarify the situation regards the ability of MET V8.0
to
> > > run in a non-root environment and, if not, what alternatives
there are?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in docker-container on DoD HPC (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Oct 16 16:37:11 2018
CLASSIFICATION: UNCLASSIFIED
John -
I think you are correct about requiring something beyond just user
privileges to build the image from an already built container using
Singularity on the HPC. I think they call it "Super User Privileges".
>From my discussions with the system admin person, if no mods are
required to run in a non-root environment, then it's just a matter of
downloading it and copying it to the HPC where the next step is to get
it into the Singularity workflow.
I also see a common thread with the steps required to build the image
and deploy it with Singularity as described by my HPC system admin and
your description of the processes required in your email below (i.e
Once you have Singularity available to you on a Linux node, you may be
able to...)
Based on the above and your info, I concur with pursuing the
installation on my HPC vice windows 10.
I'll find out more as they deploy Singularity in the next month or so.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, October 16, 2018 3:48 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
docker-container on DoD HPC (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
No changes would be required to the MET software itself to run it via
a Singularity. While Singularity containers can be *run* without root
permission, I thought they still had to be built using root. But I'm
not sure.
I'd suggest working with your sys admins to get Singularity
installed/accessible on your HPC.
Once you have Singularity available to you on a Linux node, you may be
able to...
- create a singularity container from an existing Docker image
- use spython to convert MET's Dockerfile to a singularity recipe
file...
and use that recipe to make a singularity container.
If you're new to software containers, I do think starting with Docker
is a good way to go. It's the industry standard and singularity is
basically intended to make it possible to run Docker containers on
HPC.
Thanks,
John
On Tue, Oct 16, 2018 at 3:13 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I've asked the question of our windows 10 system admin regarding the
> requirement for users to have root access in order to run the
> containerized MET and I'll find out what their ruling is. Is
> Singularity a possibility for running on windows to resolve the
issue?
>
> Will MET require modification in order to run with Singularity in a
> non-root environment on the HPC?
>
> R/
> John
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Tuesday, October 16, 2018 2:16 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> docker-container on DoD HPC (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> I have only run Docker on a Mac laptop myself. Tara Jensen has run
it
> on Windows. We've found that running on Windows does work but we
did
> run into a few more headaches.
>
> Docker should be a free download. If you're running Windows 10
> professional or better than that'll work fine. If not, you'd use
the
> older Docker toolbox. You should not need to purchase any software.
>
> The catch is that you'll need root access on your laptop to
> install/run Docker. That may be fine for testing but is problematic
> on HPC, as you know. But I do think that working through that
online
> tutorial is a great way to get familiar with containers.
>
> Thanks,
> John
>
> On Tue, Oct 16, 2018 at 2:06 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > 402 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John - For windows 10, the Docker sw I find using the link
provided at:
> >
> > Caution-Caution-
https://dtcenter.org/met/users/downloads/docker_cont
> > ainer/METv
> > 8.0_docker_container.php
> >
> > is called Docker Desktop and is described at:
> > Caution-Caution-https://www.docker.com/products/windows-containers
> >
> > Is this the sw I need to purchase?
> >
> > Thanks.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Tuesday, October 16, 2018 1:28 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> > docker-container on DoD HPC (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > That's interesting that you may be able to run Singularity on your
> > HPC machine.
> >
> > We taught a short-course at AMS last year and will be doing so
again
> > this year in Phoenix. It's on the Saturday before AMS begins.
> > We'll be working through the steps of running an end-to-end NWP
> > workflow using Docker containers. The online instructions from
last
> > year can be
> found here:
> > Caution-
> > Caution-Caution-https://dtcenter.org/met/docker-
nwp/tutorial/contain
> > er_nwp_tut
> > orial
> >
> > Singularity is different from Docker. However, it's supposed to
be
> > pretty straight-forward to start with a "Dockerfile", which
defines
> > the Docker image, and use that to create a Singularity container.
> > In fact, one of my co-workers is working on that now.
> >
> > Here's where you'd want to start. This is the github repo
> > containing the Dockerfile for met-8.0:
> >
> > Caution-Caution-Caution-https://github.com/NCAR/container-dtc-
met/tr
> > ee/v8.0
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> > On Tue, Oct 16, 2018 at 9:50 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Oct 16 09:49:43 2018: Request 87402 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: MET in docker-container on DoD HPC (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=87
> > > 402 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > Recent development from my HPC resource indicates the
possibility
> > > of using Singularity to enable the use of containerized
> > > applications, but the question arises about whether the
container
> > > needs to be modified to run in a non-root environment. If not,
> > > then the guidance is that the user needs only to download and
copy
> > > the container to the
> HPC.
> > > Singularity provides the way to deploy and run containers on the
HPC.
> > > Could you clarify the situation regards the ability of MET V8.0
to
> > > run in a non-root environment and, if not, what alternatives
there are?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET in docker-container on DoD HPC (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Oct 16 17:16:13 2018
John,
One more detail... we do plan on posting the pre-built docker
containers up
to docker hub. Currently they just live as tarballs on the MET
website,
but making them available through docker hub would make things much
easier
for you.
You may literally be able to run one singularity command to pull the
met-8.0 image from docker hub and run it.
Once that’s working, the next step is figuring out how to pass in
arguments
to get the container to actually do something useful... like run
Point-Stat
on your data.
I believe that running an existing Docker image through singularity is
easy. But converting a dockerfile to a singularity recipe to *build*
a
singularity image is a bit more work.
Thanks
John
On Tue, Oct 16, 2018 at 4:37 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I think you are correct about requiring something beyond just user
> privileges to build the image from an already built container using
> Singularity on the HPC. I think they call it "Super User
Privileges". From
> my discussions with the system admin person, if no mods are required
to run
> in a non-root environment, then it's just a matter of downloading it
and
> copying it to the HPC where the next step is to get it into the
Singularity
> workflow.
>
> I also see a common thread with the steps required to build the
image and
> deploy it with Singularity as described by my HPC system admin and
your
> description of the processes required in your email below (i.e Once
you
> have Singularity available to you on a Linux node, you may be able
to...)
>
> Based on the above and your info, I concur with pursuing the
installation
> on my HPC vice windows 10.
>
> I'll find out more as they deploy Singularity in the next month or
so.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, October 16, 2018 3:48 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> docker-container on DoD HPC (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> No changes would be required to the MET software itself to run it
via a
> Singularity. While Singularity containers can be *run* without root
> permission, I thought they still had to be built using root. But
I'm not
> sure.
>
> I'd suggest working with your sys admins to get Singularity
> installed/accessible on your HPC.
>
> Once you have Singularity available to you on a Linux node, you may
be
> able to...
> - create a singularity container from an existing Docker image
> - use spython to convert MET's Dockerfile to a singularity recipe
file...
> and use that recipe to make a singularity container.
>
> If you're new to software containers, I do think starting with
Docker is a
> good way to go. It's the industry standard and singularity is
basically
> intended to make it possible to run Docker containers on HPC.
>
> Thanks,
> John
>
> On Tue, Oct 16, 2018 at 3:13 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > I've asked the question of our windows 10 system admin regarding
the
> > requirement for users to have root access in order to run the
> > containerized MET and I'll find out what their ruling is. Is
> > Singularity a possibility for running on windows to resolve the
issue?
> >
> > Will MET require modification in order to run with Singularity in
a
> > non-root environment on the HPC?
> >
> > R/
> > John
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Tuesday, October 16, 2018 2:16 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> > docker-container on DoD HPC (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I have only run Docker on a Mac laptop myself. Tara Jensen has
run it
> > on Windows. We've found that running on Windows does work but we
did
> > run into a few more headaches.
> >
> > Docker should be a free download. If you're running Windows 10
> > professional or better than that'll work fine. If not, you'd use
the
> > older Docker toolbox. You should not need to purchase any
software.
> >
> > The catch is that you'll need root access on your laptop to
> > install/run Docker. That may be fine for testing but is
problematic
> > on HPC, as you know. But I do think that working through that
online
> > tutorial is a great way to get familiar with containers.
> >
> > Thanks,
> > John
> >
> > On Tue, Oct 16, 2018 at 2:06 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > > 402 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John - For windows 10, the Docker sw I find using the link
provided at:
> > >
> > > Caution-Caution-
https://dtcenter.org/met/users/downloads/docker_cont
> > > ainer/METv
> > > 8.0_docker_container.php
> > >
> > > is called Docker Desktop and is described at:
> > > Caution-Caution-https://www.docker.com/products/windows-
containers
> > >
> > > Is this the sw I need to purchase?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Tuesday, October 16, 2018 1:28 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> > > docker-container on DoD HPC (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > That's interesting that you may be able to run Singularity on
your
> > > HPC machine.
> > >
> > > We taught a short-course at AMS last year and will be doing so
again
> > > this year in Phoenix. It's on the Saturday before AMS begins.
> > > We'll be working through the steps of running an end-to-end NWP
> > > workflow using Docker containers. The online instructions from
last
> > > year can be
> > found here:
> > > Caution-
> > > Caution-Caution-https://dtcenter.org/met/docker-
nwp/tutorial/contain
> > > er_nwp_tut
> > > orial
> > >
> > > Singularity is different from Docker. However, it's supposed to
be
> > > pretty straight-forward to start with a "Dockerfile", which
defines
> > > the Docker image, and use that to create a Singularity
container.
> > > In fact, one of my co-workers is working on that now.
> > >
> > > Here's where you'd want to start. This is the github repo
> > > containing the Dockerfile for met-8.0:
> > >
> > > Caution-Caution-Caution-https://github.com/NCAR/container-dtc-
met/tr
> > > ee/v8.0
> > >
> > > Hope that helps.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Oct 16, 2018 at 9:50 AM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Oct 16 09:49:43 2018: Request 87402 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: MET in docker-container on DoD HPC
(UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=87
> > > > 402 >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > Recent development from my HPC resource indicates the
possibility
> > > > of using Singularity to enable the use of containerized
> > > > applications, but the question arises about whether the
container
> > > > needs to be modified to run in a non-root environment. If not,
> > > > then the guidance is that the user needs only to download and
copy
> > > > the container to the
> > HPC.
> > > > Singularity provides the way to deploy and run containers on
the HPC.
> > > > Could you clarify the situation regards the ability of MET
V8.0 to
> > > > run in a non-root environment and, if not, what alternatives
there
> are?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in docker-container on DoD HPC (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Oct 17 06:59:33 2018
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for the update on plans for pre-built docker containers. I
thought I saw a link in Tara's email announcing the release which
provided access to the docker hub:
https://github.com/NCAR/container-dtc-met
There will be a learning curve for sure. The HPC folks are gearing up
for their Singularity release in the coming couple of months and I
will try to keep up on their progress.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, October 16, 2018 5:16 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
docker-container on DoD HPC (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
One more detail... we do plan on posting the pre-built docker
containers up to docker hub. Currently they just live as tarballs on
the MET website, but making them available through docker hub would
make things much easier for you.
You may literally be able to run one singularity command to pull the
met-8.0 image from docker hub and run it.
Once that’s working, the next step is figuring out how to pass in
arguments to get the container to actually do something useful... like
run Point-Stat on your data.
I believe that running an existing Docker image through singularity is
easy. But converting a dockerfile to a singularity recipe to *build*
a singularity image is a bit more work.
Thanks
John
On Tue, Oct 16, 2018 at 4:37 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I think you are correct about requiring something beyond just user
> privileges to build the image from an already built container using
> Singularity on the HPC. I think they call it "Super User
Privileges".
> From my discussions with the system admin person, if no mods are
> required to run in a non-root environment, then it's just a matter
of
> downloading it and copying it to the HPC where the next step is to
get
> it into the Singularity workflow.
>
> I also see a common thread with the steps required to build the
image
> and deploy it with Singularity as described by my HPC system admin
and
> your description of the processes required in your email below (i.e
> Once you have Singularity available to you on a Linux node, you may
be
> able to...)
>
> Based on the above and your info, I concur with pursuing the
> installation on my HPC vice windows 10.
>
> I'll find out more as they deploy Singularity in the next month or
so.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Tuesday, October 16, 2018 3:48 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> docker-container on DoD HPC (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> No changes would be required to the MET software itself to run it
via
> a Singularity. While Singularity containers can be *run* without
root
> permission, I thought they still had to be built using root. But
I'm
> not sure.
>
> I'd suggest working with your sys admins to get Singularity
> installed/accessible on your HPC.
>
> Once you have Singularity available to you on a Linux node, you may
be
> able to...
> - create a singularity container from an existing Docker image
> - use spython to convert MET's Dockerfile to a singularity recipe
file...
> and use that recipe to make a singularity container.
>
> If you're new to software containers, I do think starting with
Docker
> is a good way to go. It's the industry standard and singularity is
> basically intended to make it possible to run Docker containers on
HPC.
>
> Thanks,
> John
>
> On Tue, Oct 16, 2018 at 3:13 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > 402 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > I've asked the question of our windows 10 system admin regarding
the
> > requirement for users to have root access in order to run the
> > containerized MET and I'll find out what their ruling is. Is
> > Singularity a possibility for running on windows to resolve the
issue?
> >
> > Will MET require modification in order to run with Singularity in
a
> > non-root environment on the HPC?
> >
> > R/
> > John
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Tuesday, October 16, 2018 2:16 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> > docker-container on DoD HPC (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I have only run Docker on a Mac laptop myself. Tara Jensen has
run
> > it on Windows. We've found that running on Windows does work but
we
> > did run into a few more headaches.
> >
> > Docker should be a free download. If you're running Windows 10
> > professional or better than that'll work fine. If not, you'd use
> > the older Docker toolbox. You should not need to purchase any
software.
> >
> > The catch is that you'll need root access on your laptop to
> > install/run Docker. That may be fine for testing but is
problematic
> > on HPC, as you know. But I do think that working through that
> > online tutorial is a great way to get familiar with containers.
> >
> > Thanks,
> > John
> >
> > On Tue, Oct 16, 2018 at 2:06 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=87
> > > 402 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John - For windows 10, the Docker sw I find using the link
provided at:
> > >
> > > Caution-Caution-Caution-
https://dtcenter.org/met/users/downloads/d
> > > ocker_cont
> > > ainer/METv
> > > 8.0_docker_container.php
> > >
> > > is called Docker Desktop and is described at:
> > > Caution-Caution-Caution-https://www.docker.com/products/windows-
co
> > > ntainers
> > >
> > > Is this the sw I need to purchase?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Tuesday, October 16, 2018 1:28 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> > > docker-container on DoD HPC (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
> > > pasting the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > That's interesting that you may be able to run Singularity on
your
> > > HPC machine.
> > >
> > > We taught a short-course at AMS last year and will be doing so
> > > again this year in Phoenix. It's on the Saturday before AMS
begins.
> > > We'll be working through the steps of running an end-to-end NWP
> > > workflow using Docker containers. The online instructions from
> > > last year can be
> > found here:
> > > Caution-
> > > Caution-Caution-Caution-https://dtcenter.org/met/docker-
nwp/tutori
> > > al/contain
> > > er_nwp_tut
> > > orial
> > >
> > > Singularity is different from Docker. However, it's supposed to
> > > be pretty straight-forward to start with a "Dockerfile", which
> > > defines the Docker image, and use that to create a Singularity
container.
> > > In fact, one of my co-workers is working on that now.
> > >
> > > Here's where you'd want to start. This is the github repo
> > > containing the Dockerfile for met-8.0:
> > >
> > > Caution-Caution-Caution-Caution-
https://github.com/NCAR/container-
> > > dtc-met/tr
> > > ee/v8.0
> > >
> > > Hope that helps.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Oct 16, 2018 at 9:50 AM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Oct 16 09:49:43 2018: Request 87402 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: MET in docker-container on DoD HPC
(UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=87
> > > > 402 >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > Recent development from my HPC resource indicates the
> > > > possibility of using Singularity to enable the use of
> > > > containerized applications, but the question arises about
> > > > whether the container needs to be modified to run in a non-
root
> > > > environment. If not, then the guidance is that the user needs
> > > > only to download and copy the container to the
> > HPC.
> > > > Singularity provides the way to deploy and run containers on
the HPC.
> > > > Could you clarify the situation regards the ability of MET
V8.0
> > > > to run in a non-root environment and, if not, what
alternatives
> > > > there
> are?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET in docker-container on DoD HPC (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed Oct 17 08:08:33 2018
John,
Please let us know how it goes. While we're able to run Docker
containers
on laptops at NCAR, they haven't yet made singularity available on our
HPC,
cheyenne.
I'll go ahead and resolve this ticket.
Thanks,
John
On Wed, Oct 17, 2018 at 7:00 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for the update on plans for pre-built docker containers. I
thought
> I saw a link in Tara's email announcing the release which provided
access
> to the docker hub:
> https://github.com/NCAR/container-dtc-met
>
> There will be a learning curve for sure. The HPC folks are gearing
up for
> their Singularity release in the coming couple of months and I will
try to
> keep up on their progress.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, October 16, 2018 5:16 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> docker-container on DoD HPC (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> One more detail... we do plan on posting the pre-built docker
containers
> up to docker hub. Currently they just live as tarballs on the MET
website,
> but making them available through docker hub would make things much
easier
> for you.
>
> You may literally be able to run one singularity command to pull the
> met-8.0 image from docker hub and run it.
>
> Once that’s working, the next step is figuring out how to pass in
> arguments to get the container to actually do something useful...
like run
> Point-Stat on your data.
>
> I believe that running an existing Docker image through singularity
is
> easy. But converting a dockerfile to a singularity recipe to
*build* a
> singularity image is a bit more work.
>
> Thanks
> John
>
> On Tue, Oct 16, 2018 at 4:37 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87402 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > I think you are correct about requiring something beyond just user
> > privileges to build the image from an already built container
using
> > Singularity on the HPC. I think they call it "Super User
Privileges".
> > From my discussions with the system admin person, if no mods are
> > required to run in a non-root environment, then it's just a matter
of
> > downloading it and copying it to the HPC where the next step is to
get
> > it into the Singularity workflow.
> >
> > I also see a common thread with the steps required to build the
image
> > and deploy it with Singularity as described by my HPC system admin
and
> > your description of the processes required in your email below
(i.e
> > Once you have Singularity available to you on a Linux node, you
may be
> > able to...)
> >
> > Based on the above and your info, I concur with pursuing the
> > installation on my HPC vice windows 10.
> >
> > I'll find out more as they deploy Singularity in the next month or
so.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Tuesday, October 16, 2018 3:48 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> > docker-container on DoD HPC (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > No changes would be required to the MET software itself to run it
via
> > a Singularity. While Singularity containers can be *run* without
root
> > permission, I thought they still had to be built using root. But
I'm
> > not sure.
> >
> > I'd suggest working with your sys admins to get Singularity
> > installed/accessible on your HPC.
> >
> > Once you have Singularity available to you on a Linux node, you
may be
> > able to...
> > - create a singularity container from an existing Docker image
> > - use spython to convert MET's Dockerfile to a singularity recipe
file...
> > and use that recipe to make a singularity container.
> >
> > If you're new to software containers, I do think starting with
Docker
> > is a good way to go. It's the industry standard and singularity
is
> > basically intended to make it possible to run Docker containers on
HPC.
> >
> > Thanks,
> > John
> >
> > On Tue, Oct 16, 2018 at 3:13 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > > 402 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > I've asked the question of our windows 10 system admin regarding
the
> > > requirement for users to have root access in order to run the
> > > containerized MET and I'll find out what their ruling is. Is
> > > Singularity a possibility for running on windows to resolve the
issue?
> > >
> > > Will MET require modification in order to run with Singularity
in a
> > > non-root environment on the HPC?
> > >
> > > R/
> > > John
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Tuesday, October 16, 2018 2:16 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET
in
> > > docker-container on DoD HPC (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > I have only run Docker on a Mac laptop myself. Tara Jensen has
run
> > > it on Windows. We've found that running on Windows does work
but we
> > > did run into a few more headaches.
> > >
> > > Docker should be a free download. If you're running Windows 10
> > > professional or better than that'll work fine. If not, you'd
use
> > > the older Docker toolbox. You should not need to purchase any
> software.
> > >
> > > The catch is that you'll need root access on your laptop to
> > > install/run Docker. That may be fine for testing but is
problematic
> > > on HPC, as you know. But I do think that working through that
> > > online tutorial is a great way to get familiar with containers.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Oct 16, 2018 at 2:06 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=87
> > > > 402 >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > John - For windows 10, the Docker sw I find using the link
provided
> at:
> > > >
> > > > Caution-Caution-Caution-
https://dtcenter.org/met/users/downloads/d
> > > > ocker_cont
> > > > ainer/METv
> > > > 8.0_docker_container.php
> > > >
> > > > is called Docker Desktop and is described at:
> > > > Caution-Caution-Caution-
https://www.docker.com/products/windows-co
> > > > ntainers
> > > >
> > > > Is this the sw I need to purchase?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT
> > > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > > Sent: Tuesday, October 16, 2018 1:28 PM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > <john.w.raby2.civ at mail.mil>
> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #87402] MET in
> > > > docker-container on DoD HPC (UNCLASSIFIED)
> > > >
> > > > All active links contained in this email were disabled.
Please
> > > > verify the identity of the sender, and confirm the
authenticity of
> > > > all links contained within the message prior to copying and
> > > > pasting the address to a Web browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > John,
> > > >
> > > > That's interesting that you may be able to run Singularity on
your
> > > > HPC machine.
> > > >
> > > > We taught a short-course at AMS last year and will be doing so
> > > > again this year in Phoenix. It's on the Saturday before AMS
begins.
> > > > We'll be working through the steps of running an end-to-end
NWP
> > > > workflow using Docker containers. The online instructions
from
> > > > last year can be
> > > found here:
> > > > Caution-
> > > > Caution-Caution-Caution-https://dtcenter.org/met/docker-
nwp/tutori
> > > > al/contain
> > > > er_nwp_tut
> > > > orial
> > > >
> > > > Singularity is different from Docker. However, it's supposed
to
> > > > be pretty straight-forward to start with a "Dockerfile", which
> > > > defines the Docker image, and use that to create a Singularity
> container.
> > > > In fact, one of my co-workers is working on that now.
> > > >
> > > > Here's where you'd want to start. This is the github repo
> > > > containing the Dockerfile for met-8.0:
> > > >
> > > > Caution-Caution-Caution-Caution-
https://github.com/NCAR/container-
> > > > dtc-met/tr
> > > > ee/v8.0
> > > >
> > > > Hope that helps.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, Oct 16, 2018 at 9:50 AM Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Tue Oct 16 09:49:43 2018: Request 87402 was acted upon.
> > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > > Queue: met_help
> > > > > Subject: MET in docker-container on DoD HPC
(UNCLASSIFIED)
> > > > > Owner: Nobody
> > > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > Status: new
> > > > > Ticket <Caution-Caution-Caution-Caution-url:
> > > > > Caution-Caution-Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > > html?id=87
> > > > > 402 >
> > > > >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > Recent development from my HPC resource indicates the
> > > > > possibility of using Singularity to enable the use of
> > > > > containerized applications, but the question arises about
> > > > > whether the container needs to be modified to run in a non-
root
> > > > > environment. If not, then the guidance is that the user
needs
> > > > > only to download and copy the container to the
> > > HPC.
> > > > > Singularity provides the way to deploy and run containers on
the
> HPC.
> > > > > Could you clarify the situation regards the ability of MET
V8.0
> > > > > to run in a non-root environment and, if not, what
alternatives
> > > > > there
> > are?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > R/
> > > > > John
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > >
> > > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
More information about the Met_help
mailing list