[Met_help] [rt.rap.ucar.edu #87728] History for MET docker-container in Singularity (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Tue Apr 30 16:08:34 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
CLASSIFICATION: UNCLASSIFIED
I received the email below from the developer who is implementing Singularity on the HPC in response to my question about running an already-built docker-container using Singularity Runtime. They are asking users to start test runs with their already-built Singularity containers. From his guidance below, can the MET docker-container be made to run in Singularity Runtime?
Thanks.
R/
John
Mr. John W. Raby
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
-----Original Message-----
From: Dwyer, Matthew R CTR USARMY RDECOM (US)
Sent: Friday, November 9, 2018 8:35 AM
To: Raby, John W CIV USARMY RDECOM ARL (US) <john.w.raby2.civ at mail.mil>
Subject: RE: Singularity Runtime installed on Centennial (UNCLASSIFIED)
You would need to convert the docker image to a singularity image. From there you might have to change some permissions or re-configure to use non-root directories. It depends on how tightly coupled the application is to root inside the image. You will need root on a machine to convert the image to Singularity.
Thanks,
Matt
________________________________________
From: Raby, John W CIV USARMY RDECOM ARL (US)
Sent: Thursday, November 08, 2018 5:24 PM
To: Dwyer, Matthew R CTR USARMY RDECOM (US)
Subject: RE: Singularity Runtime installed on Centennial (UNCLASSIFIED)
CLASSIFICATION: UNCLASSIFIED
Thanks for all your work on implementing Singularity and for the notification, Matt. I most certainly will send this out to others I know who are interested. I know of an already-built docker-container which I'm interested in. Does this fall into the category of being suitable for testing on Centennial?
R/
John
-----Original Message-----
From: Dwyer, Matthew R CTR USARMY RDECOM (US)
Sent: Thursday, November 8, 2018 3:18 PM
To: Dwyer, Matthew R CTR USARMY RDECOM (US) <matthew.r.dwyer7.ctr at mail.mil>
Subject: Singularity Runtime installed on Centennial
Good News!
Containers are installed on Centennial (for early access). This is the first production machine container installation at the ARL DSRC.
Singularity runtime is installed on Centennial which means you still cannot build your containers on ARL DSRC resources. If you have already-built singularity containers and would like to run them please go ahead and test them out now. The more bugs/misconfigurations we find now will provide a smoother user-experience with container workloads. Let me know of successes and failures and please send to others who are interested.
Thank You,
Matt Dwyer
4102786819
CLASSIFICATION: UNCLASSIFIED
CLASSIFICATION: UNCLASSIFIED
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: MET docker-container in Singularity (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed Nov 14 16:39:45 2018
John,
I forwarded your question about Singularity to other folks in the DTC
and
am forwarding their response.
Specifically, it looks like once we get the MET container posted to
DockerHub than importing it into Singularity becomes a one-liner.
Seems
like a good motivation to get that done!
Thanks,
John
---------- Forwarded message ---------
From: Michael Kavulich <kavulich at ucar.edu>
Date: Wed, Nov 14, 2018 at 12:16 PM
Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container in
Singularity
(UNCLASSIFIED)
To: John Halley Gotway <johnhg at ucar.edu>
Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
<harrold at ucar.edu>,
Tara Jensen <jensen at ucar.edu>
John,
I have only tested singularity with standard images from dockerhub.
This
process is relatively easy:
https://www.sylabs.io/guides/2.5/user-
guide/singularity_and_docker.html?highlight=docker
Converting images locally is a bit more involved, and I have not tried
this
myself:
https://groups.google.com/a/lbl.gov/forum/#!topic/singularity/ZXOaNaM_6MY
This all requires that you are on a machine with both Docker and
Singularity installed, which is a big limitation.
The easiest thing for users would be if we did this conversion
ourselves
and provided both Docker and Singularity images. Provided I have hours
left
after the prep for the tutorial is completed this is something I can
look
into.
Michael Kavulich, Jr.
Associate Scientist, National Center for Atmospheric Research (NCAR)
Joint Numerical Testbed (JNT), Research Applications Laboratory (RAL)
*kavulich at ucar.edu <kavulich at ucar.edu>*
On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Mike,
>
> John Raby is a MET user from the Army Research Lab who is interested
in
> running MET through a container. We currently have it in a Docker
> container, but apparently Singularity is now available on their
> super-computer, Centennial, for testing.
>
> He's written MET-Help asking how to convert the MET Docker container
to
> Singularity for testing on Centennial.
>
> I have never run Singularity but know that you have. What should I
do?
>
> Thanks,
> John
>
>
> ---------- Forwarded message ---------
> From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> Date: Fri, Nov 9, 2018 at 9:07 AM
> Subject: [rt.rap.ucar.edu #87728] MET docker-container in
Singularity
> (UNCLASSIFIED)
> To:
>
>
>
> Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET docker-container in Singularity (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I received the email below from the developer who is implementing
> Singularity on the HPC in response to my question about running an
> already-built docker-container using Singularity Runtime. They are
asking
> users to start test runs with their already-built Singularity
containers.
> From his guidance below, can the MET docker-container be made to run
in
> Singularity Runtime?
>
> Thanks.
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
> -----Original Message-----
> From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> Sent: Friday, November 9, 2018 8:35 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: RE: Singularity Runtime installed on Centennial
(UNCLASSIFIED)
>
>
> You would need to convert the docker image to a singularity image.
From
> there you might have to change some permissions or re-configure to
use
> non-root directories. It depends on how tightly coupled the
application is
> to root inside the image. You will need root on a machine to convert
the
> image to Singularity.
>
> Thanks,
> Matt
>
> ________________________________________
> From: Raby, John W CIV USARMY RDECOM ARL (US)
> Sent: Thursday, November 08, 2018 5:24 PM
> To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> Subject: RE: Singularity Runtime installed on Centennial
(UNCLASSIFIED)
>
> CLASSIFICATION: UNCLASSIFIED
>
> Thanks for all your work on implementing Singularity and for the
> notification, Matt. I most certainly will send this out to others I
know
> who are interested. I know of an already-built docker-container
which I'm
> interested in. Does this fall into the category of being suitable
for
> testing on Centennial?
>
> R/
> John
>
>
>
> -----Original Message-----
> From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> Sent: Thursday, November 8, 2018 3:18 PM
> To: Dwyer, Matthew R CTR USARMY RDECOM (US)
<matthew.r.dwyer7.ctr at mail.mil
> >
> Subject: Singularity Runtime installed on Centennial
>
> Good News!
>
>
> Containers are installed on Centennial (for early access). This is
the
> first production machine container installation at the ARL DSRC.
>
> Singularity runtime is installed on Centennial which means you still
> cannot build your containers on ARL DSRC resources. If you have
> already-built singularity containers and would like to run them
please go
> ahead and test them out now. The more bugs/misconfigurations we find
now
> will provide a smoother user-experience with container workloads.
Let me
> know of successes and failures and please send to others who are
interested.
>
>
> Thank You,
> Matt Dwyer
> 4102786819
>
> CLASSIFICATION: UNCLASSIFIED
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET docker-container in Singularity (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Nov 15 08:37:10 2018
CLASSIFICATION: UNCLASSIFIED
John -
Sorry for the delayed response. I'm just reading your email and the
developer response now and am glad to hear the positive assessment of
possible development work which might enable the production of a MET
Singularity container . Keep me posted on further developments of this
enhancement of the MET distribution. If I can assist in any way please
let me know.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, November 14, 2018 4:40 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET docker-
container in Singularity (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
I forwarded your question about Singularity to other folks in the DTC
and am forwarding their response.
Specifically, it looks like once we get the MET container posted to
DockerHub than importing it into Singularity becomes a one-liner.
Seems like a good motivation to get that done!
Thanks,
John
---------- Forwarded message ---------
From: Michael Kavulich <kavulich at ucar.edu>
Date: Wed, Nov 14, 2018 at 12:16 PM
Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container in
Singularity
(UNCLASSIFIED)
To: John Halley Gotway <johnhg at ucar.edu>
Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
<harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
John,
I have only tested singularity with standard images from dockerhub.
This process is relatively easy:
Caution-https://www.sylabs.io/guides/2.5/user-
guide/singularity_and_docker.html?highlight=docker
Converting images locally is a bit more involved, and I have not tried
this
myself:
Caution-
https://groups.google.com/a/lbl.gov/forum/#!topic/singularity/ZXOaNaM_6MY
This all requires that you are on a machine with both Docker and
Singularity installed, which is a big limitation.
The easiest thing for users would be if we did this conversion
ourselves and provided both Docker and Singularity images. Provided I
have hours left after the prep for the tutorial is completed this is
something I can look into.
Michael Kavulich, Jr.
Associate Scientist, National Center for Atmospheric Research (NCAR)
Joint Numerical Testbed (JNT), Research Applications Laboratory (RAL)
*kavulich at ucar.edu <kavulich at ucar.edu>*
On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Mike,
>
> John Raby is a MET user from the Army Research Lab who is interested
> in running MET through a container. We currently have it in a
Docker
> container, but apparently Singularity is now available on their
> super-computer, Centennial, for testing.
>
> He's written MET-Help asking how to convert the MET Docker container
> to Singularity for testing on Centennial.
>
> I have never run Singularity but know that you have. What should I
do?
>
> Thanks,
> John
>
>
> ---------- Forwarded message ---------
> From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> Date: Fri, Nov 9, 2018 at 9:07 AM
> Subject: [rt.rap.ucar.edu #87728] MET docker-container in
Singularity
> (UNCLASSIFIED)
> To:
>
>
>
> Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET docker-container in Singularity (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I received the email below from the developer who is implementing
> Singularity on the HPC in response to my question about running an
> already-built docker-container using Singularity Runtime. They are
> asking users to start test runs with their already-built Singularity
containers.
> From his guidance below, can the MET docker-container be made to run
> in Singularity Runtime?
>
> Thanks.
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
> -----Original Message-----
> From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> Sent: Friday, November 9, 2018 8:35 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: RE: Singularity Runtime installed on Centennial
> (UNCLASSIFIED)
>
>
> You would need to convert the docker image to a singularity image.
> From there you might have to change some permissions or re-configure
> to use non-root directories. It depends on how tightly coupled the
> application is to root inside the image. You will need root on a
> machine to convert the image to Singularity.
>
> Thanks,
> Matt
>
> ________________________________________
> From: Raby, John W CIV USARMY RDECOM ARL (US)
> Sent: Thursday, November 08, 2018 5:24 PM
> To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> Subject: RE: Singularity Runtime installed on Centennial
> (UNCLASSIFIED)
>
> CLASSIFICATION: UNCLASSIFIED
>
> Thanks for all your work on implementing Singularity and for the
> notification, Matt. I most certainly will send this out to others I
> know who are interested. I know of an already-built docker-container
> which I'm interested in. Does this fall into the category of being
> suitable for testing on Centennial?
>
> R/
> John
>
>
>
> -----Original Message-----
> From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> Sent: Thursday, November 8, 2018 3:18 PM
> To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> <matthew.r.dwyer7.ctr at mail.mil
> >
> Subject: Singularity Runtime installed on Centennial
>
> Good News!
>
>
> Containers are installed on Centennial (for early access). This is
the
> first production machine container installation at the ARL DSRC.
>
> Singularity runtime is installed on Centennial which means you still
> cannot build your containers on ARL DSRC resources. If you have
> already-built singularity containers and would like to run them
please
> go ahead and test them out now. The more bugs/misconfigurations we
> find now will provide a smoother user-experience with container
> workloads. Let me know of successes and failures and please send to
others who are interested.
>
>
> Thank You,
> Matt Dwyer
> 4102786819
>
> CLASSIFICATION: UNCLASSIFIED
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET docker-container in Singularity (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Mon Jan 07 15:01:08 2019
CLASSIFICATION: UNCLASSIFIED
John -
Ref the email update below on the possibility of a Singularity
container for MET. Has there been any progress on this?
Thanks.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, November 14, 2018 4:40 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET docker-
container in Singularity (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
I forwarded your question about Singularity to other folks in the DTC
and am forwarding their response.
Specifically, it looks like once we get the MET container posted to
DockerHub than importing it into Singularity becomes a one-liner.
Seems like a good motivation to get that done!
Thanks,
John
---------- Forwarded message ---------
From: Michael Kavulich <kavulich at ucar.edu>
Date: Wed, Nov 14, 2018 at 12:16 PM
Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container in
Singularity
(UNCLASSIFIED)
To: John Halley Gotway <johnhg at ucar.edu>
Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
<harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
John,
I have only tested singularity with standard images from dockerhub.
This process is relatively easy:
Caution-https://www.sylabs.io/guides/2.5/user-
guide/singularity_and_docker.html?highlight=docker
Converting images locally is a bit more involved, and I have not tried
this
myself:
Caution-
https://groups.google.com/a/lbl.gov/forum/#!topic/singularity/ZXOaNaM_6MY
This all requires that you are on a machine with both Docker and
Singularity installed, which is a big limitation.
The easiest thing for users would be if we did this conversion
ourselves and provided both Docker and Singularity images. Provided I
have hours left after the prep for the tutorial is completed this is
something I can look into.
Michael Kavulich, Jr.
Associate Scientist, National Center for Atmospheric Research (NCAR)
Joint Numerical Testbed (JNT), Research Applications Laboratory (RAL)
*kavulich at ucar.edu <kavulich at ucar.edu>*
On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Mike,
>
> John Raby is a MET user from the Army Research Lab who is interested
> in running MET through a container. We currently have it in a
Docker
> container, but apparently Singularity is now available on their
> super-computer, Centennial, for testing.
>
> He's written MET-Help asking how to convert the MET Docker container
> to Singularity for testing on Centennial.
>
> I have never run Singularity but know that you have. What should I
do?
>
> Thanks,
> John
>
>
> ---------- Forwarded message ---------
> From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> Date: Fri, Nov 9, 2018 at 9:07 AM
> Subject: [rt.rap.ucar.edu #87728] MET docker-container in
Singularity
> (UNCLASSIFIED)
> To:
>
>
>
> Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET docker-container in Singularity (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
> I received the email below from the developer who is implementing
> Singularity on the HPC in response to my question about running an
> already-built docker-container using Singularity Runtime. They are
> asking users to start test runs with their already-built Singularity
containers.
> From his guidance below, can the MET docker-container be made to run
> in Singularity Runtime?
>
> Thanks.
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
> -----Original Message-----
> From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> Sent: Friday, November 9, 2018 8:35 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: RE: Singularity Runtime installed on Centennial
> (UNCLASSIFIED)
>
>
> You would need to convert the docker image to a singularity image.
> From there you might have to change some permissions or re-configure
> to use non-root directories. It depends on how tightly coupled the
> application is to root inside the image. You will need root on a
> machine to convert the image to Singularity.
>
> Thanks,
> Matt
>
> ________________________________________
> From: Raby, John W CIV USARMY RDECOM ARL (US)
> Sent: Thursday, November 08, 2018 5:24 PM
> To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> Subject: RE: Singularity Runtime installed on Centennial
> (UNCLASSIFIED)
>
> CLASSIFICATION: UNCLASSIFIED
>
> Thanks for all your work on implementing Singularity and for the
> notification, Matt. I most certainly will send this out to others I
> know who are interested. I know of an already-built docker-container
> which I'm interested in. Does this fall into the category of being
> suitable for testing on Centennial?
>
> R/
> John
>
>
>
> -----Original Message-----
> From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> Sent: Thursday, November 8, 2018 3:18 PM
> To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> <matthew.r.dwyer7.ctr at mail.mil
> >
> Subject: Singularity Runtime installed on Centennial
>
> Good News!
>
>
> Containers are installed on Centennial (for early access). This is
the
> first production machine container installation at the ARL DSRC.
>
> Singularity runtime is installed on Centennial which means you still
> cannot build your containers on ARL DSRC resources. If you have
> already-built singularity containers and would like to run them
please
> go ahead and test them out now. The more bugs/misconfigurations we
> find now will provide a smoother user-experience with container
> workloads. Let me know of successes and failures and please send to
others who are interested.
>
>
> Thank You,
> Matt Dwyer
> 4102786819
>
> CLASSIFICATION: UNCLASSIFIED
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET docker-container in Singularity (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Jan 10 08:03:47 2019
John,
I forwarded your question to Mike Kavulich, who's planning on working
with
Singularity. Listed below is my question and his response. We are
both at
AMS this week.
---------------------------------------
Mike,
I got a followup question from John Raby from the Army. He's
wondering if
we've made any progress in using Singularity containers.
If I remember correctly, once we make containers available via
DockerHub,
this is very easy. But until then, it's more difficult. Do you
happen to
have a Singularity version of the MET container he could use for
testing?
---------------------------------------
John,
I do not have a singularity container ready, I haven't re-installed
the
necessary software since my mac had to be serviced. I can get to it
next
week at the earliest.
Mike
---------------------------------------
On Mon, Jan 7, 2019 at 3:01 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Ref the email update below on the possibility of a Singularity
container
> for MET. Has there been any progress on this?
>
> Thanks.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, November 14, 2018 4:40 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> docker-container in Singularity (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I forwarded your question about Singularity to other folks in the
DTC and
> am forwarding their response.
>
> Specifically, it looks like once we get the MET container posted to
> DockerHub than importing it into Singularity becomes a one-liner.
Seems
> like a good motivation to get that done!
>
> Thanks,
> John
>
> ---------- Forwarded message ---------
> From: Michael Kavulich <kavulich at ucar.edu>
> Date: Wed, Nov 14, 2018 at 12:16 PM
> Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container in
Singularity
> (UNCLASSIFIED)
> To: John Halley Gotway <johnhg at ucar.edu>
> Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
<harrold at ucar.edu>,
> Tara Jensen <jensen at ucar.edu>
>
>
> John,
>
> I have only tested singularity with standard images from dockerhub.
This
> process is relatively easy:
> Caution-
> https://www.sylabs.io/guides/2.5/user-
guide/singularity_and_docker.html?highlight=docker
>
> Converting images locally is a bit more involved, and I have not
tried this
> myself:
> Caution-
>
https://groups.google.com/a/lbl.gov/forum/#!topic/singularity/ZXOaNaM_6MY
>
> This all requires that you are on a machine with both Docker and
> Singularity installed, which is a big limitation.
>
> The easiest thing for users would be if we did this conversion
ourselves
> and provided both Docker and Singularity images. Provided I have
hours left
> after the prep for the tutorial is completed this is something I can
look
> into.
>
> Michael Kavulich, Jr.
> Associate Scientist, National Center for Atmospheric Research (NCAR)
Joint
> Numerical Testbed (JNT), Research Applications Laboratory (RAL) *
> kavulich at ucar.edu <kavulich at ucar.edu>*
>
>
> On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Mike,
> >
> > John Raby is a MET user from the Army Research Lab who is
interested
> > in running MET through a container. We currently have it in a
Docker
> > container, but apparently Singularity is now available on their
> > super-computer, Centennial, for testing.
> >
> > He's written MET-Help asking how to convert the MET Docker
container
> > to Singularity for testing on Centennial.
> >
> > I have never run Singularity but know that you have. What should
I do?
> >
> > Thanks,
> > John
> >
> >
> > ---------- Forwarded message ---------
> > From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> > Date: Fri, Nov 9, 2018 at 9:07 AM
> > Subject: [rt.rap.ucar.edu #87728] MET docker-container in
Singularity
> > (UNCLASSIFIED)
> > To:
> >
> >
> >
> > Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET docker-container in Singularity (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I received the email below from the developer who is implementing
> > Singularity on the HPC in response to my question about running an
> > already-built docker-container using Singularity Runtime. They are
> > asking users to start test runs with their already-built
Singularity
> containers.
> > From his guidance below, can the MET docker-container be made to
run
> > in Singularity Runtime?
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > -----Original Message-----
> > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > Sent: Friday, November 9, 2018 8:35 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: RE: Singularity Runtime installed on Centennial
> > (UNCLASSIFIED)
> >
> >
> > You would need to convert the docker image to a singularity image.
> > From there you might have to change some permissions or re-
configure
> > to use non-root directories. It depends on how tightly coupled the
> > application is to root inside the image. You will need root on a
> > machine to convert the image to Singularity.
> >
> > Thanks,
> > Matt
> >
> > ________________________________________
> > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > Sent: Thursday, November 08, 2018 5:24 PM
> > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > Subject: RE: Singularity Runtime installed on Centennial
> > (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Thanks for all your work on implementing Singularity and for the
> > notification, Matt. I most certainly will send this out to others
I
> > know who are interested. I know of an already-built docker-
container
> > which I'm interested in. Does this fall into the category of being
> > suitable for testing on Centennial?
> >
> > R/
> > John
> >
> >
> >
> > -----Original Message-----
> > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > Sent: Thursday, November 8, 2018 3:18 PM
> > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > <matthew.r.dwyer7.ctr at mail.mil
> > >
> > Subject: Singularity Runtime installed on Centennial
> >
> > Good News!
> >
> >
> > Containers are installed on Centennial (for early access). This is
the
> > first production machine container installation at the ARL DSRC.
> >
> > Singularity runtime is installed on Centennial which means you
still
> > cannot build your containers on ARL DSRC resources. If you have
> > already-built singularity containers and would like to run them
please
> > go ahead and test them out now. The more bugs/misconfigurations we
> > find now will provide a smoother user-experience with container
> > workloads. Let me know of successes and failures and please send
to
> others who are interested.
> >
> >
> > Thank You,
> > Matt Dwyer
> > 4102786819
> >
> > CLASSIFICATION: UNCLASSIFIED
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET docker-container in Singularity (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Jan 10 08:13:24 2019
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for responding while at the mtg and for following up with Mike.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, January 10, 2019 8:04 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
docker-container in Singularity (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
I forwarded your question to Mike Kavulich, who's planning on working
with Singularity. Listed below is my question and his response. We
are both at AMS this week.
---------------------------------------
Mike,
I got a followup question from John Raby from the Army. He's
wondering if we've made any progress in using Singularity containers.
If I remember correctly, once we make containers available via
DockerHub, this is very easy. But until then, it's more difficult.
Do you happen to have a Singularity version of the MET container he
could use for testing?
---------------------------------------
John,
I do not have a singularity container ready, I haven't re-installed
the necessary software since my mac had to be serviced. I can get to
it next week at the earliest.
Mike
---------------------------------------
On Mon, Jan 7, 2019 at 3:01 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Ref the email update below on the possibility of a Singularity
> container for MET. Has there been any progress on this?
>
> Thanks.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Wednesday, November 14, 2018 4:40 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> docker-container in Singularity (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> I forwarded your question about Singularity to other folks in the
DTC
> and am forwarding their response.
>
> Specifically, it looks like once we get the MET container posted to
> DockerHub than importing it into Singularity becomes a one-liner.
> Seems like a good motivation to get that done!
>
> Thanks,
> John
>
> ---------- Forwarded message ---------
> From: Michael Kavulich <kavulich at ucar.edu>
> Date: Wed, Nov 14, 2018 at 12:16 PM
> Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container in
> Singularity
> (UNCLASSIFIED)
> To: John Halley Gotway <johnhg at ucar.edu>
> Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
> <harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
>
>
> John,
>
> I have only tested singularity with standard images from dockerhub.
> This process is relatively easy:
> Caution-
> Caution-https://www.sylabs.io/guides/2.5/user-
guide/singularity_and_do
> cker.html?highlight=docker
>
> Converting images locally is a bit more involved, and I have not
tried
> this
> myself:
> Caution-
> Caution-
https://groups.google.com/a/lbl.gov/forum/#!topic/singularity/
> ZXOaNaM_6MY
>
> This all requires that you are on a machine with both Docker and
> Singularity installed, which is a big limitation.
>
> The easiest thing for users would be if we did this conversion
> ourselves and provided both Docker and Singularity images. Provided
I
> have hours left after the prep for the tutorial is completed this is
> something I can look into.
>
> Michael Kavulich, Jr.
> Associate Scientist, National Center for Atmospheric Research (NCAR)
> Joint Numerical Testbed (JNT), Research Applications Laboratory
(RAL)
> * kavulich at ucar.edu <kavulich at ucar.edu>*
>
>
> On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Mike,
> >
> > John Raby is a MET user from the Army Research Lab who is
interested
> > in running MET through a container. We currently have it in a
> > Docker container, but apparently Singularity is now available on
> > their super-computer, Centennial, for testing.
> >
> > He's written MET-Help asking how to convert the MET Docker
container
> > to Singularity for testing on Centennial.
> >
> > I have never run Singularity but know that you have. What should
I do?
> >
> > Thanks,
> > John
> >
> >
> > ---------- Forwarded message ---------
> > From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> > Date: Fri, Nov 9, 2018 at 9:07 AM
> > Subject: [rt.rap.ucar.edu #87728] MET docker-container in
> > Singularity
> > (UNCLASSIFIED)
> > To:
> >
> >
> >
> > Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET docker-container in Singularity (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > 728 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I received the email below from the developer who is implementing
> > Singularity on the HPC in response to my question about running an
> > already-built docker-container using Singularity Runtime. They are
> > asking users to start test runs with their already-built
Singularity
> containers.
> > From his guidance below, can the MET docker-container be made to
run
> > in Singularity Runtime?
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > -----Original Message-----
> > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > Sent: Friday, November 9, 2018 8:35 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: RE: Singularity Runtime installed on Centennial
> > (UNCLASSIFIED)
> >
> >
> > You would need to convert the docker image to a singularity image.
> > From there you might have to change some permissions or re-
configure
> > to use non-root directories. It depends on how tightly coupled the
> > application is to root inside the image. You will need root on a
> > machine to convert the image to Singularity.
> >
> > Thanks,
> > Matt
> >
> > ________________________________________
> > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > Sent: Thursday, November 08, 2018 5:24 PM
> > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > Subject: RE: Singularity Runtime installed on Centennial
> > (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Thanks for all your work on implementing Singularity and for the
> > notification, Matt. I most certainly will send this out to others
I
> > know who are interested. I know of an already-built docker-
container
> > which I'm interested in. Does this fall into the category of being
> > suitable for testing on Centennial?
> >
> > R/
> > John
> >
> >
> >
> > -----Original Message-----
> > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > Sent: Thursday, November 8, 2018 3:18 PM
> > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > <matthew.r.dwyer7.ctr at mail.mil
> > >
> > Subject: Singularity Runtime installed on Centennial
> >
> > Good News!
> >
> >
> > Containers are installed on Centennial (for early access). This is
> > the first production machine container installation at the ARL
DSRC.
> >
> > Singularity runtime is installed on Centennial which means you
still
> > cannot build your containers on ARL DSRC resources. If you have
> > already-built singularity containers and would like to run them
> > please go ahead and test them out now. The more
> > bugs/misconfigurations we find now will provide a smoother
> > user-experience with container workloads. Let me know of successes
> > and failures and please send to
> others who are interested.
> >
> >
> > Thank You,
> > Matt Dwyer
> > 4102786819
> >
> > CLASSIFICATION: UNCLASSIFIED
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET docker-container in Singularity (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Mar 07 08:18:09 2019
CLASSIFICATION: UNCLASSIFIED
Has there been any progress with developing a Singularity version of
the MET container?
R/
John
Mr. John W. Raby
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
-----Original Message-----
From: Raby, John W CIV USARMY RDECOM ARL (US)
Sent: Thursday, January 10, 2019 8:13 AM
To: 'met_help at ucar.edu' <met_help at ucar.edu>
Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
docker-container in Singularity (UNCLASSIFIED)
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for responding while at the mtg and for following up with Mike.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, January 10, 2019 8:04 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
docker-container in Singularity (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
I forwarded your question to Mike Kavulich, who's planning on working
with Singularity. Listed below is my question and his response. We
are both at AMS this week.
---------------------------------------
Mike,
I got a followup question from John Raby from the Army. He's
wondering if we've made any progress in using Singularity containers.
If I remember correctly, once we make containers available via
DockerHub, this is very easy. But until then, it's more difficult.
Do you happen to have a Singularity version of the MET container he
could use for testing?
---------------------------------------
John,
I do not have a singularity container ready, I haven't re-installed
the necessary software since my mac had to be serviced. I can get to
it next week at the earliest.
Mike
---------------------------------------
On Mon, Jan 7, 2019 at 3:01 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Ref the email update below on the possibility of a Singularity
> container for MET. Has there been any progress on this?
>
> Thanks.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Wednesday, November 14, 2018 4:40 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> docker-container in Singularity (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> I forwarded your question about Singularity to other folks in the
DTC
> and am forwarding their response.
>
> Specifically, it looks like once we get the MET container posted to
> DockerHub than importing it into Singularity becomes a one-liner.
> Seems like a good motivation to get that done!
>
> Thanks,
> John
>
> ---------- Forwarded message ---------
> From: Michael Kavulich <kavulich at ucar.edu>
> Date: Wed, Nov 14, 2018 at 12:16 PM
> Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container in
> Singularity
> (UNCLASSIFIED)
> To: John Halley Gotway <johnhg at ucar.edu>
> Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
> <harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
>
>
> John,
>
> I have only tested singularity with standard images from dockerhub.
> This process is relatively easy:
> Caution-
> Caution-https://www.sylabs.io/guides/2.5/user-
guide/singularity_and_do
> cker.html?highlight=docker
>
> Converting images locally is a bit more involved, and I have not
tried
> this
> myself:
> Caution-
> Caution-
https://groups.google.com/a/lbl.gov/forum/#!topic/singularity/
> ZXOaNaM_6MY
>
> This all requires that you are on a machine with both Docker and
> Singularity installed, which is a big limitation.
>
> The easiest thing for users would be if we did this conversion
> ourselves and provided both Docker and Singularity images. Provided
I
> have hours left after the prep for the tutorial is completed this is
> something I can look into.
>
> Michael Kavulich, Jr.
> Associate Scientist, National Center for Atmospheric Research (NCAR)
> Joint Numerical Testbed (JNT), Research Applications Laboratory
(RAL)
> * kavulich at ucar.edu <kavulich at ucar.edu>*
>
>
> On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Mike,
> >
> > John Raby is a MET user from the Army Research Lab who is
interested
> > in running MET through a container. We currently have it in a
> > Docker container, but apparently Singularity is now available on
> > their super-computer, Centennial, for testing.
> >
> > He's written MET-Help asking how to convert the MET Docker
container
> > to Singularity for testing on Centennial.
> >
> > I have never run Singularity but know that you have. What should
I do?
> >
> > Thanks,
> > John
> >
> >
> > ---------- Forwarded message ---------
> > From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> > Date: Fri, Nov 9, 2018 at 9:07 AM
> > Subject: [rt.rap.ucar.edu #87728] MET docker-container in
> > Singularity
> > (UNCLASSIFIED)
> > To:
> >
> >
> >
> > Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET docker-container in Singularity (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > 728 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > I received the email below from the developer who is implementing
> > Singularity on the HPC in response to my question about running an
> > already-built docker-container using Singularity Runtime. They are
> > asking users to start test runs with their already-built
Singularity
> containers.
> > From his guidance below, can the MET docker-container be made to
run
> > in Singularity Runtime?
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > -----Original Message-----
> > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > Sent: Friday, November 9, 2018 8:35 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: RE: Singularity Runtime installed on Centennial
> > (UNCLASSIFIED)
> >
> >
> > You would need to convert the docker image to a singularity image.
> > From there you might have to change some permissions or re-
configure
> > to use non-root directories. It depends on how tightly coupled the
> > application is to root inside the image. You will need root on a
> > machine to convert the image to Singularity.
> >
> > Thanks,
> > Matt
> >
> > ________________________________________
> > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > Sent: Thursday, November 08, 2018 5:24 PM
> > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > Subject: RE: Singularity Runtime installed on Centennial
> > (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Thanks for all your work on implementing Singularity and for the
> > notification, Matt. I most certainly will send this out to others
I
> > know who are interested. I know of an already-built docker-
container
> > which I'm interested in. Does this fall into the category of being
> > suitable for testing on Centennial?
> >
> > R/
> > John
> >
> >
> >
> > -----Original Message-----
> > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > Sent: Thursday, November 8, 2018 3:18 PM
> > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > <matthew.r.dwyer7.ctr at mail.mil
> > >
> > Subject: Singularity Runtime installed on Centennial
> >
> > Good News!
> >
> >
> > Containers are installed on Centennial (for early access). This is
> > the first production machine container installation at the ARL
DSRC.
> >
> > Singularity runtime is installed on Centennial which means you
still
> > cannot build your containers on ARL DSRC resources. If you have
> > already-built singularity containers and would like to run them
> > please go ahead and test them out now. The more
> > bugs/misconfigurations we find now will provide a smoother
> > user-experience with container workloads. Let me know of successes
> > and failures and please send to
> others who are interested.
> >
> >
> > Thank You,
> > Matt Dwyer
> > 4102786819
> >
> > CLASSIFICATION: UNCLASSIFIED
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET docker-container in Singularity (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Mar 14 16:40:14 2019
Hi John,
I wanted to let you know that I just posted the docker image for met-
8.0 up
to DockerHub. So retrieving it is now very easy:
docker pull dtcenter/met:8.0
In addition, once it's on DockerHub, I'm told that running it through
a
singularity container is pretty straight-forward. However, I haven't
actually done so myself:
https://www.sylabs.io/guides/3.0/user-
guide/build_a_container.html#downloading-an-existing-container-from-
docker-hub
Thanks,
John
On Thu, Mar 7, 2019 at 8:18 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Has there been any progress with developing a Singularity version of
the
> MET container?
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
> -----Original Message-----
> From: Raby, John W CIV USARMY RDECOM ARL (US)
> Sent: Thursday, January 10, 2019 8:13 AM
> To: 'met_help at ucar.edu' <met_help at ucar.edu>
> Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> docker-container in Singularity (UNCLASSIFIED)
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for responding while at the mtg and for following up with
Mike.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, January 10, 2019 8:04 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> docker-container in Singularity (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I forwarded your question to Mike Kavulich, who's planning on
working with
> Singularity. Listed below is my question and his response. We are
both at
> AMS this week.
>
> ---------------------------------------
>
> Mike,
>
> I got a followup question from John Raby from the Army. He's
wondering if
> we've made any progress in using Singularity containers.
>
> If I remember correctly, once we make containers available via
DockerHub,
> this is very easy. But until then, it's more difficult. Do you
happen to
> have a Singularity version of the MET container he could use for
testing?
>
> ---------------------------------------
>
> John,
>
> I do not have a singularity container ready, I haven't re-installed
the
> necessary software since my mac had to be serviced. I can get to it
next
> week at the earliest.
>
> Mike
>
> ---------------------------------------
>
>
> On Mon, Jan 7, 2019 at 3:01 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Ref the email update below on the possibility of a Singularity
> > container for MET. Has there been any progress on this?
> >
> > Thanks.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Wednesday, November 14, 2018 4:40 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > docker-container in Singularity (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I forwarded your question about Singularity to other folks in the
DTC
> > and am forwarding their response.
> >
> > Specifically, it looks like once we get the MET container posted
to
> > DockerHub than importing it into Singularity becomes a one-liner.
> > Seems like a good motivation to get that done!
> >
> > Thanks,
> > John
> >
> > ---------- Forwarded message ---------
> > From: Michael Kavulich <kavulich at ucar.edu>
> > Date: Wed, Nov 14, 2018 at 12:16 PM
> > Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container in
> > Singularity
> > (UNCLASSIFIED)
> > To: John Halley Gotway <johnhg at ucar.edu>
> > Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
> > <harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
> >
> >
> > John,
> >
> > I have only tested singularity with standard images from
dockerhub.
> > This process is relatively easy:
> > Caution-
> > Caution-https://www.sylabs.io/guides/2.5/user-
guide/singularity_and_do
> > cker.html?highlight=docker
> >
> > Converting images locally is a bit more involved, and I have not
tried
> > this
> > myself:
> > Caution-
> > Caution-
https://groups.google.com/a/lbl.gov/forum/#!topic/singularity/
> > ZXOaNaM_6MY
> >
> > This all requires that you are on a machine with both Docker and
> > Singularity installed, which is a big limitation.
> >
> > The easiest thing for users would be if we did this conversion
> > ourselves and provided both Docker and Singularity images.
Provided I
> > have hours left after the prep for the tutorial is completed this
is
> > something I can look into.
> >
> > Michael Kavulich, Jr.
> > Associate Scientist, National Center for Atmospheric Research
(NCAR)
> > Joint Numerical Testbed (JNT), Research Applications Laboratory
(RAL)
> > * kavulich at ucar.edu <kavulich at ucar.edu>*
> >
> >
> > On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
> >
> > > Mike,
> > >
> > > John Raby is a MET user from the Army Research Lab who is
interested
> > > in running MET through a container. We currently have it in a
> > > Docker container, but apparently Singularity is now available on
> > > their super-computer, Centennial, for testing.
> > >
> > > He's written MET-Help asking how to convert the MET Docker
container
> > > to Singularity for testing on Centennial.
> > >
> > > I have never run Singularity but know that you have. What
should I do?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > ---------- Forwarded message ---------
> > > From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> > > Date: Fri, Nov 9, 2018 at 9:07 AM
> > > Subject: [rt.rap.ucar.edu #87728] MET docker-container in
> > > Singularity
> > > (UNCLASSIFIED)
> > > To:
> > >
> > >
> > >
> > > Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: MET docker-container in Singularity (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > > 728 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > I received the email below from the developer who is
implementing
> > > Singularity on the HPC in response to my question about running
an
> > > already-built docker-container using Singularity Runtime. They
are
> > > asking users to start test runs with their already-built
Singularity
> > containers.
> > > From his guidance below, can the MET docker-container be made to
run
> > > in Singularity Runtime?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > (575) 678-2004 DSN 258-2004
> > > FAX (575) 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > > -----Original Message-----
> > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > Sent: Friday, November 9, 2018 8:35 AM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: RE: Singularity Runtime installed on Centennial
> > > (UNCLASSIFIED)
> > >
> > >
> > > You would need to convert the docker image to a singularity
image.
> > > From there you might have to change some permissions or re-
configure
> > > to use non-root directories. It depends on how tightly coupled
the
> > > application is to root inside the image. You will need root on a
> > > machine to convert the image to Singularity.
> > >
> > > Thanks,
> > > Matt
> > >
> > > ________________________________________
> > > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Sent: Thursday, November 08, 2018 5:24 PM
> > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > Subject: RE: Singularity Runtime installed on Centennial
> > > (UNCLASSIFIED)
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > Thanks for all your work on implementing Singularity and for the
> > > notification, Matt. I most certainly will send this out to
others I
> > > know who are interested. I know of an already-built docker-
container
> > > which I'm interested in. Does this fall into the category of
being
> > > suitable for testing on Centennial?
> > >
> > > R/
> > > John
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > Sent: Thursday, November 8, 2018 3:18 PM
> > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > <matthew.r.dwyer7.ctr at mail.mil
> > > >
> > > Subject: Singularity Runtime installed on Centennial
> > >
> > > Good News!
> > >
> > >
> > > Containers are installed on Centennial (for early access). This
is
> > > the first production machine container installation at the ARL
DSRC.
> > >
> > > Singularity runtime is installed on Centennial which means you
still
> > > cannot build your containers on ARL DSRC resources. If you have
> > > already-built singularity containers and would like to run them
> > > please go ahead and test them out now. The more
> > > bugs/misconfigurations we find now will provide a smoother
> > > user-experience with container workloads. Let me know of
successes
> > > and failures and please send to
> > others who are interested.
> > >
> > >
> > > Thank You,
> > > Matt Dwyer
> > > 4102786819
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET docker-container in Singularity (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Fri Mar 15 06:45:10 2019
CLASSIFICATION: UNCLASSIFIED
Thanks for the update, John. I'm at home today, but I check email once
in a while.
So I understand this correctly, the version of MET in DockerHub is
capable of being run in a singularity container?
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, March 14, 2019 4:40 PM
To: Raby, John W CIV USARMY FUTURES COMMAND (USA)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
docker-container in Singularity (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
Hi John,
I wanted to let you know that I just posted the docker image for met-
8.0 up to DockerHub. So retrieving it is now very easy:
docker pull dtcenter/met:8.0
In addition, once it's on DockerHub, I'm told that running it through
a singularity container is pretty straight-forward. However, I
haven't actually done so myself:
Caution-https://www.sylabs.io/guides/3.0/user-
guide/build_a_container.html#downloading-an-existing-container-from-
docker-hub
Thanks,
John
On Thu, Mar 7, 2019 at 8:18 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Has there been any progress with developing a Singularity version of
> the MET container?
>
> R/
> John
>
> Mr. John W. Raby
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
> -----Original Message-----
> From: Raby, John W CIV USARMY RDECOM ARL (US)
> Sent: Thursday, January 10, 2019 8:13 AM
> To: 'met_help at ucar.edu' <met_help at ucar.edu>
> Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> docker-container in Singularity (UNCLASSIFIED)
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for responding while at the mtg and for following up with
Mike.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Thursday, January 10, 2019 8:04 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> docker-container in Singularity (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> I forwarded your question to Mike Kavulich, who's planning on
working
> with Singularity. Listed below is my question and his response. We
> are both at AMS this week.
>
> ---------------------------------------
>
> Mike,
>
> I got a followup question from John Raby from the Army. He's
> wondering if we've made any progress in using Singularity
containers.
>
> If I remember correctly, once we make containers available via
> DockerHub, this is very easy. But until then, it's more difficult.
> Do you happen to have a Singularity version of the MET container he
could use for testing?
>
> ---------------------------------------
>
> John,
>
> I do not have a singularity container ready, I haven't re-installed
> the necessary software since my mac had to be serviced. I can get to
> it next week at the earliest.
>
> Mike
>
> ---------------------------------------
>
>
> On Mon, Jan 7, 2019 at 3:01 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > 728 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Ref the email update below on the possibility of a Singularity
> > container for MET. Has there been any progress on this?
> >
> > Thanks.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Wednesday, November 14, 2018 4:40 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > docker-container in Singularity (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I forwarded your question about Singularity to other folks in the
> > DTC and am forwarding their response.
> >
> > Specifically, it looks like once we get the MET container posted
to
> > DockerHub than importing it into Singularity becomes a one-liner.
> > Seems like a good motivation to get that done!
> >
> > Thanks,
> > John
> >
> > ---------- Forwarded message ---------
> > From: Michael Kavulich <kavulich at ucar.edu>
> > Date: Wed, Nov 14, 2018 at 12:16 PM
> > Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container in
> > Singularity
> > (UNCLASSIFIED)
> > To: John Halley Gotway <johnhg at ucar.edu>
> > Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
> > <harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
> >
> >
> > John,
> >
> > I have only tested singularity with standard images from
dockerhub.
> > This process is relatively easy:
> > Caution-
> > Caution-Caution-https://www.sylabs.io/guides/2.5/user-
guide/singular
> > ity_and_do
> > cker.html?highlight=docker
> >
> > Converting images locally is a bit more involved, and I have not
> > tried this
> > myself:
> > Caution-
> > Caution-Caution-
https://groups.google.com/a/lbl.gov/forum/#!topic/si
> > ngularity/
> > ZXOaNaM_6MY
> >
> > This all requires that you are on a machine with both Docker and
> > Singularity installed, which is a big limitation.
> >
> > The easiest thing for users would be if we did this conversion
> > ourselves and provided both Docker and Singularity images.
Provided
> > I have hours left after the prep for the tutorial is completed
this
> > is something I can look into.
> >
> > Michael Kavulich, Jr.
> > Associate Scientist, National Center for Atmospheric Research
(NCAR)
> > Joint Numerical Testbed (JNT), Research Applications Laboratory
> > (RAL)
> > * kavulich at ucar.edu <kavulich at ucar.edu>*
> >
> >
> > On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
> >
> > > Mike,
> > >
> > > John Raby is a MET user from the Army Research Lab who is
> > > interested in running MET through a container. We currently
have
> > > it in a Docker container, but apparently Singularity is now
> > > available on their super-computer, Centennial, for testing.
> > >
> > > He's written MET-Help asking how to convert the MET Docker
> > > container to Singularity for testing on Centennial.
> > >
> > > I have never run Singularity but know that you have. What
should I do?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > ---------- Forwarded message ---------
> > > From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> > > Date: Fri, Nov 9, 2018 at 9:07 AM
> > > Subject: [rt.rap.ucar.edu #87728] MET docker-container in
> > > Singularity
> > > (UNCLASSIFIED)
> > > To:
> > >
> > >
> > >
> > > Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: MET docker-container in Singularity (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=87
> > > 728 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > I received the email below from the developer who is
implementing
> > > Singularity on the HPC in response to my question about running
an
> > > already-built docker-container using Singularity Runtime. They
are
> > > asking users to start test runs with their already-built
> > > Singularity
> > containers.
> > > From his guidance below, can the MET docker-container be made to
> > > run in Singularity Runtime?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > (575) 678-2004 DSN 258-2004
> > > FAX (575) 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > > -----Original Message-----
> > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > Sent: Friday, November 9, 2018 8:35 AM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: RE: Singularity Runtime installed on Centennial
> > > (UNCLASSIFIED)
> > >
> > >
> > > You would need to convert the docker image to a singularity
image.
> > > From there you might have to change some permissions or
> > > re-configure to use non-root directories. It depends on how
> > > tightly coupled the application is to root inside the image. You
> > > will need root on a machine to convert the image to Singularity.
> > >
> > > Thanks,
> > > Matt
> > >
> > > ________________________________________
> > > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Sent: Thursday, November 08, 2018 5:24 PM
> > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > Subject: RE: Singularity Runtime installed on Centennial
> > > (UNCLASSIFIED)
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > Thanks for all your work on implementing Singularity and for the
> > > notification, Matt. I most certainly will send this out to
others
> > > I know who are interested. I know of an already-built
> > > docker-container which I'm interested in. Does this fall into
the
> > > category of being suitable for testing on Centennial?
> > >
> > > R/
> > > John
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > Sent: Thursday, November 8, 2018 3:18 PM
> > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > <matthew.r.dwyer7.ctr at mail.mil
> > > >
> > > Subject: Singularity Runtime installed on Centennial
> > >
> > > Good News!
> > >
> > >
> > > Containers are installed on Centennial (for early access). This
is
> > > the first production machine container installation at the ARL
DSRC.
> > >
> > > Singularity runtime is installed on Centennial which means you
> > > still cannot build your containers on ARL DSRC resources. If you
> > > have already-built singularity containers and would like to run
> > > them please go ahead and test them out now. The more
> > > bugs/misconfigurations we find now will provide a smoother
> > > user-experience with container workloads. Let me know of
successes
> > > and failures and please send to
> > others who are interested.
> > >
> > >
> > > Thank You,
> > > Matt Dwyer
> > > 4102786819
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET docker-container in Singularity (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Mar 15 09:15:05 2019
John,
Yes, that's the idea. As long as the image lives on DockerHub, you
can run
a single command to build a singularity container from it. I have
never
actually done so myself since I don't have singularity installed on my
laptop, and it's not yet available on our HPC's.
Just take a look at the link I sent for an example of doing so.
Thanks,
John
On Fri, Mar 15, 2019 at 6:45 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Thanks for the update, John. I'm at home today, but I check email
once in
> a while.
>
> So I understand this correctly, the version of MET in DockerHub is
capable
> of being run in a singularity container?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, March 14, 2019 4:40 PM
> To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
> john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> docker-container in Singularity (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> Hi John,
>
> I wanted to let you know that I just posted the docker image for
met-8.0
> up to DockerHub. So retrieving it is now very easy:
>
> docker pull dtcenter/met:8.0
>
> In addition, once it's on DockerHub, I'm told that running it
through a
> singularity container is pretty straight-forward. However, I
haven't
> actually done so myself:
>
> Caution-
> https://www.sylabs.io/guides/3.0/user-
guide/build_a_container.html#downloading-an-existing-container-from-
docker-hub
>
> Thanks,
> John
>
> On Thu, Mar 7, 2019 at 8:18 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Has there been any progress with developing a Singularity version
of
> > the MET container?
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > -----Original Message-----
> > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > Sent: Thursday, January 10, 2019 8:13 AM
> > To: 'met_help at ucar.edu' <met_help at ucar.edu>
> > Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > docker-container in Singularity (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for responding while at the mtg and for following up with
Mike.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Thursday, January 10, 2019 8:04 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > docker-container in Singularity (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I forwarded your question to Mike Kavulich, who's planning on
working
> > with Singularity. Listed below is my question and his response.
We
> > are both at AMS this week.
> >
> > ---------------------------------------
> >
> > Mike,
> >
> > I got a followup question from John Raby from the Army. He's
> > wondering if we've made any progress in using Singularity
containers.
> >
> > If I remember correctly, once we make containers available via
> > DockerHub, this is very easy. But until then, it's more
difficult.
> > Do you happen to have a Singularity version of the MET container
he
> could use for testing?
> >
> > ---------------------------------------
> >
> > John,
> >
> > I do not have a singularity container ready, I haven't re-
installed
> > the necessary software since my mac had to be serviced. I can get
to
> > it next week at the earliest.
> >
> > Mike
> >
> > ---------------------------------------
> >
> >
> > On Mon, Jan 7, 2019 at 3:01 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > > 728 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Ref the email update below on the possibility of a Singularity
> > > container for MET. Has there been any progress on this?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Wednesday, November 14, 2018 4:40 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > > docker-container in Singularity (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > I forwarded your question about Singularity to other folks in
the
> > > DTC and am forwarding their response.
> > >
> > > Specifically, it looks like once we get the MET container posted
to
> > > DockerHub than importing it into Singularity becomes a one-
liner.
> > > Seems like a good motivation to get that done!
> > >
> > > Thanks,
> > > John
> > >
> > > ---------- Forwarded message ---------
> > > From: Michael Kavulich <kavulich at ucar.edu>
> > > Date: Wed, Nov 14, 2018 at 12:16 PM
> > > Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container in
> > > Singularity
> > > (UNCLASSIFIED)
> > > To: John Halley Gotway <johnhg at ucar.edu>
> > > Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
> > > <harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
> > >
> > >
> > > John,
> > >
> > > I have only tested singularity with standard images from
dockerhub.
> > > This process is relatively easy:
> > > Caution-
> > > Caution-Caution-https://www.sylabs.io/guides/2.5/user-
guide/singular
> > > ity_and_do
> > > cker.html?highlight=docker
> > >
> > > Converting images locally is a bit more involved, and I have not
> > > tried this
> > > myself:
> > > Caution-
> > > Caution-Caution-
https://groups.google.com/a/lbl.gov/forum/#!topic/si
> > > ngularity/
> > > ZXOaNaM_6MY
> > >
> > > This all requires that you are on a machine with both Docker and
> > > Singularity installed, which is a big limitation.
> > >
> > > The easiest thing for users would be if we did this conversion
> > > ourselves and provided both Docker and Singularity images.
Provided
> > > I have hours left after the prep for the tutorial is completed
this
> > > is something I can look into.
> > >
> > > Michael Kavulich, Jr.
> > > Associate Scientist, National Center for Atmospheric Research
(NCAR)
> > > Joint Numerical Testbed (JNT), Research Applications Laboratory
> > > (RAL)
> > > * kavulich at ucar.edu <kavulich at ucar.edu>*
> > >
> > >
> > > On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> > >
> > > > Mike,
> > > >
> > > > John Raby is a MET user from the Army Research Lab who is
> > > > interested in running MET through a container. We currently
have
> > > > it in a Docker container, but apparently Singularity is now
> > > > available on their super-computer, Centennial, for testing.
> > > >
> > > > He's written MET-Help asking how to convert the MET Docker
> > > > container to Singularity for testing on Centennial.
> > > >
> > > > I have never run Singularity but know that you have. What
should I
> do?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > ---------- Forwarded message ---------
> > > > From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> > > > Date: Fri, Nov 9, 2018 at 9:07 AM
> > > > Subject: [rt.rap.ucar.edu #87728] MET docker-container in
> > > > Singularity
> > > > (UNCLASSIFIED)
> > > > To:
> > > >
> > > >
> > > >
> > > > Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: MET docker-container in Singularity
(UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=87
> > > > 728 >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > I received the email below from the developer who is
implementing
> > > > Singularity on the HPC in response to my question about
running an
> > > > already-built docker-container using Singularity Runtime. They
are
> > > > asking users to start test runs with their already-built
> > > > Singularity
> > > containers.
> > > > From his guidance below, can the MET docker-container be made
to
> > > > run in Singularity Runtime?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > Mr. John W. Raby
> > > > U.S. Army Research Laboratory
> > > > White Sands Missile Range, NM 88002
> > > > (575) 678-2004 DSN 258-2004
> > > > FAX (575) 678-1230 DSN 258-1230
> > > > Email: john.w.raby2.civ at mail.mil
> > > >
> > > > -----Original Message-----
> > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > Sent: Friday, November 9, 2018 8:35 AM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > <john.w.raby2.civ at mail.mil>
> > > > Subject: RE: Singularity Runtime installed on Centennial
> > > > (UNCLASSIFIED)
> > > >
> > > >
> > > > You would need to convert the docker image to a singularity
image.
> > > > From there you might have to change some permissions or
> > > > re-configure to use non-root directories. It depends on how
> > > > tightly coupled the application is to root inside the image.
You
> > > > will need root on a machine to convert the image to
Singularity.
> > > >
> > > > Thanks,
> > > > Matt
> > > >
> > > > ________________________________________
> > > > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > Sent: Thursday, November 08, 2018 5:24 PM
> > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > Subject: RE: Singularity Runtime installed on Centennial
> > > > (UNCLASSIFIED)
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > Thanks for all your work on implementing Singularity and for
the
> > > > notification, Matt. I most certainly will send this out to
others
> > > > I know who are interested. I know of an already-built
> > > > docker-container which I'm interested in. Does this fall into
the
> > > > category of being suitable for testing on Centennial?
> > > >
> > > > R/
> > > > John
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > Sent: Thursday, November 8, 2018 3:18 PM
> > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > <matthew.r.dwyer7.ctr at mail.mil
> > > > >
> > > > Subject: Singularity Runtime installed on Centennial
> > > >
> > > > Good News!
> > > >
> > > >
> > > > Containers are installed on Centennial (for early access).
This is
> > > > the first production machine container installation at the ARL
DSRC.
> > > >
> > > > Singularity runtime is installed on Centennial which means you
> > > > still cannot build your containers on ARL DSRC resources. If
you
> > > > have already-built singularity containers and would like to
run
> > > > them please go ahead and test them out now. The more
> > > > bugs/misconfigurations we find now will provide a smoother
> > > > user-experience with container workloads. Let me know of
successes
> > > > and failures and please send to
> > > others who are interested.
> > > >
> > > >
> > > > Thank You,
> > > > Matt Dwyer
> > > > 4102786819
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET docker-container in Singularity (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Mon Mar 18 07:20:26 2019
John -
Thanks for the clarification. I tried accessing the link you provided
but it's timing out on both attempts using firefox. I plan to provide
a summary of this latest info about how MET in DockerHub can be built
as a singularity container to my POC for the Army HPC who installed
singularity and was interested in doing some testing. I anticipate
that he will have some technical questions which will probably be over
my head, so I may ask for some input from you depending on what the
question is. How does this sound to you?
R/
John
____________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Friday, March 15, 2019 9:15 AM
To: Raby, John W CIV USARMY FUTURES COMMAND (USA)
Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
docker-container in Singularity (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Yes, that's the idea. As long as the image lives on DockerHub, you
can run
a single command to build a singularity container from it. I have
never
actually done so myself since I don't have singularity installed on my
laptop, and it's not yet available on our HPC's.
Just take a look at the link I sent for an example of doing so.
Thanks,
John
On Fri, Mar 15, 2019 at 6:45 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Thanks for the update, John. I'm at home today, but I check email
once in
> a while.
>
> So I understand this correctly, the version of MET in DockerHub is
capable
> of being run in a singularity container?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Thursday, March 14, 2019 4:40 PM
> To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
> john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> docker-container in Singularity (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> Hi John,
>
> I wanted to let you know that I just posted the docker image for
met-8.0
> up to DockerHub. So retrieving it is now very easy:
>
> docker pull dtcenter/met:8.0
>
> In addition, once it's on DockerHub, I'm told that running it
through a
> singularity container is pretty straight-forward. However, I
haven't
> actually done so myself:
>
> Caution-
> Caution-https://www.sylabs.io/guides/3.0/user-
guide/build_a_container.html#downloading-an-existing-container-from-
docker-hub
>
> Thanks,
> John
>
> On Thu, Mar 7, 2019 at 8:18 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Has there been any progress with developing a Singularity version
of
> > the MET container?
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > -----Original Message-----
> > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > Sent: Thursday, January 10, 2019 8:13 AM
> > To: 'met_help at ucar.edu' <met_help at ucar.edu>
> > Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > docker-container in Singularity (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for responding while at the mtg and for following up with
Mike.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-Caution-
mailto:met_help at ucar.edu]
> > Sent: Thursday, January 10, 2019 8:04 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > docker-container in Singularity (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I forwarded your question to Mike Kavulich, who's planning on
working
> > with Singularity. Listed below is my question and his response.
We
> > are both at AMS this week.
> >
> > ---------------------------------------
> >
> > Mike,
> >
> > I got a followup question from John Raby from the Army. He's
> > wondering if we've made any progress in using Singularity
containers.
> >
> > If I remember correctly, once we make containers available via
> > DockerHub, this is very easy. But until then, it's more
difficult.
> > Do you happen to have a Singularity version of the MET container
he
> could use for testing?
> >
> > ---------------------------------------
> >
> > John,
> >
> > I do not have a singularity container ready, I haven't re-
installed
> > the necessary software since my mac had to be serviced. I can get
to
> > it next week at the earliest.
> >
> > Mike
> >
> > ---------------------------------------
> >
> >
> > On Mon, Jan 7, 2019 at 3:01 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > > 728 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Ref the email update below on the possibility of a Singularity
> > > container for MET. Has there been any progress on this?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Wednesday, November 14, 2018 4:40 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > > docker-container in Singularity (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > I forwarded your question about Singularity to other folks in
the
> > > DTC and am forwarding their response.
> > >
> > > Specifically, it looks like once we get the MET container posted
to
> > > DockerHub than importing it into Singularity becomes a one-
liner.
> > > Seems like a good motivation to get that done!
> > >
> > > Thanks,
> > > John
> > >
> > > ---------- Forwarded message ---------
> > > From: Michael Kavulich <kavulich at ucar.edu>
> > > Date: Wed, Nov 14, 2018 at 12:16 PM
> > > Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container in
> > > Singularity
> > > (UNCLASSIFIED)
> > > To: John Halley Gotway <johnhg at ucar.edu>
> > > Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
> > > <harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
> > >
> > >
> > > John,
> > >
> > > I have only tested singularity with standard images from
dockerhub.
> > > This process is relatively easy:
> > > Caution-
> > > Caution-Caution-Caution-https://www.sylabs.io/guides/2.5/user-
guide/singular
> > > ity_and_do
> > > cker.html?highlight=docker
> > >
> > > Converting images locally is a bit more involved, and I have not
> > > tried this
> > > myself:
> > > Caution-
> > > Caution-Caution-Caution-
https://groups.google.com/a/lbl.gov/forum/#!topic/si
> > > ngularity/
> > > ZXOaNaM_6MY
> > >
> > > This all requires that you are on a machine with both Docker and
> > > Singularity installed, which is a big limitation.
> > >
> > > The easiest thing for users would be if we did this conversion
> > > ourselves and provided both Docker and Singularity images.
Provided
> > > I have hours left after the prep for the tutorial is completed
this
> > > is something I can look into.
> > >
> > > Michael Kavulich, Jr.
> > > Associate Scientist, National Center for Atmospheric Research
(NCAR)
> > > Joint Numerical Testbed (JNT), Research Applications Laboratory
> > > (RAL)
> > > * kavulich at ucar.edu <kavulich at ucar.edu>*
> > >
> > >
> > > On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> > >
> > > > Mike,
> > > >
> > > > John Raby is a MET user from the Army Research Lab who is
> > > > interested in running MET through a container. We currently
have
> > > > it in a Docker container, but apparently Singularity is now
> > > > available on their super-computer, Centennial, for testing.
> > > >
> > > > He's written MET-Help asking how to convert the MET Docker
> > > > container to Singularity for testing on Centennial.
> > > >
> > > > I have never run Singularity but know that you have. What
should I
> do?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > ---------- Forwarded message ---------
> > > > From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> > > > Date: Fri, Nov 9, 2018 at 9:07 AM
> > > > Subject: [rt.rap.ucar.edu #87728] MET docker-container in
> > > > Singularity
> > > > (UNCLASSIFIED)
> > > > To:
> > > >
> > > >
> > > >
> > > > Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: MET docker-container in Singularity
(UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=87
> > > > 728 >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > I received the email below from the developer who is
implementing
> > > > Singularity on the HPC in response to my question about
running an
> > > > already-built docker-container using Singularity Runtime. They
are
> > > > asking users to start test runs with their already-built
> > > > Singularity
> > > containers.
> > > > From his guidance below, can the MET docker-container be made
to
> > > > run in Singularity Runtime?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > Mr. John W. Raby
> > > > U.S. Army Research Laboratory
> > > > White Sands Missile Range, NM 88002
> > > > (575) 678-2004 DSN 258-2004
> > > > FAX (575) 678-1230 DSN 258-1230
> > > > Email: john.w.raby2.civ at mail.mil
> > > >
> > > > -----Original Message-----
> > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > Sent: Friday, November 9, 2018 8:35 AM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > <john.w.raby2.civ at mail.mil>
> > > > Subject: RE: Singularity Runtime installed on Centennial
> > > > (UNCLASSIFIED)
> > > >
> > > >
> > > > You would need to convert the docker image to a singularity
image.
> > > > From there you might have to change some permissions or
> > > > re-configure to use non-root directories. It depends on how
> > > > tightly coupled the application is to root inside the image.
You
> > > > will need root on a machine to convert the image to
Singularity.
> > > >
> > > > Thanks,
> > > > Matt
> > > >
> > > > ________________________________________
> > > > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > Sent: Thursday, November 08, 2018 5:24 PM
> > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > Subject: RE: Singularity Runtime installed on Centennial
> > > > (UNCLASSIFIED)
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > Thanks for all your work on implementing Singularity and for
the
> > > > notification, Matt. I most certainly will send this out to
others
> > > > I know who are interested. I know of an already-built
> > > > docker-container which I'm interested in. Does this fall into
the
> > > > category of being suitable for testing on Centennial?
> > > >
> > > > R/
> > > > John
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > Sent: Thursday, November 8, 2018 3:18 PM
> > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > <matthew.r.dwyer7.ctr at mail.mil
> > > > >
> > > > Subject: Singularity Runtime installed on Centennial
> > > >
> > > > Good News!
> > > >
> > > >
> > > > Containers are installed on Centennial (for early access).
This is
> > > > the first production machine container installation at the ARL
DSRC.
> > > >
> > > > Singularity runtime is installed on Centennial which means you
> > > > still cannot build your containers on ARL DSRC resources. If
you
> > > > have already-built singularity containers and would like to
run
> > > > them please go ahead and test them out now. The more
> > > > bugs/misconfigurations we find now will provide a smoother
> > > > user-experience with container workloads. Let me know of
successes
> > > > and failures and please send to
> > > others who are interested.
> > > >
> > > >
> > > > Thank You,
> > > > Matt Dwyer
> > > > 4102786819
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET docker-container in Singularity (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Mon Mar 18 08:06:06 2019
CLASSIFICATION: UNCLASSIFIED
John -
I just tried access the link from my windows laptop using IE and
firefox and get the same issue. IE reports can't access this page.
Firefox just times out.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, March 15, 2019 9:15 AM
To: Raby, John W CIV USARMY FUTURES COMMAND (USA)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
docker-container in Singularity (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Yes, that's the idea. As long as the image lives on DockerHub, you
can run a single command to build a singularity container from it. I
have never actually done so myself since I don't have singularity
installed on my laptop, and it's not yet available on our HPC's.
Just take a look at the link I sent for an example of doing so.
Thanks,
John
On Fri, Mar 15, 2019 at 6:45 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> Thanks for the update, John. I'm at home today, but I check email
once
> in a while.
>
> So I understand this correctly, the version of MET in DockerHub is
> capable of being run in a singularity container?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Thursday, March 14, 2019 4:40 PM
> To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
> john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> docker-container in Singularity (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> Hi John,
>
> I wanted to let you know that I just posted the docker image for
> met-8.0 up to DockerHub. So retrieving it is now very easy:
>
> docker pull dtcenter/met:8.0
>
> In addition, once it's on DockerHub, I'm told that running it
through
> a singularity container is pretty straight-forward. However, I
> haven't actually done so myself:
>
> Caution-
> Caution-https://www.sylabs.io/guides/3.0/user-
guide/build_a_container.
> html#downloading-an-existing-container-from-docker-hub
>
> Thanks,
> John
>
> On Thu, Mar 7, 2019 at 8:18 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > 728 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Has there been any progress with developing a Singularity version
of
> > the MET container?
> >
> > R/
> > John
> >
> > Mr. John W. Raby
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> > -----Original Message-----
> > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > Sent: Thursday, January 10, 2019 8:13 AM
> > To: 'met_help at ucar.edu' <met_help at ucar.edu>
> > Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > docker-container in Singularity (UNCLASSIFIED)
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for responding while at the mtg and for following up with
Mike.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Thursday, January 10, 2019 8:04 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > docker-container in Singularity (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I forwarded your question to Mike Kavulich, who's planning on
> > working with Singularity. Listed below is my question and his
> > response. We are both at AMS this week.
> >
> > ---------------------------------------
> >
> > Mike,
> >
> > I got a followup question from John Raby from the Army. He's
> > wondering if we've made any progress in using Singularity
containers.
> >
> > If I remember correctly, once we make containers available via
> > DockerHub, this is very easy. But until then, it's more
difficult.
> > Do you happen to have a Singularity version of the MET container
he
> could use for testing?
> >
> > ---------------------------------------
> >
> > John,
> >
> > I do not have a singularity container ready, I haven't re-
installed
> > the necessary software since my mac had to be serviced. I can get
to
> > it next week at the earliest.
> >
> > Mike
> >
> > ---------------------------------------
> >
> >
> > On Mon, Jan 7, 2019 at 3:01 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=87
> > > 728 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Ref the email update below on the possibility of a Singularity
> > > container for MET. Has there been any progress on this?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Wednesday, November 14, 2018 4:40 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > > docker-container in Singularity (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
> > > pasting the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > I forwarded your question about Singularity to other folks in
the
> > > DTC and am forwarding their response.
> > >
> > > Specifically, it looks like once we get the MET container posted
> > > to DockerHub than importing it into Singularity becomes a one-
liner.
> > > Seems like a good motivation to get that done!
> > >
> > > Thanks,
> > > John
> > >
> > > ---------- Forwarded message ---------
> > > From: Michael Kavulich <kavulich at ucar.edu>
> > > Date: Wed, Nov 14, 2018 at 12:16 PM
> > > Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container in
> > > Singularity
> > > (UNCLASSIFIED)
> > > To: John Halley Gotway <johnhg at ucar.edu>
> > > Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
> > > <harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
> > >
> > >
> > > John,
> > >
> > > I have only tested singularity with standard images from
dockerhub.
> > > This process is relatively easy:
> > > Caution-
> > > Caution-Caution-Caution-https://www.sylabs.io/guides/2.5/user-
guid
> > > e/singular
> > > ity_and_do
> > > cker.html?highlight=docker
> > >
> > > Converting images locally is a bit more involved, and I have not
> > > tried this
> > > myself:
> > > Caution-
> > > Caution-Caution-Caution-
https://groups.google.com/a/lbl.gov/forum/
> > > #!topic/si
> > > ngularity/
> > > ZXOaNaM_6MY
> > >
> > > This all requires that you are on a machine with both Docker and
> > > Singularity installed, which is a big limitation.
> > >
> > > The easiest thing for users would be if we did this conversion
> > > ourselves and provided both Docker and Singularity images.
> > > Provided I have hours left after the prep for the tutorial is
> > > completed this is something I can look into.
> > >
> > > Michael Kavulich, Jr.
> > > Associate Scientist, National Center for Atmospheric Research
> > > (NCAR) Joint Numerical Testbed (JNT), Research Applications
> > > Laboratory
> > > (RAL)
> > > * kavulich at ucar.edu <kavulich at ucar.edu>*
> > >
> > >
> > > On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway
> > > <johnhg at ucar.edu>
> > wrote:
> > >
> > > > Mike,
> > > >
> > > > John Raby is a MET user from the Army Research Lab who is
> > > > interested in running MET through a container. We currently
> > > > have it in a Docker container, but apparently Singularity is
now
> > > > available on their super-computer, Centennial, for testing.
> > > >
> > > > He's written MET-Help asking how to convert the MET Docker
> > > > container to Singularity for testing on Centennial.
> > > >
> > > > I have never run Singularity but know that you have. What
> > > > should I
> do?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > > ---------- Forwarded message ---------
> > > > From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> > > > Date: Fri, Nov 9, 2018 at 9:07 AM
> > > > Subject: [rt.rap.ucar.edu #87728] MET docker-container in
> > > > Singularity
> > > > (UNCLASSIFIED)
> > > > To:
> > > >
> > > >
> > > >
> > > > Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: MET docker-container in Singularity
(UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=87
> > > > 728 >
> > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > I received the email below from the developer who is
> > > > implementing Singularity on the HPC in response to my question
> > > > about running an already-built docker-container using
> > > > Singularity Runtime. They are asking users to start test runs
> > > > with their already-built Singularity
> > > containers.
> > > > From his guidance below, can the MET docker-container be made
to
> > > > run in Singularity Runtime?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > Mr. John W. Raby
> > > > U.S. Army Research Laboratory
> > > > White Sands Missile Range, NM 88002
> > > > (575) 678-2004 DSN 258-2004
> > > > FAX (575) 678-1230 DSN 258-1230
> > > > Email: john.w.raby2.civ at mail.mil
> > > >
> > > > -----Original Message-----
> > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > Sent: Friday, November 9, 2018 8:35 AM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > <john.w.raby2.civ at mail.mil>
> > > > Subject: RE: Singularity Runtime installed on Centennial
> > > > (UNCLASSIFIED)
> > > >
> > > >
> > > > You would need to convert the docker image to a singularity
image.
> > > > From there you might have to change some permissions or
> > > > re-configure to use non-root directories. It depends on how
> > > > tightly coupled the application is to root inside the image.
You
> > > > will need root on a machine to convert the image to
Singularity.
> > > >
> > > > Thanks,
> > > > Matt
> > > >
> > > > ________________________________________
> > > > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > Sent: Thursday, November 08, 2018 5:24 PM
> > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > Subject: RE: Singularity Runtime installed on Centennial
> > > > (UNCLASSIFIED)
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > Thanks for all your work on implementing Singularity and for
the
> > > > notification, Matt. I most certainly will send this out to
> > > > others I know who are interested. I know of an already-built
> > > > docker-container which I'm interested in. Does this fall into
> > > > the category of being suitable for testing on Centennial?
> > > >
> > > > R/
> > > > John
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > Sent: Thursday, November 8, 2018 3:18 PM
> > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > <matthew.r.dwyer7.ctr at mail.mil
> > > > >
> > > > Subject: Singularity Runtime installed on Centennial
> > > >
> > > > Good News!
> > > >
> > > >
> > > > Containers are installed on Centennial (for early access).
This
> > > > is the first production machine container installation at the
ARL DSRC.
> > > >
> > > > Singularity runtime is installed on Centennial which means you
> > > > still cannot build your containers on ARL DSRC resources. If
you
> > > > have already-built singularity containers and would like to
run
> > > > them please go ahead and test them out now. The more
> > > > bugs/misconfigurations we find now will provide a smoother
> > > > user-experience with container workloads. Let me know of
> > > > successes and failures and please send to
> > > others who are interested.
> > > >
> > > >
> > > > Thank You,
> > > > Matt Dwyer
> > > > 4102786819
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET docker-container in Singularity (UNCLASSIFIED)
From: John Halley Gotway
Time: Mon Mar 18 11:54:25 2019
John,
Very odd. I just googled "singularity and docker". And the first hit
is:
https://singularity.lbl.gov/docs-docker
At the top of this page it redirects me to a new URL here:
https://www.sylabs.io/docs/
Hopefully you can track down the content on one of these links.
John
On Mon, Mar 18, 2019 at 8:06 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I just tried access the link from my windows laptop using IE and
firefox
> and get the same issue. IE reports can't access this page. Firefox
just
> times out.
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, March 15, 2019 9:15 AM
> To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
> john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> docker-container in Singularity (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Yes, that's the idea. As long as the image lives on DockerHub, you
can
> run a single command to build a singularity container from it. I
have
> never actually done so myself since I don't have singularity
installed on
> my laptop, and it's not yet available on our HPC's.
>
> Just take a look at the link I sent for an example of doing so.
>
> Thanks,
> John
>
> On Fri, Mar 15, 2019 at 6:45 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Thanks for the update, John. I'm at home today, but I check email
once
> > in a while.
> >
> > So I understand this correctly, the version of MET in DockerHub is
> > capable of being run in a singularity container?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Thursday, March 14, 2019 4:40 PM
> > To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
> > john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > docker-container in Singularity (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > Hi John,
> >
> > I wanted to let you know that I just posted the docker image for
> > met-8.0 up to DockerHub. So retrieving it is now very easy:
> >
> > docker pull dtcenter/met:8.0
> >
> > In addition, once it's on DockerHub, I'm told that running it
through
> > a singularity container is pretty straight-forward. However, I
> > haven't actually done so myself:
> >
> > Caution-
> > Caution-https://www.sylabs.io/guides/3.0/user-
guide/build_a_container.
> > html#downloading-an-existing-container-from-docker-hub
> >
> > Thanks,
> > John
> >
> > On Thu, Mar 7, 2019 at 8:18 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > > 728 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > Has there been any progress with developing a Singularity
version of
> > > the MET container?
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > (575) 678-2004 DSN 258-2004
> > > FAX (575) 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > > -----Original Message-----
> > > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Sent: Thursday, January 10, 2019 8:13 AM
> > > To: 'met_help at ucar.edu' <met_help at ucar.edu>
> > > Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > > docker-container in Singularity (UNCLASSIFIED)
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Thanks for responding while at the mtg and for following up with
Mike.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Thursday, January 10, 2019 8:04 AM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > > docker-container in Singularity (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > I forwarded your question to Mike Kavulich, who's planning on
> > > working with Singularity. Listed below is my question and his
> > > response. We are both at AMS this week.
> > >
> > > ---------------------------------------
> > >
> > > Mike,
> > >
> > > I got a followup question from John Raby from the Army. He's
> > > wondering if we've made any progress in using Singularity
containers.
> > >
> > > If I remember correctly, once we make containers available via
> > > DockerHub, this is very easy. But until then, it's more
difficult.
> > > Do you happen to have a Singularity version of the MET container
he
> > could use for testing?
> > >
> > > ---------------------------------------
> > >
> > > John,
> > >
> > > I do not have a singularity container ready, I haven't re-
installed
> > > the necessary software since my mac had to be serviced. I can
get to
> > > it next week at the earliest.
> > >
> > > Mike
> > >
> > > ---------------------------------------
> > >
> > >
> > > On Mon, Jan 7, 2019 at 3:01 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=87
> > > > 728 >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > John -
> > > >
> > > > Ref the email update below on the possibility of a Singularity
> > > > container for MET. Has there been any progress on this?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT
> > > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > > Sent: Wednesday, November 14, 2018 4:40 PM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > <john.w.raby2.civ at mail.mil>
> > > > Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > > > docker-container in Singularity (UNCLASSIFIED)
> > > >
> > > > All active links contained in this email were disabled.
Please
> > > > verify the identity of the sender, and confirm the
authenticity of
> > > > all links contained within the message prior to copying and
> > > > pasting the address to a Web browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > John,
> > > >
> > > > I forwarded your question about Singularity to other folks in
the
> > > > DTC and am forwarding their response.
> > > >
> > > > Specifically, it looks like once we get the MET container
posted
> > > > to DockerHub than importing it into Singularity becomes a one-
liner.
> > > > Seems like a good motivation to get that done!
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > ---------- Forwarded message ---------
> > > > From: Michael Kavulich <kavulich at ucar.edu>
> > > > Date: Wed, Nov 14, 2018 at 12:16 PM
> > > > Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container in
> > > > Singularity
> > > > (UNCLASSIFIED)
> > > > To: John Halley Gotway <johnhg at ucar.edu>
> > > > Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
> > > > <harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
> > > >
> > > >
> > > > John,
> > > >
> > > > I have only tested singularity with standard images from
dockerhub.
> > > > This process is relatively easy:
> > > > Caution-
> > > > Caution-Caution-Caution-https://www.sylabs.io/guides/2.5/user-
guid
> > > > e/singular
> > > > ity_and_do
> > > > cker.html?highlight=docker
> > > >
> > > > Converting images locally is a bit more involved, and I have
not
> > > > tried this
> > > > myself:
> > > > Caution-
> > > > Caution-Caution-Caution-
https://groups.google.com/a/lbl.gov/forum/
> > > > #!topic/si
> > > > ngularity/
> > > > ZXOaNaM_6MY
> > > >
> > > > This all requires that you are on a machine with both Docker
and
> > > > Singularity installed, which is a big limitation.
> > > >
> > > > The easiest thing for users would be if we did this conversion
> > > > ourselves and provided both Docker and Singularity images.
> > > > Provided I have hours left after the prep for the tutorial is
> > > > completed this is something I can look into.
> > > >
> > > > Michael Kavulich, Jr.
> > > > Associate Scientist, National Center for Atmospheric Research
> > > > (NCAR) Joint Numerical Testbed (JNT), Research Applications
> > > > Laboratory
> > > > (RAL)
> > > > * kavulich at ucar.edu <kavulich at ucar.edu>*
> > > >
> > > >
> > > > On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway
> > > > <johnhg at ucar.edu>
> > > wrote:
> > > >
> > > > > Mike,
> > > > >
> > > > > John Raby is a MET user from the Army Research Lab who is
> > > > > interested in running MET through a container. We currently
> > > > > have it in a Docker container, but apparently Singularity is
now
> > > > > available on their super-computer, Centennial, for testing.
> > > > >
> > > > > He's written MET-Help asking how to convert the MET Docker
> > > > > container to Singularity for testing on Centennial.
> > > > >
> > > > > I have never run Singularity but know that you have. What
> > > > > should I
> > do?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > ---------- Forwarded message ---------
> > > > > From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> > > > > Date: Fri, Nov 9, 2018 at 9:07 AM
> > > > > Subject: [rt.rap.ucar.edu #87728] MET docker-container in
> > > > > Singularity
> > > > > (UNCLASSIFIED)
> > > > > To:
> > > > >
> > > > >
> > > > >
> > > > > Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > > Queue: met_help
> > > > > Subject: MET docker-container in Singularity
(UNCLASSIFIED)
> > > > > Owner: Nobody
> > > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > Status: new
> > > > > Ticket <Caution-Caution-Caution-Caution-url:
> > > > > Caution-Caution-Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > > html?id=87
> > > > > 728 >
> > > > >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > I received the email below from the developer who is
> > > > > implementing Singularity on the HPC in response to my
question
> > > > > about running an already-built docker-container using
> > > > > Singularity Runtime. They are asking users to start test
runs
> > > > > with their already-built Singularity
> > > > containers.
> > > > > From his guidance below, can the MET docker-container be
made to
> > > > > run in Singularity Runtime?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > > Mr. John W. Raby
> > > > > U.S. Army Research Laboratory
> > > > > White Sands Missile Range, NM 88002
> > > > > (575) 678-2004 DSN 258-2004
> > > > > FAX (575) 678-1230 DSN 258-1230
> > > > > Email: john.w.raby2.civ at mail.mil
> > > > >
> > > > > -----Original Message-----
> > > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > > Sent: Friday, November 9, 2018 8:35 AM
> > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > <john.w.raby2.civ at mail.mil>
> > > > > Subject: RE: Singularity Runtime installed on Centennial
> > > > > (UNCLASSIFIED)
> > > > >
> > > > >
> > > > > You would need to convert the docker image to a singularity
image.
> > > > > From there you might have to change some permissions or
> > > > > re-configure to use non-root directories. It depends on how
> > > > > tightly coupled the application is to root inside the image.
You
> > > > > will need root on a machine to convert the image to
Singularity.
> > > > >
> > > > > Thanks,
> > > > > Matt
> > > > >
> > > > > ________________________________________
> > > > > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > Sent: Thursday, November 08, 2018 5:24 PM
> > > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > > Subject: RE: Singularity Runtime installed on Centennial
> > > > > (UNCLASSIFIED)
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > Thanks for all your work on implementing Singularity and for
the
> > > > > notification, Matt. I most certainly will send this out to
> > > > > others I know who are interested. I know of an already-built
> > > > > docker-container which I'm interested in. Does this fall
into
> > > > > the category of being suitable for testing on Centennial?
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > > Sent: Thursday, November 8, 2018 3:18 PM
> > > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > > <matthew.r.dwyer7.ctr at mail.mil
> > > > > >
> > > > > Subject: Singularity Runtime installed on Centennial
> > > > >
> > > > > Good News!
> > > > >
> > > > >
> > > > > Containers are installed on Centennial (for early access).
This
> > > > > is the first production machine container installation at
the ARL
> DSRC.
> > > > >
> > > > > Singularity runtime is installed on Centennial which means
you
> > > > > still cannot build your containers on ARL DSRC resources. If
you
> > > > > have already-built singularity containers and would like to
run
> > > > > them please go ahead and test them out now. The more
> > > > > bugs/misconfigurations we find now will provide a smoother
> > > > > user-experience with container workloads. Let me know of
> > > > > successes and failures and please send to
> > > > others who are interested.
> > > > >
> > > > >
> > > > > Thank You,
> > > > > Matt Dwyer
> > > > > 4102786819
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > >
> > > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET docker-container in Singularity (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Mon Mar 18 12:30:17 2019
CLASSIFICATION: UNCLASSIFIED
John -
I think I was trying a different link which was present in the older
emails, but was not the intended one. I used your suggestions for
related content and successfully found the site you originally
provided (I think). See the following URL:
https://www.sylabs.io/guides/3.0/user-guide/build_a_container.html
At the above site I see instructions and the command I think you were
referring to as follows:
Downloading an existing container from Docker Hub
You can use build to download layers from Docker Hub and assemble them
into Singularity containers.
$ sudo singularity build lolcow.sif docker://godlovedc/lolcow
Is this the info you were referring to?
Thanks.
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, March 18, 2019 11:54 AM
To: Raby, John W CIV USARMY FUTURES COMMAND (USA)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
docker-container in Singularity (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Very odd. I just googled "singularity and docker". And the first hit
is:
Caution-https://singularity.lbl.gov/docs-docker
At the top of this page it redirects me to a new URL here:
Caution-https://www.sylabs.io/docs/
Hopefully you can track down the content on one of these links.
John
On Mon, Mar 18, 2019 at 8:06 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I just tried access the link from my windows laptop using IE and
> firefox and get the same issue. IE reports can't access this page.
> Firefox just times out.
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Friday, March 15, 2019 9:15 AM
> To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
> john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> docker-container in Singularity (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Yes, that's the idea. As long as the image lives on DockerHub, you
> can run a single command to build a singularity container from it.
I
> have never actually done so myself since I don't have singularity
> installed on my laptop, and it's not yet available on our HPC's.
>
> Just take a look at the link I sent for an example of doing so.
>
> Thanks,
> John
>
> On Fri, Mar 15, 2019 at 6:45 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > 728 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > Thanks for the update, John. I'm at home today, but I check email
> > once in a while.
> >
> > So I understand this correctly, the version of MET in DockerHub is
> > capable of being run in a singularity container?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Thursday, March 14, 2019 4:40 PM
> > To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
> > john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > docker-container in Singularity (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > Hi John,
> >
> > I wanted to let you know that I just posted the docker image for
> > met-8.0 up to DockerHub. So retrieving it is now very easy:
> >
> > docker pull dtcenter/met:8.0
> >
> > In addition, once it's on DockerHub, I'm told that running it
> > through a singularity container is pretty straight-forward.
> > However, I haven't actually done so myself:
> >
> > Caution-
> > Caution-Caution-https://www.sylabs.io/guides/3.0/user-
guide/build_a_container.
> > html#downloading-an-existing-container-from-docker-hub
> >
> > Thanks,
> > John
> >
> > On Thu, Mar 7, 2019 at 8:18 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=87
> > > 728 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > Has there been any progress with developing a Singularity
version
> > > of the MET container?
> > >
> > > R/
> > > John
> > >
> > > Mr. John W. Raby
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > (575) 678-2004 DSN 258-2004
> > > FAX (575) 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > > -----Original Message-----
> > > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Sent: Thursday, January 10, 2019 8:13 AM
> > > To: 'met_help at ucar.edu' <met_help at ucar.edu>
> > > Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > > docker-container in Singularity (UNCLASSIFIED)
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Thanks for responding while at the mtg and for following up with
Mike.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Thursday, January 10, 2019 8:04 AM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > > docker-container in Singularity (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
> > > pasting the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > I forwarded your question to Mike Kavulich, who's planning on
> > > working with Singularity. Listed below is my question and his
> > > response. We are both at AMS this week.
> > >
> > > ---------------------------------------
> > >
> > > Mike,
> > >
> > > I got a followup question from John Raby from the Army. He's
> > > wondering if we've made any progress in using Singularity
containers.
> > >
> > > If I remember correctly, once we make containers available via
> > > DockerHub, this is very easy. But until then, it's more
difficult.
> > > Do you happen to have a Singularity version of the MET container
> > > he
> > could use for testing?
> > >
> > > ---------------------------------------
> > >
> > > John,
> > >
> > > I do not have a singularity container ready, I haven't
> > > re-installed the necessary software since my mac had to be
> > > serviced. I can get to it next week at the earliest.
> > >
> > > Mike
> > >
> > > ---------------------------------------
> > >
> > >
> > > On Mon, Jan 7, 2019 at 3:01 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=87
> > > > 728 >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > John -
> > > >
> > > > Ref the email update below on the possibility of a Singularity
> > > > container for MET. Has there been any progress on this?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT
> > > > [Caution-Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > > Sent: Wednesday, November 14, 2018 4:40 PM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > <john.w.raby2.civ at mail.mil>
> > > > Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > > > docker-container in Singularity (UNCLASSIFIED)
> > > >
> > > > All active links contained in this email were disabled.
Please
> > > > verify the identity of the sender, and confirm the
authenticity
> > > > of all links contained within the message prior to copying and
> > > > pasting the address to a Web browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > John,
> > > >
> > > > I forwarded your question about Singularity to other folks in
> > > > the DTC and am forwarding their response.
> > > >
> > > > Specifically, it looks like once we get the MET container
posted
> > > > to DockerHub than importing it into Singularity becomes a one-
liner.
> > > > Seems like a good motivation to get that done!
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > ---------- Forwarded message ---------
> > > > From: Michael Kavulich <kavulich at ucar.edu>
> > > > Date: Wed, Nov 14, 2018 at 12:16 PM
> > > > Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container in
> > > > Singularity
> > > > (UNCLASSIFIED)
> > > > To: John Halley Gotway <johnhg at ucar.edu>
> > > > Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
> > > > <harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
> > > >
> > > >
> > > > John,
> > > >
> > > > I have only tested singularity with standard images from
dockerhub.
> > > > This process is relatively easy:
> > > > Caution-
> > > > Caution-Caution-Caution-Caution-
https://www.sylabs.io/guides/2.5
> > > > /user-guid
> > > > e/singular
> > > > ity_and_do
> > > > cker.html?highlight=docker
> > > >
> > > > Converting images locally is a bit more involved, and I have
not
> > > > tried this
> > > > myself:
> > > > Caution-
> > > > Caution-Caution-Caution-Caution-
https://groups.google.com/a/lbl.
> > > > gov/forum/
> > > > #!topic/si
> > > > ngularity/
> > > > ZXOaNaM_6MY
> > > >
> > > > This all requires that you are on a machine with both Docker
and
> > > > Singularity installed, which is a big limitation.
> > > >
> > > > The easiest thing for users would be if we did this conversion
> > > > ourselves and provided both Docker and Singularity images.
> > > > Provided I have hours left after the prep for the tutorial is
> > > > completed this is something I can look into.
> > > >
> > > > Michael Kavulich, Jr.
> > > > Associate Scientist, National Center for Atmospheric Research
> > > > (NCAR) Joint Numerical Testbed (JNT), Research Applications
> > > > Laboratory
> > > > (RAL)
> > > > * kavulich at ucar.edu <kavulich at ucar.edu>*
> > > >
> > > >
> > > > On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway
> > > > <johnhg at ucar.edu>
> > > wrote:
> > > >
> > > > > Mike,
> > > > >
> > > > > John Raby is a MET user from the Army Research Lab who is
> > > > > interested in running MET through a container. We currently
> > > > > have it in a Docker container, but apparently Singularity is
> > > > > now available on their super-computer, Centennial, for
testing.
> > > > >
> > > > > He's written MET-Help asking how to convert the MET Docker
> > > > > container to Singularity for testing on Centennial.
> > > > >
> > > > > I have never run Singularity but know that you have. What
> > > > > should I
> > do?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > > ---------- Forwarded message ---------
> > > > > From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> > > > > Date: Fri, Nov 9, 2018 at 9:07 AM
> > > > > Subject: [rt.rap.ucar.edu #87728] MET docker-container in
> > > > > Singularity
> > > > > (UNCLASSIFIED)
> > > > > To:
> > > > >
> > > > >
> > > > >
> > > > > Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > > Queue: met_help
> > > > > Subject: MET docker-container in Singularity
(UNCLASSIFIED)
> > > > > Owner: Nobody
> > > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > Status: new
> > > > > Ticket <Caution-Caution-Caution-Caution-Caution-url:
> > > > > Caution-Caution-Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > > html?id=87
> > > > > 728 >
> > > > >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > I received the email below from the developer who is
> > > > > implementing Singularity on the HPC in response to my
question
> > > > > about running an already-built docker-container using
> > > > > Singularity Runtime. They are asking users to start test
runs
> > > > > with their already-built Singularity
> > > > containers.
> > > > > From his guidance below, can the MET docker-container be
made
> > > > > to run in Singularity Runtime?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > > Mr. John W. Raby
> > > > > U.S. Army Research Laboratory
> > > > > White Sands Missile Range, NM 88002
> > > > > (575) 678-2004 DSN 258-2004
> > > > > FAX (575) 678-1230 DSN 258-1230
> > > > > Email: john.w.raby2.civ at mail.mil
> > > > >
> > > > > -----Original Message-----
> > > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > > Sent: Friday, November 9, 2018 8:35 AM
> > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > <john.w.raby2.civ at mail.mil>
> > > > > Subject: RE: Singularity Runtime installed on Centennial
> > > > > (UNCLASSIFIED)
> > > > >
> > > > >
> > > > > You would need to convert the docker image to a singularity
image.
> > > > > From there you might have to change some permissions or
> > > > > re-configure to use non-root directories. It depends on how
> > > > > tightly coupled the application is to root inside the image.
> > > > > You will need root on a machine to convert the image to
Singularity.
> > > > >
> > > > > Thanks,
> > > > > Matt
> > > > >
> > > > > ________________________________________
> > > > > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > Sent: Thursday, November 08, 2018 5:24 PM
> > > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > > Subject: RE: Singularity Runtime installed on Centennial
> > > > > (UNCLASSIFIED)
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > Thanks for all your work on implementing Singularity and for
> > > > > the notification, Matt. I most certainly will send this out
to
> > > > > others I know who are interested. I know of an already-built
> > > > > docker-container which I'm interested in. Does this fall
into
> > > > > the category of being suitable for testing on Centennial?
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > > Sent: Thursday, November 8, 2018 3:18 PM
> > > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > > <matthew.r.dwyer7.ctr at mail.mil
> > > > > >
> > > > > Subject: Singularity Runtime installed on Centennial
> > > > >
> > > > > Good News!
> > > > >
> > > > >
> > > > > Containers are installed on Centennial (for early access).
> > > > > This is the first production machine container installation
at
> > > > > the ARL
> DSRC.
> > > > >
> > > > > Singularity runtime is installed on Centennial which means
you
> > > > > still cannot build your containers on ARL DSRC resources. If
> > > > > you have already-built singularity containers and would like
> > > > > to run them please go ahead and test them out now. The more
> > > > > bugs/misconfigurations we find now will provide a smoother
> > > > > user-experience with container workloads. Let me know of
> > > > > successes and failures and please send to
> > > > others who are interested.
> > > > >
> > > > >
> > > > > Thank You,
> > > > > Matt Dwyer
> > > > > 4102786819
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > >
> > > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET docker-container in Singularity (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Mar 22 10:27:16 2019
John,
Yes, that's right. That's what I was referring to. And we're happy
to
answer questions you or your sys-admin may have about Singularity.
However, I have not actually run it myself. So I'll likely refer you
to
Mike Kavulich, a colleague of mine who will be trying out Singularity
containers later this year.
I'll go ahead and resolve this ticket now. But when more questions
arise,
please just send a new email to met_help at ucar.edu.
Thanks,
John
On Mon, Mar 18, 2019 at 12:30 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I think I was trying a different link which was present in the
older
> emails, but was not the intended one. I used your suggestions for
related
> content and successfully found the site you originally provided (I
think).
> See the following URL:
>
> https://www.sylabs.io/guides/3.0/user-guide/build_a_container.html
>
> At the above site I see instructions and the command I think you
were
> referring to as follows:
>
> Downloading an existing container from Docker Hub
>
> You can use build to download layers from Docker Hub and assemble
them
> into Singularity containers.
>
>
> $ sudo singularity build lolcow.sif docker://godlovedc/lolcow
>
> Is this the info you were referring to?
>
> Thanks.
>
>
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, March 18, 2019 11:54 AM
> To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
> john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> docker-container in Singularity (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Very odd. I just googled "singularity and docker". And the first
hit is:
> Caution-https://singularity.lbl.gov/docs-docker
>
> At the top of this page it redirects me to a new URL here:
> Caution-https://www.sylabs.io/docs/
>
> Hopefully you can track down the content on one of these links.
>
> John
>
> On Mon, Mar 18, 2019 at 8:06 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > I just tried access the link from my windows laptop using IE and
> > firefox and get the same issue. IE reports can't access this page.
> > Firefox just times out.
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Friday, March 15, 2019 9:15 AM
> > To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
> > john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > docker-container in Singularity (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Yes, that's the idea. As long as the image lives on DockerHub,
you
> > can run a single command to build a singularity container from it.
I
> > have never actually done so myself since I don't have singularity
> > installed on my laptop, and it's not yet available on our HPC's.
> >
> > Just take a look at the link I sent for an example of doing so.
> >
> > Thanks,
> > John
> >
> > On Fri, Mar 15, 2019 at 6:45 AM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
> > > 728 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > Thanks for the update, John. I'm at home today, but I check
email
> > > once in a while.
> > >
> > > So I understand this correctly, the version of MET in DockerHub
is
> > > capable of being run in a singularity container?
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Thursday, March 14, 2019 4:40 PM
> > > To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
> > > john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > > docker-container in Singularity (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > Hi John,
> > >
> > > I wanted to let you know that I just posted the docker image for
> > > met-8.0 up to DockerHub. So retrieving it is now very easy:
> > >
> > > docker pull dtcenter/met:8.0
> > >
> > > In addition, once it's on DockerHub, I'm told that running it
> > > through a singularity container is pretty straight-forward.
> > > However, I haven't actually done so myself:
> > >
> > > Caution-
> > > Caution-Caution-
> https://www.sylabs.io/guides/3.0/user-guide/build_a_container.
> > > html#downloading-an-existing-container-from-docker-hub
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Mar 7, 2019 at 8:18 AM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=87
> > > > 728 >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > Has there been any progress with developing a Singularity
version
> > > > of the MET container?
> > > >
> > > > R/
> > > > John
> > > >
> > > > Mr. John W. Raby
> > > > U.S. Army Research Laboratory
> > > > White Sands Missile Range, NM 88002
> > > > (575) 678-2004 DSN 258-2004
> > > > FAX (575) 678-1230 DSN 258-1230
> > > > Email: john.w.raby2.civ at mail.mil
> > > >
> > > > -----Original Message-----
> > > > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > Sent: Thursday, January 10, 2019 8:13 AM
> > > > To: 'met_help at ucar.edu' <met_help at ucar.edu>
> > > > Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728]
MET
> > > > docker-container in Singularity (UNCLASSIFIED)
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > > John -
> > > >
> > > > Thanks for responding while at the mtg and for following up
with
> Mike.
> > > >
> > > > R/
> > > > John
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT
> > > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > > Sent: Thursday, January 10, 2019 8:04 AM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > <john.w.raby2.civ at mail.mil>
> > > > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728]
MET
> > > > docker-container in Singularity (UNCLASSIFIED)
> > > >
> > > > All active links contained in this email were disabled.
Please
> > > > verify the identity of the sender, and confirm the
authenticity of
> > > > all links contained within the message prior to copying and
> > > > pasting the address to a Web browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > John,
> > > >
> > > > I forwarded your question to Mike Kavulich, who's planning on
> > > > working with Singularity. Listed below is my question and his
> > > > response. We are both at AMS this week.
> > > >
> > > > ---------------------------------------
> > > >
> > > > Mike,
> > > >
> > > > I got a followup question from John Raby from the Army. He's
> > > > wondering if we've made any progress in using Singularity
containers.
> > > >
> > > > If I remember correctly, once we make containers available via
> > > > DockerHub, this is very easy. But until then, it's more
difficult.
> > > > Do you happen to have a Singularity version of the MET
container
> > > > he
> > > could use for testing?
> > > >
> > > > ---------------------------------------
> > > >
> > > > John,
> > > >
> > > > I do not have a singularity container ready, I haven't
> > > > re-installed the necessary software since my mac had to be
> > > > serviced. I can get to it next week at the earliest.
> > > >
> > > > Mike
> > > >
> > > > ---------------------------------------
> > > >
> > > >
> > > > On Mon, Jan 7, 2019 at 3:01 PM Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <Caution-Caution-Caution-Caution-url:
> > > > > Caution-Caution-Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > > html?id=87
> > > > > 728 >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > > John -
> > > > >
> > > > > Ref the email update below on the possibility of a
Singularity
> > > > > container for MET. Has there been any progress on this?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > > -----Original Message-----
> > > > > From: John Halley Gotway via RT
> > > > > [Caution-Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > > > Sent: Wednesday, November 14, 2018 4:40 PM
> > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > <john.w.raby2.civ at mail.mil>
> > > > > Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
> > > > > docker-container in Singularity (UNCLASSIFIED)
> > > > >
> > > > > All active links contained in this email were disabled.
Please
> > > > > verify the identity of the sender, and confirm the
authenticity
> > > > > of all links contained within the message prior to copying
and
> > > > > pasting the address to a Web browser.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ----
> > > > >
> > > > > John,
> > > > >
> > > > > I forwarded your question about Singularity to other folks
in
> > > > > the DTC and am forwarding their response.
> > > > >
> > > > > Specifically, it looks like once we get the MET container
posted
> > > > > to DockerHub than importing it into Singularity becomes a
> one-liner.
> > > > > Seems like a good motivation to get that done!
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > ---------- Forwarded message ---------
> > > > > From: Michael Kavulich <kavulich at ucar.edu>
> > > > > Date: Wed, Nov 14, 2018 at 12:16 PM
> > > > > Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container
in
> > > > > Singularity
> > > > > (UNCLASSIFIED)
> > > > > To: John Halley Gotway <johnhg at ucar.edu>
> > > > > Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
> > > > > <harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
> > > > >
> > > > >
> > > > > John,
> > > > >
> > > > > I have only tested singularity with standard images from
dockerhub.
> > > > > This process is relatively easy:
> > > > > Caution-
> > > > > Caution-Caution-Caution-Caution-
https://www.sylabs.io/guides/2.5
> > > > > /user-guid
> > > > > e/singular
> > > > > ity_and_do
> > > > > cker.html?highlight=docker
> > > > >
> > > > > Converting images locally is a bit more involved, and I have
not
> > > > > tried this
> > > > > myself:
> > > > > Caution-
> > > > > Caution-Caution-Caution-Caution-
https://groups.google.com/a/lbl.
> > > > > gov/forum/
> > > > > #!topic/si
> > > > > ngularity/
> > > > > ZXOaNaM_6MY
> > > > >
> > > > > This all requires that you are on a machine with both Docker
and
> > > > > Singularity installed, which is a big limitation.
> > > > >
> > > > > The easiest thing for users would be if we did this
conversion
> > > > > ourselves and provided both Docker and Singularity images.
> > > > > Provided I have hours left after the prep for the tutorial
is
> > > > > completed this is something I can look into.
> > > > >
> > > > > Michael Kavulich, Jr.
> > > > > Associate Scientist, National Center for Atmospheric
Research
> > > > > (NCAR) Joint Numerical Testbed (JNT), Research Applications
> > > > > Laboratory
> > > > > (RAL)
> > > > > * kavulich at ucar.edu <kavulich at ucar.edu>*
> > > > >
> > > > >
> > > > > On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway
> > > > > <johnhg at ucar.edu>
> > > > wrote:
> > > > >
> > > > > > Mike,
> > > > > >
> > > > > > John Raby is a MET user from the Army Research Lab who is
> > > > > > interested in running MET through a container. We
currently
> > > > > > have it in a Docker container, but apparently Singularity
is
> > > > > > now available on their super-computer, Centennial, for
testing.
> > > > > >
> > > > > > He's written MET-Help asking how to convert the MET Docker
> > > > > > container to Singularity for testing on Centennial.
> > > > > >
> > > > > > I have never run Singularity but know that you have. What
> > > > > > should I
> > > do?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > > ---------- Forwarded message ---------
> > > > > > From: Raby, John W USA CIV via RT <met_help at ucar.edu>
> > > > > > Date: Fri, Nov 9, 2018 at 9:07 AM
> > > > > > Subject: [rt.rap.ucar.edu #87728] MET docker-container in
> > > > > > Singularity
> > > > > > (UNCLASSIFIED)
> > > > > > To:
> > > > > >
> > > > > >
> > > > > >
> > > > > > Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
> > > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > > > Queue: met_help
> > > > > > Subject: MET docker-container in Singularity
(UNCLASSIFIED)
> > > > > > Owner: Nobody
> > > > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > > Status: new
> > > > > > Ticket <Caution-Caution-Caution-Caution-Caution-url:
> > > > > > Caution-Caution-Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > > > html?id=87
> > > > > > 728 >
> > > > > >
> > > > > >
> > > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > >
> > > > > > I received the email below from the developer who is
> > > > > > implementing Singularity on the HPC in response to my
question
> > > > > > about running an already-built docker-container using
> > > > > > Singularity Runtime. They are asking users to start test
runs
> > > > > > with their already-built Singularity
> > > > > containers.
> > > > > > From his guidance below, can the MET docker-container be
made
> > > > > > to run in Singularity Runtime?
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > R/
> > > > > > John
> > > > > >
> > > > > > Mr. John W. Raby
> > > > > > U.S. Army Research Laboratory
> > > > > > White Sands Missile Range, NM 88002
> > > > > > (575) 678-2004 DSN 258-2004
> > > > > > FAX (575) 678-1230 DSN 258-1230
> > > > > > Email: john.w.raby2.civ at mail.mil
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > > > Sent: Friday, November 9, 2018 8:35 AM
> > > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > > <john.w.raby2.civ at mail.mil>
> > > > > > Subject: RE: Singularity Runtime installed on Centennial
> > > > > > (UNCLASSIFIED)
> > > > > >
> > > > > >
> > > > > > You would need to convert the docker image to a
singularity
> image.
> > > > > > From there you might have to change some permissions or
> > > > > > re-configure to use non-root directories. It depends on
how
> > > > > > tightly coupled the application is to root inside the
image.
> > > > > > You will need root on a machine to convert the image to
> Singularity.
> > > > > >
> > > > > > Thanks,
> > > > > > Matt
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > > Sent: Thursday, November 08, 2018 5:24 PM
> > > > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > > > Subject: RE: Singularity Runtime installed on Centennial
> > > > > > (UNCLASSIFIED)
> > > > > >
> > > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > >
> > > > > > Thanks for all your work on implementing Singularity and
for
> > > > > > the notification, Matt. I most certainly will send this
out to
> > > > > > others I know who are interested. I know of an already-
built
> > > > > > docker-container which I'm interested in. Does this fall
into
> > > > > > the category of being suitable for testing on Centennial?
> > > > > >
> > > > > > R/
> > > > > > John
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > > > Sent: Thursday, November 8, 2018 3:18 PM
> > > > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
> > > > > > <matthew.r.dwyer7.ctr at mail.mil
> > > > > > >
> > > > > > Subject: Singularity Runtime installed on Centennial
> > > > > >
> > > > > > Good News!
> > > > > >
> > > > > >
> > > > > > Containers are installed on Centennial (for early access).
> > > > > > This is the first production machine container
installation at
> > > > > > the ARL
> > DSRC.
> > > > > >
> > > > > > Singularity runtime is installed on Centennial which means
you
> > > > > > still cannot build your containers on ARL DSRC resources.
If
> > > > > > you have already-built singularity containers and would
like
> > > > > > to run them please go ahead and test them out now. The
more
> > > > > > bugs/misconfigurations we find now will provide a smoother
> > > > > > user-experience with container workloads. Let me know of
> > > > > > successes and failures and please send to
> > > > > others who are interested.
> > > > > >
> > > > > >
> > > > > > Thank You,
> > > > > > Matt Dwyer
> > > > > > 4102786819
> > > > > >
> > > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > > CLASSIFICATION: UNCLASSIFIED
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > CLASSIFICATION: UNCLASSIFIED
> > > > >
> > > > >
> > > > >
> > > >
> > > > CLASSIFICATION: UNCLASSIFIED
> > > > CLASSIFICATION: UNCLASSIFIED
> > > >
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: MET docker-container in Singularity (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Apr 30 15:19:31 2019
Hi John,
I'm forwarding an email from Mike Kavulich about running a Docker
image
from DockerHub via Singularity. So instantiating the MET 8.0 image
from
DockerHub should be very easy.
Thanks,
John
Hi all, I've finally really dived in to the investigation of
singularity
containers and I've made a few observations. First: as I suspected,
running
singularity containers from dockerhub is almost trivially easy. From
an
environment with singularity installed, this was all I had to do to
pull
the UPP container as a singularity image:
sudo singularity pull docker://dtcenter/upp
Using sudo is probably not required, I think it's just due to how I
set
everything up permissions wise.
Setting up singularity in the first place is way harder, but we can
leverage the resources of JEDI's efforts to make that fairly simple as
well:
https://wiki.ucar.edu/display/JEDI/Install+Singularity+on+Mac+OS+X
One thing that may present a roadblock: I'm not sure if there is a
straightforward way to create data containers. This can be worked
around by
"binding" (essentially mounting) local directories into each instance,
but
it would require some reworking of the tutorial. I'm not 100% sure
about
this, and the singularity developers may be working on a more "docker-
like"
solution per this page:
https://www.hpcwire.com/2017/11/01/sc17-singularity-preps-version-3-0-
nears-1m-containers-served-daily/
On Fri, Mar 22, 2019 at 10:26 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> John,
>
> Yes, that's right. That's what I was referring to. And we're happy
to
> answer questions you or your sys-admin may have about Singularity.
> However, I have not actually run it myself. So I'll likely refer
you to
> Mike Kavulich, a colleague of mine who will be trying out
Singularity
> containers later this year.
>
> I'll go ahead and resolve this ticket now. But when more questions
arise,
> please just send a new email to met_help at ucar.edu.
>
> Thanks,
> John
>
> On Mon, Mar 18, 2019 at 12:30 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>>
>> CLASSIFICATION: UNCLASSIFIED
>>
>> John -
>>
>> I think I was trying a different link which was present in the
older
>> emails, but was not the intended one. I used your suggestions for
related
>> content and successfully found the site you originally provided (I
think).
>> See the following URL:
>>
>> https://www.sylabs.io/guides/3.0/user-guide/build_a_container.html
>>
>> At the above site I see instructions and the command I think you
were
>> referring to as follows:
>>
>> Downloading an existing container from Docker Hub
>>
>> You can use build to download layers from Docker Hub and assemble
them
>> into Singularity containers.
>>
>>
>> $ sudo singularity build lolcow.sif docker://godlovedc/lolcow
>>
>> Is this the info you were referring to?
>>
>> Thanks.
>>
>>
>> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
>> Sent: Monday, March 18, 2019 11:54 AM
>> To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
>> john.w.raby2.civ at mail.mil>
>> Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
>> docker-container in Singularity (UNCLASSIFIED)
>>
>> All active links contained in this email were disabled. Please
verify
>> the identity of the sender, and confirm the authenticity of all
links
>> contained within the message prior to copying and pasting the
address to a
>> Web browser.
>>
>>
>>
>>
>> ----
>>
>> John,
>>
>> Very odd. I just googled "singularity and docker". And the first
hit is:
>> Caution-https://singularity.lbl.gov/docs-docker
>>
>> At the top of this page it redirects me to a new URL here:
>> Caution-https://www.sylabs.io/docs/
>>
>> Hopefully you can track down the content on one of these links.
>>
>> John
>>
>> On Mon, Mar 18, 2019 at 8:06 AM Raby, John W USA CIV via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <Caution-url:
>> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>> >
>> > CLASSIFICATION: UNCLASSIFIED
>> >
>> > John -
>> >
>> > I just tried access the link from my windows laptop using IE and
>> > firefox and get the same issue. IE reports can't access this
page.
>> > Firefox just times out.
>> >
>> > R/
>> > John
>> >
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT [Caution-
mailto:met_help at ucar.edu]
>> > Sent: Friday, March 15, 2019 9:15 AM
>> > To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
>> > john.w.raby2.civ at mail.mil>
>> > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
>> > docker-container in Singularity (UNCLASSIFIED)
>> >
>> > All active links contained in this email were disabled. Please
verify
>> > the identity of the sender, and confirm the authenticity of all
links
>> > contained within the message prior to copying and pasting the
address
>> > to a Web browser.
>> >
>> >
>> >
>> >
>> > ----
>> >
>> > John,
>> >
>> > Yes, that's the idea. As long as the image lives on DockerHub,
you
>> > can run a single command to build a singularity container from
it. I
>> > have never actually done so myself since I don't have singularity
>> > installed on my laptop, and it's not yet available on our HPC's.
>> >
>> > Just take a look at the link I sent for an example of doing so.
>> >
>> > Thanks,
>> > John
>> >
>> > On Fri, Mar 15, 2019 at 6:45 AM Raby, John W USA CIV via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <Caution-Caution-url:
>> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
>> > > 728 >
>> > >
>> > > CLASSIFICATION: UNCLASSIFIED
>> > >
>> > > Thanks for the update, John. I'm at home today, but I check
email
>> > > once in a while.
>> > >
>> > > So I understand this correctly, the version of MET in DockerHub
is
>> > > capable of being run in a singularity container?
>> > >
>> > > R/
>> > > John
>> > >
>> > > -----Original Message-----
>> > > From: John Halley Gotway via RT
>> > > [Caution-Caution-mailto:met_help at ucar.edu]
>> > > Sent: Thursday, March 14, 2019 4:40 PM
>> > > To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
>> > > john.w.raby2.civ at mail.mil>
>> > > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
>> > > docker-container in Singularity (UNCLASSIFIED)
>> > >
>> > > All active links contained in this email were disabled. Please
>> > > verify the identity of the sender, and confirm the authenticity
of
>> > > all links contained within the message prior to copying and
pasting
>> > > the address to a Web browser.
>> > >
>> > >
>> > >
>> > >
>> > > ----
>> > >
>> > > Hi John,
>> > >
>> > > I wanted to let you know that I just posted the docker image
for
>> > > met-8.0 up to DockerHub. So retrieving it is now very easy:
>> > >
>> > > docker pull dtcenter/met:8.0
>> > >
>> > > In addition, once it's on DockerHub, I'm told that running it
>> > > through a singularity container is pretty straight-forward.
>> > > However, I haven't actually done so myself:
>> > >
>> > > Caution-
>> > > Caution-Caution-
>> https://www.sylabs.io/guides/3.0/user-guide/build_a_container.
>> > > html#downloading-an-existing-container-from-docker-hub
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Thu, Mar 7, 2019 at 8:18 AM Raby, John W USA CIV via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <Caution-Caution-Caution-url:
>> > > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
>> > > > html?id=87
>> > > > 728 >
>> > > >
>> > > > CLASSIFICATION: UNCLASSIFIED
>> > > >
>> > > > Has there been any progress with developing a Singularity
version
>> > > > of the MET container?
>> > > >
>> > > > R/
>> > > > John
>> > > >
>> > > > Mr. John W. Raby
>> > > > U.S. Army Research Laboratory
>> > > > White Sands Missile Range, NM 88002
>> > > > (575) 678-2004 DSN 258-2004
>> > > > FAX (575) 678-1230 DSN 258-1230
>> > > > Email: john.w.raby2.civ at mail.mil
>> > > >
>> > > > -----Original Message-----
>> > > > From: Raby, John W CIV USARMY RDECOM ARL (US)
>> > > > Sent: Thursday, January 10, 2019 8:13 AM
>> > > > To: 'met_help at ucar.edu' <met_help at ucar.edu>
>> > > > Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728]
MET
>> > > > docker-container in Singularity (UNCLASSIFIED)
>> > > >
>> > > > CLASSIFICATION: UNCLASSIFIED
>> > > >
>> > > > John -
>> > > >
>> > > > Thanks for responding while at the mtg and for following up
with
>> Mike.
>> > > >
>> > > > R/
>> > > > John
>> > > >
>> > > > -----Original Message-----
>> > > > From: John Halley Gotway via RT
>> > > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
>> > > > Sent: Thursday, January 10, 2019 8:04 AM
>> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
>> > > > <john.w.raby2.civ at mail.mil>
>> > > > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728]
MET
>> > > > docker-container in Singularity (UNCLASSIFIED)
>> > > >
>> > > > All active links contained in this email were disabled.
Please
>> > > > verify the identity of the sender, and confirm the
authenticity of
>> > > > all links contained within the message prior to copying and
>> > > > pasting the address to a Web browser.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > ----
>> > > >
>> > > > John,
>> > > >
>> > > > I forwarded your question to Mike Kavulich, who's planning on
>> > > > working with Singularity. Listed below is my question and
his
>> > > > response. We are both at AMS this week.
>> > > >
>> > > > ---------------------------------------
>> > > >
>> > > > Mike,
>> > > >
>> > > > I got a followup question from John Raby from the Army. He's
>> > > > wondering if we've made any progress in using Singularity
>> containers.
>> > > >
>> > > > If I remember correctly, once we make containers available
via
>> > > > DockerHub, this is very easy. But until then, it's more
difficult.
>> > > > Do you happen to have a Singularity version of the MET
container
>> > > > he
>> > > could use for testing?
>> > > >
>> > > > ---------------------------------------
>> > > >
>> > > > John,
>> > > >
>> > > > I do not have a singularity container ready, I haven't
>> > > > re-installed the necessary software since my mac had to be
>> > > > serviced. I can get to it next week at the earliest.
>> > > >
>> > > > Mike
>> > > >
>> > > > ---------------------------------------
>> > > >
>> > > >
>> > > > On Mon, Jan 7, 2019 at 3:01 PM Raby, John W USA CIV via RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > <Caution-Caution-Caution-Caution-url:
>> > > > > Caution-Caution-Caution-Caution-
>> https://rt.rap.ucar.edu/rt/Ticket/Display.
>> > > > > html?id=87
>> > > > > 728 >
>> > > > >
>> > > > > CLASSIFICATION: UNCLASSIFIED
>> > > > >
>> > > > > John -
>> > > > >
>> > > > > Ref the email update below on the possibility of a
Singularity
>> > > > > container for MET. Has there been any progress on this?
>> > > > >
>> > > > > Thanks.
>> > > > >
>> > > > > R/
>> > > > > John
>> > > > >
>> > > > > -----Original Message-----
>> > > > > From: John Halley Gotway via RT
>> > > > > [Caution-Caution-Caution-Caution-mailto:met_help at ucar.edu]
>> > > > > Sent: Wednesday, November 14, 2018 4:40 PM
>> > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
>> > > > > <john.w.raby2.civ at mail.mil>
>> > > > > Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
>> > > > > docker-container in Singularity (UNCLASSIFIED)
>> > > > >
>> > > > > All active links contained in this email were disabled.
Please
>> > > > > verify the identity of the sender, and confirm the
authenticity
>> > > > > of all links contained within the message prior to copying
and
>> > > > > pasting the address to a Web browser.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > ----
>> > > > >
>> > > > > John,
>> > > > >
>> > > > > I forwarded your question about Singularity to other folks
in
>> > > > > the DTC and am forwarding their response.
>> > > > >
>> > > > > Specifically, it looks like once we get the MET container
posted
>> > > > > to DockerHub than importing it into Singularity becomes a
>> one-liner.
>> > > > > Seems like a good motivation to get that done!
>> > > > >
>> > > > > Thanks,
>> > > > > John
>> > > > >
>> > > > > ---------- Forwarded message ---------
>> > > > > From: Michael Kavulich <kavulich at ucar.edu>
>> > > > > Date: Wed, Nov 14, 2018 at 12:16 PM
>> > > > > Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container
in
>> > > > > Singularity
>> > > > > (UNCLASSIFIED)
>> > > > > To: John Halley Gotway <johnhg at ucar.edu>
>> > > > > Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
>> > > > > <harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
>> > > > >
>> > > > >
>> > > > > John,
>> > > > >
>> > > > > I have only tested singularity with standard images from
>> dockerhub.
>> > > > > This process is relatively easy:
>> > > > > Caution-
>> > > > > Caution-Caution-Caution-Caution-
https://www.sylabs.io/guides/2.5
>> > > > > /user-guid
>> > > > > e/singular
>> > > > > ity_and_do
>> > > > > cker.html?highlight=docker
>> > > > >
>> > > > > Converting images locally is a bit more involved, and I
have not
>> > > > > tried this
>> > > > > myself:
>> > > > > Caution-
>> > > > > Caution-Caution-Caution-Caution-
https://groups.google.com/a/lbl.
>> > > > > gov/forum/
>> > > > > #!topic/si
>> > > > > ngularity/
>> > > > > ZXOaNaM_6MY
>> > > > >
>> > > > > This all requires that you are on a machine with both
Docker and
>> > > > > Singularity installed, which is a big limitation.
>> > > > >
>> > > > > The easiest thing for users would be if we did this
conversion
>> > > > > ourselves and provided both Docker and Singularity images.
>> > > > > Provided I have hours left after the prep for the tutorial
is
>> > > > > completed this is something I can look into.
>> > > > >
>> > > > > Michael Kavulich, Jr.
>> > > > > Associate Scientist, National Center for Atmospheric
Research
>> > > > > (NCAR) Joint Numerical Testbed (JNT), Research Applications
>> > > > > Laboratory
>> > > > > (RAL)
>> > > > > * kavulich at ucar.edu <kavulich at ucar.edu>*
>> > > > >
>> > > > >
>> > > > > On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway
>> > > > > <johnhg at ucar.edu>
>> > > > wrote:
>> > > > >
>> > > > > > Mike,
>> > > > > >
>> > > > > > John Raby is a MET user from the Army Research Lab who is
>> > > > > > interested in running MET through a container. We
currently
>> > > > > > have it in a Docker container, but apparently Singularity
is
>> > > > > > now available on their super-computer, Centennial, for
testing.
>> > > > > >
>> > > > > > He's written MET-Help asking how to convert the MET
Docker
>> > > > > > container to Singularity for testing on Centennial.
>> > > > > >
>> > > > > > I have never run Singularity but know that you have.
What
>> > > > > > should I
>> > > do?
>> > > > > >
>> > > > > > Thanks,
>> > > > > > John
>> > > > > >
>> > > > > >
>> > > > > > ---------- Forwarded message ---------
>> > > > > > From: Raby, John W USA CIV via RT <met_help at ucar.edu>
>> > > > > > Date: Fri, Nov 9, 2018 at 9:07 AM
>> > > > > > Subject: [rt.rap.ucar.edu #87728] MET docker-container in
>> > > > > > Singularity
>> > > > > > (UNCLASSIFIED)
>> > > > > > To:
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
>> > > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
>> > > > > > Queue: met_help
>> > > > > > Subject: MET docker-container in Singularity
(UNCLASSIFIED)
>> > > > > > Owner: Nobody
>> > > > > > Requestors: john.w.raby2.civ at mail.mil
>> > > > > > Status: new
>> > > > > > Ticket <Caution-Caution-Caution-Caution-Caution-url:
>> > > > > > Caution-Caution-Caution-Caution-
>> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.
>> > > > > > html?id=87
>> > > > > > 728 >
>> > > > > >
>> > > > > >
>> > > > > > CLASSIFICATION: UNCLASSIFIED
>> > > > > >
>> > > > > > I received the email below from the developer who is
>> > > > > > implementing Singularity on the HPC in response to my
question
>> > > > > > about running an already-built docker-container using
>> > > > > > Singularity Runtime. They are asking users to start test
runs
>> > > > > > with their already-built Singularity
>> > > > > containers.
>> > > > > > From his guidance below, can the MET docker-container be
made
>> > > > > > to run in Singularity Runtime?
>> > > > > >
>> > > > > > Thanks.
>> > > > > >
>> > > > > > R/
>> > > > > > John
>> > > > > >
>> > > > > > Mr. John W. Raby
>> > > > > > U.S. Army Research Laboratory
>> > > > > > White Sands Missile Range, NM 88002
>> > > > > > (575) 678-2004 DSN 258-2004
>> > > > > > FAX (575) 678-1230 DSN 258-1230
>> > > > > > Email: john.w.raby2.civ at mail.mil
>> > > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
>> > > > > > Sent: Friday, November 9, 2018 8:35 AM
>> > > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
>> > > > > > <john.w.raby2.civ at mail.mil>
>> > > > > > Subject: RE: Singularity Runtime installed on Centennial
>> > > > > > (UNCLASSIFIED)
>> > > > > >
>> > > > > >
>> > > > > > You would need to convert the docker image to a
singularity
>> image.
>> > > > > > From there you might have to change some permissions or
>> > > > > > re-configure to use non-root directories. It depends on
how
>> > > > > > tightly coupled the application is to root inside the
image.
>> > > > > > You will need root on a machine to convert the image to
>> Singularity.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Matt
>> > > > > >
>> > > > > > ________________________________________
>> > > > > > From: Raby, John W CIV USARMY RDECOM ARL (US)
>> > > > > > Sent: Thursday, November 08, 2018 5:24 PM
>> > > > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
>> > > > > > Subject: RE: Singularity Runtime installed on Centennial
>> > > > > > (UNCLASSIFIED)
>> > > > > >
>> > > > > > CLASSIFICATION: UNCLASSIFIED
>> > > > > >
>> > > > > > Thanks for all your work on implementing Singularity and
for
>> > > > > > the notification, Matt. I most certainly will send this
out to
>> > > > > > others I know who are interested. I know of an already-
built
>> > > > > > docker-container which I'm interested in. Does this fall
into
>> > > > > > the category of being suitable for testing on Centennial?
>> > > > > >
>> > > > > > R/
>> > > > > > John
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
>> > > > > > Sent: Thursday, November 8, 2018 3:18 PM
>> > > > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
>> > > > > > <matthew.r.dwyer7.ctr at mail.mil
>> > > > > > >
>> > > > > > Subject: Singularity Runtime installed on Centennial
>> > > > > >
>> > > > > > Good News!
>> > > > > >
>> > > > > >
>> > > > > > Containers are installed on Centennial (for early
access).
>> > > > > > This is the first production machine container
installation at
>> > > > > > the ARL
>> > DSRC.
>> > > > > >
>> > > > > > Singularity runtime is installed on Centennial which
means you
>> > > > > > still cannot build your containers on ARL DSRC resources.
If
>> > > > > > you have already-built singularity containers and would
like
>> > > > > > to run them please go ahead and test them out now. The
more
>> > > > > > bugs/misconfigurations we find now will provide a
smoother
>> > > > > > user-experience with container workloads. Let me know of
>> > > > > > successes and failures and please send to
>> > > > > others who are interested.
>> > > > > >
>> > > > > >
>> > > > > > Thank You,
>> > > > > > Matt Dwyer
>> > > > > > 4102786819
>> > > > > >
>> > > > > > CLASSIFICATION: UNCLASSIFIED
>> > > > > > CLASSIFICATION: UNCLASSIFIED
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > > CLASSIFICATION: UNCLASSIFIED
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > > CLASSIFICATION: UNCLASSIFIED
>> > > > CLASSIFICATION: UNCLASSIFIED
>> > > >
>> > > >
>> > > >
>> > >
>> > > CLASSIFICATION: UNCLASSIFIED
>> > >
>> > >
>> > >
>> >
>> > CLASSIFICATION: UNCLASSIFIED
>> >
>> >
>> >
>>
>> CLASSIFICATION: UNCLASSIFIED
>>
>>
>>
------------------------------------------------
Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET docker-container in Singularity (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Apr 30 15:34:31 2019
John -
Thanks for forwarding the info on this development on running a Docker
image from DockerHub via Singularity. I assume that a similar approach
could be used for MET 8.0. I will provide this info to my system admin
who has Singuiarity running on the ARL HPC and see what they can do
with it.Will let you know what I hear from them.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Tuesday, April 30, 2019 3:19 PM
To: Raby, John W CIV USARMY CCDC ARL (USA)
Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
docker-container in Singularity (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
Hi John,
I'm forwarding an email from Mike Kavulich about running a Docker
image
from DockerHub via Singularity. So instantiating the MET 8.0 image
from
DockerHub should be very easy.
Thanks,
John
Hi all, I've finally really dived in to the investigation of
singularity
containers and I've made a few observations. First: as I suspected,
running
singularity containers from dockerhub is almost trivially easy. From
an
environment with singularity installed, this was all I had to do to
pull
the UPP container as a singularity image:
sudo singularity pull docker://dtcenter/upp
Using sudo is probably not required, I think it's just due to how I
set
everything up permissions wise.
Setting up singularity in the first place is way harder, but we can
leverage the resources of JEDI's efforts to make that fairly simple as
well: Caution-
https://wiki.ucar.edu/display/JEDI/Install+Singularity+on+Mac+OS+X
One thing that may present a roadblock: I'm not sure if there is a
straightforward way to create data containers. This can be worked
around by
"binding" (essentially mounting) local directories into each instance,
but
it would require some reworking of the tutorial. I'm not 100% sure
about
this, and the singularity developers may be working on a more "docker-
like"
solution per this page:
Caution-https://www.hpcwire.com/2017/11/01/sc17-singularity-preps-
version-3-0-nears-1m-containers-served-daily/
On Fri, Mar 22, 2019 at 10:26 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> John,
>
> Yes, that's right. That's what I was referring to. And we're happy
to
> answer questions you or your sys-admin may have about Singularity.
> However, I have not actually run it myself. So I'll likely refer
you to
> Mike Kavulich, a colleague of mine who will be trying out
Singularity
> containers later this year.
>
> I'll go ahead and resolve this ticket now. But when more questions
arise,
> please just send a new email to met_help at ucar.edu.
>
> Thanks,
> John
>
> On Mon, Mar 18, 2019 at 12:30 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>>
>> CLASSIFICATION: UNCLASSIFIED
>>
>> John -
>>
>> I think I was trying a different link which was present in the
older
>> emails, but was not the intended one. I used your suggestions for
related
>> content and successfully found the site you originally provided (I
think).
>> See the following Caution-url:
>>
>> Caution-https://www.sylabs.io/guides/3.0/user-
guide/build_a_container.html
>>
>> At the above site I see instructions and the command I think you
were
>> referring to as follows:
>>
>> Downloading an existing container from Docker Hub
>>
>> You can use build to download layers from Docker Hub and assemble
them
>> into Singularity containers.
>>
>>
>> $ sudo singularity build lolcow.sif docker://godlovedc/lolcow
>>
>> Is this the info you were referring to?
>>
>> Thanks.
>>
>>
>> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
>> Sent: Monday, March 18, 2019 11:54 AM
>> To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
>> john.w.raby2.civ at mail.mil>
>> Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
>> docker-container in Singularity (UNCLASSIFIED)
>>
>> All active links contained in this email were disabled. Please
verify
>> the identity of the sender, and confirm the authenticity of all
links
>> contained within the message prior to copying and pasting the
address to a
>> Web browser.
>>
>>
>>
>>
>> ----
>>
>> John,
>>
>> Very odd. I just googled "singularity and docker". And the first
hit is:
>> Caution-Caution-https://singularity.lbl.gov/docs-docker
>>
>> At the top of this page it redirects me to a new URL here:
>> Caution-Caution-https://www.sylabs.io/docs/
>>
>> Hopefully you can track down the content on one of these links.
>>
>> John
>>
>> On Mon, Mar 18, 2019 at 8:06 AM Raby, John W USA CIV via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <Caution-Caution-url:
>> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87728 >
>> >
>> > CLASSIFICATION: UNCLASSIFIED
>> >
>> > John -
>> >
>> > I just tried access the link from my windows laptop using IE and
>> > firefox and get the same issue. IE reports can't access this
page.
>> > Firefox just times out.
>> >
>> > R/
>> > John
>> >
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT [Caution-Caution-
mailto:met_help at ucar.edu]
>> > Sent: Friday, March 15, 2019 9:15 AM
>> > To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
>> > john.w.raby2.civ at mail.mil>
>> > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
>> > docker-container in Singularity (UNCLASSIFIED)
>> >
>> > All active links contained in this email were disabled. Please
verify
>> > the identity of the sender, and confirm the authenticity of all
links
>> > contained within the message prior to copying and pasting the
address
>> > to a Web browser.
>> >
>> >
>> >
>> >
>> > ----
>> >
>> > John,
>> >
>> > Yes, that's the idea. As long as the image lives on DockerHub,
you
>> > can run a single command to build a singularity container from
it. I
>> > have never actually done so myself since I don't have singularity
>> > installed on my laptop, and it's not yet available on our HPC's.
>> >
>> > Just take a look at the link I sent for an example of doing so.
>> >
>> > Thanks,
>> > John
>> >
>> > On Fri, Mar 15, 2019 at 6:45 AM Raby, John W USA CIV via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <Caution-Caution-Caution-url:
>> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87
>> > > 728 >
>> > >
>> > > CLASSIFICATION: UNCLASSIFIED
>> > >
>> > > Thanks for the update, John. I'm at home today, but I check
email
>> > > once in a while.
>> > >
>> > > So I understand this correctly, the version of MET in DockerHub
is
>> > > capable of being run in a singularity container?
>> > >
>> > > R/
>> > > John
>> > >
>> > > -----Original Message-----
>> > > From: John Halley Gotway via RT
>> > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
>> > > Sent: Thursday, March 14, 2019 4:40 PM
>> > > To: Raby, John W CIV USARMY FUTURES COMMAND (USA) <
>> > > john.w.raby2.civ at mail.mil>
>> > > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
>> > > docker-container in Singularity (UNCLASSIFIED)
>> > >
>> > > All active links contained in this email were disabled. Please
>> > > verify the identity of the sender, and confirm the authenticity
of
>> > > all links contained within the message prior to copying and
pasting
>> > > the address to a Web browser.
>> > >
>> > >
>> > >
>> > >
>> > > ----
>> > >
>> > > Hi John,
>> > >
>> > > I wanted to let you know that I just posted the docker image
for
>> > > met-8.0 up to DockerHub. So retrieving it is now very easy:
>> > >
>> > > docker pull dtcenter/met:8.0
>> > >
>> > > In addition, once it's on DockerHub, I'm told that running it
>> > > through a singularity container is pretty straight-forward.
>> > > However, I haven't actually done so myself:
>> > >
>> > > Caution-
>> > > Caution-Caution-
>> Caution-https://www.sylabs.io/guides/3.0/user-
guide/build_a_container.
>> > > html#downloading-an-existing-container-from-docker-hub
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Thu, Mar 7, 2019 at 8:18 AM Raby, John W USA CIV via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <Caution-Caution-Caution-Caution-url:
>> > > > Caution-Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
>> > > > html?id=87
>> > > > 728 >
>> > > >
>> > > > CLASSIFICATION: UNCLASSIFIED
>> > > >
>> > > > Has there been any progress with developing a Singularity
version
>> > > > of the MET container?
>> > > >
>> > > > R/
>> > > > John
>> > > >
>> > > > Mr. John W. Raby
>> > > > U.S. Army Research Laboratory
>> > > > White Sands Missile Range, NM 88002
>> > > > (575) 678-2004 DSN 258-2004
>> > > > FAX (575) 678-1230 DSN 258-1230
>> > > > Email: john.w.raby2.civ at mail.mil
>> > > >
>> > > > -----Original Message-----
>> > > > From: Raby, John W CIV USARMY RDECOM ARL (US)
>> > > > Sent: Thursday, January 10, 2019 8:13 AM
>> > > > To: 'met_help at ucar.edu' <met_help at ucar.edu>
>> > > > Subject: RE: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728]
MET
>> > > > docker-container in Singularity (UNCLASSIFIED)
>> > > >
>> > > > CLASSIFICATION: UNCLASSIFIED
>> > > >
>> > > > John -
>> > > >
>> > > > Thanks for responding while at the mtg and for following up
with
>> Mike.
>> > > >
>> > > > R/
>> > > > John
>> > > >
>> > > > -----Original Message-----
>> > > > From: John Halley Gotway via RT
>> > > > [Caution-Caution-Caution-Caution-mailto:met_help at ucar.edu]
>> > > > Sent: Thursday, January 10, 2019 8:04 AM
>> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
>> > > > <john.w.raby2.civ at mail.mil>
>> > > > Subject: Re: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728]
MET
>> > > > docker-container in Singularity (UNCLASSIFIED)
>> > > >
>> > > > All active links contained in this email were disabled.
Please
>> > > > verify the identity of the sender, and confirm the
authenticity of
>> > > > all links contained within the message prior to copying and
>> > > > pasting the address to a Web browser.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > ----
>> > > >
>> > > > John,
>> > > >
>> > > > I forwarded your question to Mike Kavulich, who's planning on
>> > > > working with Singularity. Listed below is my question and
his
>> > > > response. We are both at AMS this week.
>> > > >
>> > > > ---------------------------------------
>> > > >
>> > > > Mike,
>> > > >
>> > > > I got a followup question from John Raby from the Army. He's
>> > > > wondering if we've made any progress in using Singularity
>> containers.
>> > > >
>> > > > If I remember correctly, once we make containers available
via
>> > > > DockerHub, this is very easy. But until then, it's more
difficult.
>> > > > Do you happen to have a Singularity version of the MET
container
>> > > > he
>> > > could use for testing?
>> > > >
>> > > > ---------------------------------------
>> > > >
>> > > > John,
>> > > >
>> > > > I do not have a singularity container ready, I haven't
>> > > > re-installed the necessary software since my mac had to be
>> > > > serviced. I can get to it next week at the earliest.
>> > > >
>> > > > Mike
>> > > >
>> > > > ---------------------------------------
>> > > >
>> > > >
>> > > > On Mon, Jan 7, 2019 at 3:01 PM Raby, John W USA CIV via RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > <Caution-Caution-Caution-Caution-Caution-url:
>> > > > > Caution-Caution-Caution-Caution-
>> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.
>> > > > > html?id=87
>> > > > > 728 >
>> > > > >
>> > > > > CLASSIFICATION: UNCLASSIFIED
>> > > > >
>> > > > > John -
>> > > > >
>> > > > > Ref the email update below on the possibility of a
Singularity
>> > > > > container for MET. Has there been any progress on this?
>> > > > >
>> > > > > Thanks.
>> > > > >
>> > > > > R/
>> > > > > John
>> > > > >
>> > > > > -----Original Message-----
>> > > > > From: John Halley Gotway via RT
>> > > > > [Caution-Caution-Caution-Caution-Caution-
mailto:met_help at ucar.edu]
>> > > > > Sent: Wednesday, November 14, 2018 4:40 PM
>> > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
>> > > > > <john.w.raby2.civ at mail.mil>
>> > > > > Subject: [Non-DoD Source] Fwd: [rt.rap.ucar.edu #87728] MET
>> > > > > docker-container in Singularity (UNCLASSIFIED)
>> > > > >
>> > > > > All active links contained in this email were disabled.
Please
>> > > > > verify the identity of the sender, and confirm the
authenticity
>> > > > > of all links contained within the message prior to copying
and
>> > > > > pasting the address to a Web browser.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > ----
>> > > > >
>> > > > > John,
>> > > > >
>> > > > > I forwarded your question about Singularity to other folks
in
>> > > > > the DTC and am forwarding their response.
>> > > > >
>> > > > > Specifically, it looks like once we get the MET container
posted
>> > > > > to DockerHub than importing it into Singularity becomes a
>> one-liner.
>> > > > > Seems like a good motivation to get that done!
>> > > > >
>> > > > > Thanks,
>> > > > > John
>> > > > >
>> > > > > ---------- Forwarded message ---------
>> > > > > From: Michael Kavulich <kavulich at ucar.edu>
>> > > > > Date: Wed, Nov 14, 2018 at 12:16 PM
>> > > > > Subject: Re: [rt.rap.ucar.edu #87728] MET docker-container
in
>> > > > > Singularity
>> > > > > (UNCLASSIFIED)
>> > > > > To: John Halley Gotway <johnhg at ucar.edu>
>> > > > > Cc: Jamie Wolff <jwolff at ucar.edu>, Michelle Harrold
>> > > > > <harrold at ucar.edu>, Tara Jensen <jensen at ucar.edu>
>> > > > >
>> > > > >
>> > > > > John,
>> > > > >
>> > > > > I have only tested singularity with standard images from
>> dockerhub.
>> > > > > This process is relatively easy:
>> > > > > Caution-
>> > > > > Caution-Caution-Caution-Caution-Caution-
https://www.sylabs.io/guides/2.5
>> > > > > /user-guid
>> > > > > e/singular
>> > > > > ity_and_do
>> > > > > cker.html?highlight=docker
>> > > > >
>> > > > > Converting images locally is a bit more involved, and I
have not
>> > > > > tried this
>> > > > > myself:
>> > > > > Caution-
>> > > > > Caution-Caution-Caution-Caution-Caution-
https://groups.google.com/a/lbl.
>> > > > > gov/forum/
>> > > > > #!topic/si
>> > > > > ngularity/
>> > > > > ZXOaNaM_6MY
>> > > > >
>> > > > > This all requires that you are on a machine with both
Docker and
>> > > > > Singularity installed, which is a big limitation.
>> > > > >
>> > > > > The easiest thing for users would be if we did this
conversion
>> > > > > ourselves and provided both Docker and Singularity images.
>> > > > > Provided I have hours left after the prep for the tutorial
is
>> > > > > completed this is something I can look into.
>> > > > >
>> > > > > Michael Kavulich, Jr.
>> > > > > Associate Scientist, National Center for Atmospheric
Research
>> > > > > (NCAR) Joint Numerical Testbed (JNT), Research Applications
>> > > > > Laboratory
>> > > > > (RAL)
>> > > > > * kavulich at ucar.edu <kavulich at ucar.edu>*
>> > > > >
>> > > > >
>> > > > > On Fri, Nov 9, 2018 at 9:25 AM John Halley Gotway
>> > > > > <johnhg at ucar.edu>
>> > > > wrote:
>> > > > >
>> > > > > > Mike,
>> > > > > >
>> > > > > > John Raby is a MET user from the Army Research Lab who is
>> > > > > > interested in running MET through a container. We
currently
>> > > > > > have it in a Docker container, but apparently Singularity
is
>> > > > > > now available on their super-computer, Centennial, for
testing.
>> > > > > >
>> > > > > > He's written MET-Help asking how to convert the MET
Docker
>> > > > > > container to Singularity for testing on Centennial.
>> > > > > >
>> > > > > > I have never run Singularity but know that you have.
What
>> > > > > > should I
>> > > do?
>> > > > > >
>> > > > > > Thanks,
>> > > > > > John
>> > > > > >
>> > > > > >
>> > > > > > ---------- Forwarded message ---------
>> > > > > > From: Raby, John W USA CIV via RT <met_help at ucar.edu>
>> > > > > > Date: Fri, Nov 9, 2018 at 9:07 AM
>> > > > > > Subject: [rt.rap.ucar.edu #87728] MET docker-container in
>> > > > > > Singularity
>> > > > > > (UNCLASSIFIED)
>> > > > > > To:
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > Fri Nov 09 09:07:15 2018: Request 87728 was acted upon.
>> > > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
>> > > > > > Queue: met_help
>> > > > > > Subject: MET docker-container in Singularity
(UNCLASSIFIED)
>> > > > > > Owner: Nobody
>> > > > > > Requestors: john.w.raby2.civ at mail.mil
>> > > > > > Status: new
>> > > > > > Ticket <Caution-Caution-Caution-Caution-Caution-Caution-
url:
>> > > > > > Caution-Caution-Caution-Caution-
>> > Caution-Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.
>> > > > > > html?id=87
>> > > > > > 728 >
>> > > > > >
>> > > > > >
>> > > > > > CLASSIFICATION: UNCLASSIFIED
>> > > > > >
>> > > > > > I received the email below from the developer who is
>> > > > > > implementing Singularity on the HPC in response to my
question
>> > > > > > about running an already-built docker-container using
>> > > > > > Singularity Runtime. They are asking users to start test
runs
>> > > > > > with their already-built Singularity
>> > > > > containers.
>> > > > > > From his guidance below, can the MET docker-container be
made
>> > > > > > to run in Singularity Runtime?
>> > > > > >
>> > > > > > Thanks.
>> > > > > >
>> > > > > > R/
>> > > > > > John
>> > > > > >
>> > > > > > Mr. John W. Raby
>> > > > > > U.S. Army Research Laboratory
>> > > > > > White Sands Missile Range, NM 88002
>> > > > > > (575) 678-2004 DSN 258-2004
>> > > > > > FAX (575) 678-1230 DSN 258-1230
>> > > > > > Email: john.w.raby2.civ at mail.mil
>> > > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
>> > > > > > Sent: Friday, November 9, 2018 8:35 AM
>> > > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
>> > > > > > <john.w.raby2.civ at mail.mil>
>> > > > > > Subject: RE: Singularity Runtime installed on Centennial
>> > > > > > (UNCLASSIFIED)
>> > > > > >
>> > > > > >
>> > > > > > You would need to convert the docker image to a
singularity
>> image.
>> > > > > > From there you might have to change some permissions or
>> > > > > > re-configure to use non-root directories. It depends on
how
>> > > > > > tightly coupled the application is to root inside the
image.
>> > > > > > You will need root on a machine to convert the image to
>> Singularity.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Matt
>> > > > > >
>> > > > > > ________________________________________
>> > > > > > From: Raby, John W CIV USARMY RDECOM ARL (US)
>> > > > > > Sent: Thursday, November 08, 2018 5:24 PM
>> > > > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
>> > > > > > Subject: RE: Singularity Runtime installed on Centennial
>> > > > > > (UNCLASSIFIED)
>> > > > > >
>> > > > > > CLASSIFICATION: UNCLASSIFIED
>> > > > > >
>> > > > > > Thanks for all your work on implementing Singularity and
for
>> > > > > > the notification, Matt. I most certainly will send this
out to
>> > > > > > others I know who are interested. I know of an already-
built
>> > > > > > docker-container which I'm interested in. Does this fall
into
>> > > > > > the category of being suitable for testing on Centennial?
>> > > > > >
>> > > > > > R/
>> > > > > > John
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: Dwyer, Matthew R CTR USARMY RDECOM (US)
>> > > > > > Sent: Thursday, November 8, 2018 3:18 PM
>> > > > > > To: Dwyer, Matthew R CTR USARMY RDECOM (US)
>> > > > > > <matthew.r.dwyer7.ctr at mail.mil
>> > > > > > >
>> > > > > > Subject: Singularity Runtime installed on Centennial
>> > > > > >
>> > > > > > Good News!
>> > > > > >
>> > > > > >
>> > > > > > Containers are installed on Centennial (for early
access).
>> > > > > > This is the first production machine container
installation at
>> > > > > > the ARL
>> > DSRC.
>> > > > > >
>> > > > > > Singularity runtime is installed on Centennial which
means you
>> > > > > > still cannot build your containers on ARL DSRC resources.
If
>> > > > > > you have already-built singularity containers and would
like
>> > > > > > to run them please go ahead and test them out now. The
more
>> > > > > > bugs/misconfigurations we find now will provide a
smoother
>> > > > > > user-experience with container workloads. Let me know of
>> > > > > > successes and failures and please send to
>> > > > > others who are interested.
>> > > > > >
>> > > > > >
>> > > > > > Thank You,
>> > > > > > Matt Dwyer
>> > > > > > 4102786819
>> > > > > >
>> > > > > > CLASSIFICATION: UNCLASSIFIED
>> > > > > > CLASSIFICATION: UNCLASSIFIED
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > > CLASSIFICATION: UNCLASSIFIED
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > > CLASSIFICATION: UNCLASSIFIED
>> > > > CLASSIFICATION: UNCLASSIFIED
>> > > >
>> > > >
>> > > >
>> > >
>> > > CLASSIFICATION: UNCLASSIFIED
>> > >
>> > >
>> > >
>> >
>> > CLASSIFICATION: UNCLASSIFIED
>> >
>> >
>> >
>>
>> CLASSIFICATION: UNCLASSIFIED
>>
>>
>>
------------------------------------------------
More information about the Met_help
mailing list