<div dir="ltr"><div class="gmail_default" style="font-size:small">Laura,</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">I'm honestly not sure why we are seeing the issue with the Intel version of ESMF. I am waiting for them to release a version 7.0.1 of ESMF, as they said this would have some fixes that might help with the Intel compiler.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">My guess is that the Intel compiler may be trying to optimize things under the hood, and this can cause problems. Sometimes we compile things with optimization turned off to ward against these problems.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">I'll get in touch with the ESMF group to see if they have some ideas about what's going on.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">--Mary</div><div class="gmail_default" style="font-size:small"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 25, 2017 at 3:05 PM, Laura Fowler <span dir="ltr"><<a href="mailto:laura@ucar.edu" target="_blank">laura@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Mary:<br>
<br>
Thanks for replying to my e-mail so quickly. I tried gnu and as you<br>
wrote, I did not have any issues running ESMF_regrid with my "old" and<br>
"new" meshes. I would not have thought of switching compiler on my<br>
own.<br>
<br>
Do you know if I will need to use gnu when using<br>
ESMF_regrid_with_weights? I guess that I can try intel and switch to<br>
gnu if it does not work?<br>
<br>
Thanks again to you and Rich. You have been very helpful.<br>
<span class="HOEnZb"><font color="#888888"><br>
Laura<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
On Wed, Oct 25, 2017 at 12:33 PM, Mary Haley <<a href="mailto:haley@ucar.edu">haley@ucar.edu</a>> wrote:<br>
> Hi Laura (and Rick),<br>
><br>
> I think there are issues with the Intel version of ESMF_RegridWeightGen,<br>
> which generates the weights file. We've had other issues trying to build<br>
> this application using the Intel compiler, and we've been in touch with the<br>
> ESMF group about it.<br>
><br>
> Meanwhile, things seem to work if I use the GNU version of NCL on cheyenne:<br>
><br>
> module load gnu<br>
> module load ncl<br>
> which ncl<br>
> /glade/u/apps/ch/opt/ncl/6.4.<wbr>0/gnu/6.2.0/bin/ncl<br>
><br>
> I just ran your script, and it completed with the following output:<br>
><br>
> (0) ESMF_regrid_with_weights: retrieving interpolation weights ...<br>
> (0) ESMF_regrid_with_weights: calling sparse_matrix_mult to apply weights...<br>
> (0) ESMF_regrid_with_weights: dstData<br>
> (0) Dimensions: 1798 3598<br>
> (0) minSrcData: 1.5258789063e-05<br>
> (0) maxSrcData: 0.999069673989844<br>
> (0) minDstData: 1.5258789063e-05<br>
> (0) maxDstData: 0.9990587636447192<br>
><br>
> Variable: mpas_regrid<br>
> Type: double<br>
> Total Size: 51753632 bytes<br>
> . . .<br>
> Number of Dimensions: 2<br>
> Dimensions and sizes: [lat | 1798] x [lon | 3598]<br>
> Coordinates:<br>
> lat: [-89.875..89.875]<br>
> lon: [0.125..359.875]<br>
><br>
> Number Of Attributes: 3<br>
> missing_value : 9.969209968386869e+36<br>
> remap : remapped via ESMF_regrid_with_weights: Bilinear<br>
> _FillValue : 9.969209968386869e+36<br>
><br>
><br>
> --Mary<br>
><br>
> On Tue, Oct 24, 2017 at 5:14 PM, Laura Fowler <<a href="mailto:laura@ucar.edu">laura@ucar.edu</a>> wrote:<br>
>><br>
>> Hi Rick:<br>
>><br>
>> Thank you for looking into this. The newer mesh I am trying to regrid<br>
>> has 6488066 cells while the mesh I regridded a couple of years ago had<br>
>> more cells (6848514). So I did not think that it was the size of the<br>
>> mesh that was the culprit.<br>
>><br>
>> Here is what I found:<br>
>><br>
>> 1. Using the largest mesh (6848514 cells) and on yellowstone, I tested<br>
>> the regrid script I used a couple of years ago using ncl/6.3.0 and<br>
>> ncl/<a href="http://6.4.0." rel="noreferrer" target="_blank">6.4.0.</a> I did not have any issue. You can look at the source,<br>
>> destination, and weight files in the directories<br>
>> /glade2/scratch2/laura/ncl/<wbr>yellowstone.ncl-6.3.0 and<br>
>> /glade2/scratch2/laura/ncl/<wbr>yellowstone.ncl-6.4.0. The ncl script is<br>
>> regrid.to_CMORPHdata.ncl<br>
>><br>
>><br>
>> 2. Then, I tested the same script but on cheyenne and I got the same<br>
>> SIGSEGV using ncl/6.4.0 or ncl/<a href="http://6.3.0." rel="noreferrer" target="_blank">6.3.0.</a> So it seems that it may have to<br>
>> do with ncl on cheyenne only.<br>
>><br>
>><br>
>> 3. Unfortunately, I cannot use yellowstone to regrid my newest mesh<br>
>> since it is uses the cdf5 format which I cannot read on yellowstone.<br>
>><br>
>><br>
>> Hope that this helps to resolve this issue.<br>
>> Laura<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> On Tue, Oct 24, 2017 at 4:19 PM, Rick Brownrigg <<a href="mailto:brownrig@ucar.edu">brownrig@ucar.edu</a>> wrote:<br>
>> > Just to follow up on this, the message regarding max-value-size is<br>
>> > related<br>
>> > to the debugger, so fixed limits within the ESMF software are likely not<br>
>> > the<br>
>> > issue.<br>
>> ><br>
>> > Nonetheless, the SEGV is occuring in the ESMF software and appears to be<br>
>> > happening in the NetCDF library, function<br>
>> > netcdf_expanded.f90::nf90_get_<wbr>var_2d_fourbyteint(). For anyone else<br>
>> > looking into this, the line number is 1960, and a link to the current<br>
>> > source<br>
>> > is:<br>
>> ><br>
>> ><br>
>> > <a href="https://github.com/Unidata/netcdf-fortran/blob/master/fortran/netcdf_expanded.f90" rel="noreferrer" target="_blank">https://github.com/Unidata/<wbr>netcdf-fortran/blob/master/<wbr>fortran/netcdf_expanded.f90</a><br>
>> ><br>
>> > (I don't know what version of NetCDF ESMF may be linked against, but<br>
>> > that<br>
>> > line number is in the right function).<br>
>> ><br>
>> ><br>
>> ><br>
>> > On Tue, Oct 24, 2017 at 3:48 PM, Rick Brownrigg <<a href="mailto:brownrig@ucar.edu">brownrig@ucar.edu</a>><br>
>> > wrote:<br>
>> >><br>
>> >> Hi Laura,<br>
>> >><br>
>> >> I don't really know much about the regridding process, but what I have<br>
>> >> been able to surmise running the script:<br>
>> >><br>
>> >> i) NCL reads the MPAS, and creates source*.nc and destination*.nc<br>
>> >> files.<br>
>> >> These appear to reflect the geometry of the src/dest grids<br>
>> >><br>
>> >> ii) The actual regridding is is done by ESMF software, with a command:<br>
>> >><br>
>> >> ESMF_RegridWeightGen --source <a href="http://source_grid_file.nc" rel="noreferrer" target="_blank">source_grid_file.nc</a> --destination<br>
>> >> <a href="http://destination_grid_file.nc" rel="noreferrer" target="_blank">destination_grid_file.nc</a> --weight<br>
>> >> <a href="http://weights_onCells.15-3Mesh_to_0.15rectangular.nc" rel="noreferrer" target="_blank">weights_onCells.15-3Mesh_to_0.<wbr>15rectangular.nc</a> --src_type ESMF -i<br>
>> >><br>
>> >> This program SEGVs almost immediately, with a message:<br>
>> >><br>
>> >> "values=<error reading variable: value requires 155713536 bytes, which<br>
>> >> is<br>
>> >> more than max-value-size>...."<br>
>> >><br>
>> >> That value is exactly the size of one of the variables in the<br>
>> >> source*.nc<br>
>> >> file. So it looks like some internal limit is being exceeded in the<br>
>> >> ESMF<br>
>> >> software.<br>
>> >><br>
>> >> Is this one of the larger MPAS files you've attempted to regrid? I<br>
>> >> wonder<br>
>> >> if anyone else can comment on this? Those on the glade file system can<br>
>> >> see<br>
>> >> all the relevant files under /glade/scratch/brownrig<br>
>> >><br>
>> >> I'm not sure what to tell you as a work-around. Without a debug version<br>
>> >> of<br>
>> >> the code, its nearly impossible for me to tell much more or to<br>
>> >> detemine<br>
>> >> what the limits might be. Wish I had a better answer.<br>
>> >><br>
>> >> Rick<br>
>> >><br>
>> >><br>
>> >><br>
>> >> On Tue, Oct 24, 2017 at 10:13 AM, Laura Fowler <<a href="mailto:laura@ucar.edu">laura@ucar.edu</a>> wrote:<br>
>> >>><br>
>> >>> Hi:<br>
>> >>><br>
>> >>> I am trying to regrid an MPAS unstructured mesh to a rectangular mesh<br>
>> >>> on cheyenne using ncl/<a href="http://6.4.0." rel="noreferrer" target="_blank">6.4.0.</a> My script crashes with a SIGSEGV and I am<br>
>> >>> not understanding where this comes from. I have done this successfully<br>
>> >>> in the past but do not see what I am doing wrong right now, but I<br>
>> >>> recall that it was with an ealier version of ncl. I am attaching the<br>
>> >>> output of my script in regrid.to_rectMesh.out.<br>
>> >>><br>
>> >>> The script itself can be found in<br>
>> >>><br>
>> >>><br>
>> >>> /glade2/scratch2/laura/MPAS.<wbr>PacificOcean/initialization.<wbr>centeredPacificOceanMesh.15-<wbr>3km/regrid.to_rectMesh.ncl.<br>
>> >>> I also tried to regrid the same MPAS mesh to an other unstructured<br>
>> >>> mesh and got a similar SIGSEGV (see regrid.to_2621442Mesh.ncl), so I<br>
>> >>> assume that the errors are the same.<br>
>> >>><br>
>> >>> Hope you can help me figure this one out.<br>
>> >>> Thanks,<br>
>> >>> Laura<br>
>> >>><br>
>> >>><br>
>> >>> --<br>
>> >>><br>
>> >>><br>
>> >>> !-----------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------------------<br>
>> >>> Laura D. Fowler<br>
>> >>> Mesoscale and Microscale Meteorology Division (MMM)<br>
>> >>> National Center for Atmospheric Research<br>
>> >>> P.O. Box 3000, Boulder CO 80307-3000<br>
>> >>><br>
>> >>> e-mail: <a href="mailto:laura@ucar.edu">laura@ucar.edu</a><br>
>> >>> phone: <a href="tel:303-497-1628" value="+13034971628">303-497-1628</a><br>
>> >>><br>
>> >>><br>
>> >>><br>
>> >>> !-----------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------------------<br>
>> >>><br>
>> >>> ______________________________<wbr>_________________<br>
>> >>> ncl-talk mailing list<br>
>> >>> <a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a><br>
>> >>> List instructions, subscriber options, unsubscribe:<br>
>> >>> <a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/<wbr>mailman/listinfo/ncl-talk</a><br>
>> >>><br>
>> >><br>
>> ><br>
>><br>
>><br>
>><br>
>> --<br>
>><br>
>> !-----------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------------------<br>
>> Laura D. Fowler<br>
>> Mesoscale and Microscale Meteorology Division (MMM)<br>
>> National Center for Atmospheric Research<br>
>> P.O. Box 3000, Boulder CO 80307-3000<br>
>><br>
>> e-mail: <a href="mailto:laura@ucar.edu">laura@ucar.edu</a><br>
>> phone: <a href="tel:303-497-1628" value="+13034971628">303-497-1628</a><br>
>><br>
>><br>
>> !-----------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------------------<br>
>> ______________________________<wbr>_________________<br>
>> ncl-talk mailing list<br>
>> <a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a><br>
>> List instructions, subscriber options, unsubscribe:<br>
>> <a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/<wbr>mailman/listinfo/ncl-talk</a><br>
><br>
><br>
<br>
<br>
<br>
--<br>
!-----------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------------------<br>
Laura D. Fowler<br>
Mesoscale and Microscale Meteorology Division (MMM)<br>
National Center for Atmospheric Research<br>
P.O. Box 3000, Boulder CO 80307-3000<br>
<br>
e-mail: <a href="mailto:laura@ucar.edu">laura@ucar.edu</a><br>
phone: <a href="tel:303-497-1628" value="+13034971628">303-497-1628</a><br>
<br>
!-----------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------------------<br>
</div></div></blockquote></div><br></div>