[Wrf-users] Parallel NETCDF4?

[Christopher Thomas]

Thanks for your replies Dom and Wei-keng.

Yes, I gather that the pnetCDF library cannot produce NETCDF4 files, but I
have also read that the NETCD4 library can do parallel input and output
http://www.unidata.ucar.edu/software/netcdf/docs/parallel_io.html. So I
guess what I was asking is: does anyone know how to get WRF to write in
parallel to a NETCDF4 file? I am guessing that the answer is no, or someone
would have mentioned it, but if it were possible I think that implementing
it would be a very worthwhile project.

Yes, I can post-process the data to do the compression later, but the data
volumes are very high (~1.2 TB per hour simulation time) so this sort of
post-processing is a significant overhead.

Thanks for the suggestion of I/O quilting. I'll give it a try.

To answer you questions WEi-keng: I am using a lustre file system, and two
versions of WRF:  3.7 compiled with pnetCDF, and 3.7.1 compiled with serial
I/O.

Regards

Chris

On Sat, Mar 5, 2016 at 11:16 PM, Wei-keng Liao <wkliao at eecs.northwestern.edu
> wrote:

> Hi, Chris
>
> The file size difference may be due to data compression when WRF uses HDF5
> library internally to write NetCDF4 files. PnetCDF does not do compression.
> You can use "ncgen" to convert a netCDF classical file to netCDF conformed
> HDF5 files. See netCDF user guide for command-line options of ncgen.
>
> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf_utilities_guide.html#guide_ncgen
>
> As for the files produced by PnetCDF, would you mind answering my
> 2 questions. Were you writing the outputs to a Lustre file system,
> or the name of parallel file system you used if not Lustre?
> What version of WRF are you using?
>
> Wei-keng
>
> On Mar 4, 2016, at 3:54 AM, Christopher Thomas wrote:
>
> > Hi there,
> >
> > Does anyone know how to compile WRF to use parallel NETCDF but still
> output NETCDF4 files?
> >
> > I am using 512 cores and outputting data at high temporal and spatial
> resolutions; parallel NETCDF reduces my wall times by about 1/3 but more
> than doubles output file sizes over NETCDF4 serial output.
> >
> > Chris
> > _______________________________________________
> > Wrf-users mailing list
> > Wrf-users at ucar.edu
> > http://mailman.ucar.edu/mailman/listinfo/wrf-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/wrf-users/attachments/20160306/027eab7a/attachment.html