[Dart-dev] [9976] DART/trunk/assim_model/assim_model_mod.f90: We no longer pre-fill the netCDF variables since we write entire hyperslabs of data .

[nancy at ucar.edu]

Revision: 9976
Author:   thoar
Date:     2016-03-24 13:10:01 -0600 (Thu, 24 Mar 2016)
Log Message:
-----------
We no longer pre-fill the netCDF variables since we write entire hyperslabs of data.
The creation and writing of the Prior_Diag.nc and Posterior_Diag.nc files is noticeably faster.
How much faster depends on how many copies are getting written.

Here are the results for the tests with POP, using the large file support.
I ran this in the premium queue to expedite my test.

job #    cpu_time  run_time(s)  turnaround   max_mem(MB)     GAUS
filling (historical method)
677327    214629    512          532          372992         97.63
677509    227400    521          1741         450231         99.12
677769    201727    508          1046         272958         96.7
677899    203409    454          477          339518         86.8
677959    208911    455          477          343048         86.99
                  490.00                                     93.45

no_fill
676977    160444    419          672          403190         80.27
677341    142495    404          1694         354251         83.25
677620    146005    428          603          395597         81.76
677755    138957    383          440          272061         73.73
677822    143105    477          835          338220         90.72
                  422.20                                     81.95

490 - 422.2 = 67.80 s faster per job, on average
81.95 ~ 12% cheaper than 93.45 (core-hour charge at premium)

The first job in each test was a 3 day advance and then assimilate.
The remaining jobs were 1 day advances and then assimilate.
Each assimilation had about 7000 observations.
The first jobs only wrote 14 copies to each file ... 1.6GB
The remaining jobs wrote out 34 copies to each file ... 3.8 GB
The netCDF variables were bit-for-bit identical.

Modified Paths:
--------------
    DART/trunk/assim_model/assim_model_mod.f90

-------------- next part --------------
Modified: DART/trunk/assim_model/assim_model_mod.f90
===================================================================
--- DART/trunk/assim_model/assim_model_mod.f90	2016-03-24 16:28:38 UTC (rev 9975)
+++ DART/trunk/assim_model/assim_model_mod.f90	2016-03-24 19:10:01 UTC (rev 9976)
@@ -85,10 +85,10 @@
 logical :: module_initialized = .false.
 
 ! Global storage for default restart formats
-character(len = 16) :: read_format = "unformatted", write_format = "unformatted"
+character(len=16) :: read_format = "unformatted", write_format = "unformatted"
 
 ! Global storage for error string output
-character(len = 129)  :: msgstring
+character(len=512)  :: msgstring
 
 !-------------------------------------------------------------
 ! Namelist with default values
@@ -210,7 +210,7 @@
 integer, OPTIONAL,intent(in) :: lagID
 type(netcdf_file_type)       :: ncFileID
 
-integer :: i, metadata_length, nlines, linelen, createmode
+integer :: i, metadata_length, nlines, linelen, createmode, oldmode
 
 integer ::   MemberDimID,   MemberVarID     ! for each "copy" or ensemble member
 integer ::     TimeDimID,     TimeVarID
@@ -294,9 +294,14 @@
               'init_diag_output', 'put_att assim_model_source '//trim(ncFileID%fname))
 call nc_check(nf90_put_att(ncFileID%ncid, NF90_GLOBAL, "assim_model_revision", revision ), &
               'init_diag_output', 'put_att assim_model_revision '//trim(ncFileID%fname))
+
 call nc_check(nf90_put_att(ncFileID%ncid, NF90_GLOBAL, "assim_model_revdate", revdate ), &
               'init_diag_output', 'put_att assim_model_revdate '//trim(ncFileID%fname))
 
+msgstring = nf90_inq_libvers()
+call nc_check(nf90_put_att(ncFileID%ncid, NF90_GLOBAL, "netcdf_version", trim(msgstring) ), &
+              'init_diag_output', 'put_att netcdf_version '//trim(ncFileID%fname))
+
 if (present(lagID)) then
    call nc_check(nf90_put_att(ncFileID%ncid, NF90_GLOBAL, "lag", lagID ), &
                  'init_diag_output', 'put_att lag '//trim(ncFileID%fname))
@@ -321,7 +326,6 @@
 call nc_check(nf90_put_att(ncFileID%ncid, MemberVarID, "valid_range", &
               (/ 1, copies_of_field_per_time /)), 'init_diag_output', 'put_att valid_range')
 
-
 !    Metadata for each Copy
 call nc_check(nf90_def_var(ncid=ncFileID%ncid,name="CopyMetaData", xtype=nf90_char,    &
               dimids = (/ metadataDimID, MemberDimID /),  varid=metadataVarID), &
@@ -353,7 +357,12 @@
 
 !-------------------------------------------------------------------------------
 ! Leave define mode so we can fill
+! Set the NOFILL ... noticeable performance gain from not prefilling.
+! Tests with WRF & POP show bit-for-bit under 'classic' storage layer.
 !-------------------------------------------------------------------------------
+
+i = nf90_set_fill(ncFileID%ncid, NF90_NOFILL, oldmode)
+call nc_check(i, 'init_diag_output', 'set_fill NOFILL on '//trim(ncFileID%fname))
 call nc_check(nf90_enddef(ncFileID%ncid), 'init_diag_output', 'enddef '//trim(ncFileID%fname))
 
 !-------------------------------------------------------------------------------