NOAA-C5

where the mpp_send interface is mapped to the MPI_Isend – non-blocking send. The application launches several outstanding non-blocking send operations to different peers but uses the same send buffer (msgsize fortran variable) for all operations w/o waiting. The size of the send is 4 bytes, so in most cases it proceeds through bcopy eager protocol (ie the MPI library would copy the input to internal buffers) and in that case the application continues successfully.  However, when the number of outstanding requests exceeds internal library resources the requests may go to pending mode and in this case the same buffer is modified with the next mpp_send call, causing an error. Please advise.

[A1] One suggested solution to this specific issue that eliminates this assumption is

581c581
< integer, allocatable, dimension(:) :: send_count, recv_count, recv_size2
—
> integer, allocatable, dimension(:) :: send_count, recv_count, recv_size2, send_arr
586c586
<
—
>
1140a1141,1142
>
>
1142a1145
> if (nsend_update >0) allocate(send_arr(nsend_update))
1152c1155,1156
< call mpp_send(msgsize, plen = 1, to_pe = send_pelist(p), tag=COMM_TAG_2)
—
> send_arr(p) = msgsize
> call mpp_send(send_arr(p), plen = 1, to_pe = send_pelist(p), tag=COMM_TAG_2)
1254c1258
<
—
> if(allocated(send_arr)) deallocate(send_arr)

This same change can eliminate similar issues in CM4, ESM, and SPEAR.

Less

Is there any suggested guidance on how to find the optimal balance between these models and decompositions with different rank counts?

[A2] The atm and ocean components run in parallel, so the shortest runtime requires finding the appropriate balance across the offered hardware and PE solution that optimizes the runtime of both.  Offeror should initially execute ESM4 using the current/default decomposition/inputs for the rank counts for both the atm and ocean models to establish an improvement over the baseline (NOAA C4’s Intel Broadwell 18-core) hardware configuration, and document that result as described in Table 4-2 of the Statement of Work. The stdout for each of these runs provides all of the component runtimes for atmosphere, land, ice, ocean so that Offeror may assess how well the initial configuration is balanced. Offeror may then adjust the horizontal domain decomposition to further optimize or balance the problem as they see fit. The intent of these adjustments is to optimize the ranks / threads for each component to balance the runtimes, avoiding overall load imbalance. The Offeror may experiment with different task and thread layouts for the optimal configuration. While the Offeror may change the horizontal decomposition at will, the number of vertical levels and the vertical decomposition must not be altered. It is noted that, in general, if a given component’s runtime is dominated by parallel overheads, then increasing the tasks and / or threads for that component will not improve overall runtime. Further, increasing the parallelism for each component in isolation may help determine which gives the greatest reduction in overall runtime for a given increase in parallelism.While the calculation of the Figure of Merit V uses the main loop time (the longest of these components), Company will evaluate the independent component runtimes of both the baseline and optimized case as well. Please ensure that both the baseline and optimized cases are reported in the format shown in Table 4-2, and that the stdout for both baseline and optimized is included in your response. Calculation of V is expected to use the optimized result.The demonstration of the impact of hardware features that might include multi-threading, or similar is encouraged. The use of additional threads can be changed by altering coupler_nml::atmos_nthreads in the input.nml for the specific runs.

As provided, the benchmarks take advantage of an internal framework for process pinning, rank affinity, and other similar features that interoperate with the Slurm resource manager. If the Offeror is using Slurm and wants to take advantage of this internal framework, you may modify the namelist parameter coupler_nml::use_hyper_thread to .TRUE. If the Offeror prefers to accomplish this via external tools or command-line options, the internal framework can be disabled by adding a namelist section to the input.nml:

&fms_affinity_nml

affinity = .false.

/

Less

Is this expected behavior in that the decomposition may affect pass/fail? Are there recommendations/guidelines for creating decompositions for layout_x, layout_y, io_layout?

[A3] The only validation requirement for these tests is that the answer is bit-wise reproducible given the same parallel decomposition, the same binary executable, and the same inputs. There is no specific capability of verifying correctness if the compiler flags, inputs, or decomposition are changed. To ensure accuracy of the result, the Offeror should ensure that the baseline problem validates successfully, and note this in their response.Beyond the validation of the baseline problem-If the Offeror changes the code, then Offeror should demonstrate that that same code base, with the same decomposition and compiler flags still matches the baseline result.If the Offeror uses different compiler options and/or adjusts the horizontal decomposition, potential changes to the floating-point numerics may generate a different result and, as a result, a validation failure using the provided script. If this is the case, Offeror may simply delete the cksums and INITIAL files in the ${BASEDIR}/SHiELD/VERIFICATION/small/ directory, and then rerun that case to generate a “new” result in the VERIFICATION directory. Offeror should then repeat the same run, with successful verification.

The SHiELD benchmark is a 3km average grid spacing production-sample using 3072x3072x79(L) degrees of freedom per cubed-sphere tile with an atmosphere-only run. The Offeror may assess different parallel decompositions in the horizontal dimension. Generally, the application performs better when the size of each decomposed domain is similar in each of the two dimensions. MPI overheads are not expected to play a significant role in this benchmark.

Less

Can ORNL please provide those times or state that condition a) in Section 4.3 does not apply for the SHiELD benchmark?

[A4] The baseline times for the small SHiELD case are main loop time: 10,325 seconds; total runtime: 10,510 seconds; 6144 cores. This is corrected in SOW v.1.0.3 (Amendment 1).

Less

[A5] ESM4 contains multiple inter-related dependencies that can be quite nuanced. Namelist options that control the horizontal (x and y) domain decomposition may also have dependencies with related namelist options. Further, if any of the components use MPI rank masking, modifications to the domain decomposition are further nuanced. Offerors may adjust namelist options, but should proceed cautiously as there is no expedient answer to this question. Given that the baseline case provided is balanced for the current C4 system, Offerors may choose to report the baseline without significant effort to additional decomposition efforts.

Less

If so, are there restrictions and/or guidance around such modifications?

[A6] The ocean component of SPEAR does not use MPI-rank masking, making this significantly easier to accomplish, The baseline case is structured as follows: SPEAR is running (concurrently) 4 ensembles. Each ensemble is using 108 PEs, for a total of 4*108 = 432 PEs. This is described in INPUT/MOM_layout as:

#override LAYOUT = 12,9

where the product of the two numbers, here 12,9, is the number of PEs per ensemble. As a specific example, the Offeror could adjust the number of PEs per ensemble. A target of 45 PEs per ensemble, 4 ensembles, 180 PEs total would require a change to the INPUT/MOMN_layout to, for example:

#override LAYOUT = 5,9

and change the namelist variable “ocean_npes” under the namelist “coupler_nml” to “45”.

Offeror should evaluate the impact to the main loop time, as changes to the ocean component may generate or exacerbate an imbalance.

Less

[A8] ORNL provides a file upload service that an Offeror may use to upload their response to the NOAA C-5 Solicitation and Offer. Offerors may access that service via https://ftp.ornl.gov/fileupload.html     Files uploaded using this form can only be accessed within the ORNL network. In addition, Offerors shall explicitly list only [email protected] as the recipient.  This ensures that the ORNL Procurement Officer for this specific Solicitation and Offer is the only person to receive the link to those files. Up to five individual files may be uploaded using this service. Each file is limited to 4GB. Additional information is provided at the referenced web site. After security scanning and receipt of files, they will be deleted from the FTP server by ORNL Procurement.

An Offeror may still choose to use email. The limit of the size of an individual email is 25MB for the ORNL email system. Refer to the instruction in the Solicitation and Offer for further details.

If Offeror intends to use the File Upload Service, please upload a sample document no later than 4pm Eastern Time on January 15, 2021 and ORNL will verify receipt.

Less

That table prescribes a contribution of 20% for the ESM Large benchmark while Table 4-2 specifically states that there is no contribution of ESM Large benchmark (Copies is stated as 0). Please clarify how Offeror is to interpret this discrepancy.

[A9] Table 4-2 indicates that the EMS Large benchmark does not contribute to the calculation of performance figure V, and is (effectively) optional. However, as Offeror notes, Table 4-1 (through Amendment 3) incorrectly stipulated that ESM Large would contribute 20% of the improvement above the baseline to the performance figure V. Laboratory has revised Table 4-1 to reflect the contribution of each benchmark to the overall improvement. This increases the contribution from CM4, ESM Small, SHiELD and SPEAR. UFS remains 5%. Please reference Attachment 1 version 1.0.4 for the revised weights. As this question was received very late in the solicitation process, Laboratory will extend the due date two calendar days to January 21, 2021 to allow Offerors to revisit the changes to the calculation of V.

Less