OpenFOAM

OpenFOAM works well on Spartan, but some care is required to ensure the number of files it generates is manageable.

To do this, the collated file handler and the number of IO ranks need to be set.

Collated file handler

The collated file handler can be indicated in the <case>/system/controlDict file:

//...
OptimisationSwitches
{
    fileHandler collated;
}
//...

Or via the command line when executing all the solvers/tools:

$ decomposePar -fileHandler collated

Number of IO ranks

When the collated file handler is set, you should also set the FOAM_IORANKS variable to define the group of ranks doing IO

Use the following syntax to define group of ranks of size G

$ export FOAM_IORANKS='(0 G 2G ... mG)'

For example, for grouping every 32 ranks a case with 256 processsors use:

$ export FOAM_IORANKS='(0 32 64 96 128 160 192 224 256)'

This should be set in your host environment when interacting with OpenFOAM from the command line, or in the Slurm job script before any commands invoking OpenFOAM tools or solvers. The use of groups of 32 ranks is our recommendation.