Combining MPI and Threads (FFTW 3.3.11)

### 6.11 Combining MPI and Threads ¶

In certain cases, it may be advantageous to combine MPI (distributed-memory) and threads (shared-memory) parallelization. FFTW supports this, with certain caveats. For example, if you have a cluster of 4-processor shared-memory nodes, you may want to use threads within the nodes and MPI between the nodes, instead of MPI for all parallelization.

In particular, it is possible to seamlessly combine the MPI FFTW routines with the multi-threaded FFTW routines (see Multi-threaded FFTW). However, some care must be taken in the initialization code, which should look something like this:

    int threads_ok;

    int main(int argc, char **argv)
    {
        int provided;
        MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, &provided);
        threads_ok = provided >= MPI_THREAD_FUNNELED;

        if (threads_ok) threads_ok = fftw_init_threads();
        fftw_mpi_init();

        ...
        if (threads_ok) fftw_plan_with_nthreads(...);
        ...

        MPI_Finalize();
    }

First, note that instead of calling `MPI_Init`, you should call `MPI_Init_threads`, which is the initialization routine defined by the MPI-2 standard to indicate to MPI that your program will be multithreaded. We pass `MPI_THREAD_FUNNELED`, which indicates that we will only call MPI routines from the main thread. (FFTW will launch additional threads internally, but the extra threads will not call MPI code.) (You may also pass `MPI_THREAD_SERIALIZED` or `MPI_THREAD_MULTIPLE`, which requests additional multithreading support from the MPI implementation, but this is not required by FFTW.) The `provided` parameter returns what level of threads support is actually supported by your MPI implementation; this must be at least `MPI_THREAD_FUNNELED` if you want to call the FFTW threads routines, so we define a global variable `threads_ok` to record this. You should only call `fftw_init_threads` or `fftw_plan_with_nthreads` if `threads_ok` is true. For more information on thread safety in MPI, see the MPI and Threads section of the MPI-2 standard.

Second, we must call `fftw_init_threads` before `fftw_mpi_init`. This is critical for technical reasons having to do with how FFTW initializes its list of algorithms.

Then, if you call `fftw_plan_with_nthreads(N)`, every MPI process will launch (up to) `N` threads to parallelize its transforms.

For example, in the hypothetical cluster of 4-processor nodes, you might wish to launch only a single MPI process per node, and then call `fftw_plan_with_nthreads(4)` on each process to use all processors in the nodes.

This may or may not be faster than simply using as many MPI processes as you have processors, however. On the one hand, using threads within a node eliminates the need for explicit message passing within the node. On the other hand, FFTW’s transpose routines are not multi-threaded, and this means that the communications that do take place will not benefit from parallelization within the node. Moreover, many MPI implementations already have optimizations to exploit shared memory when it is available, so adding the multithreaded FFTW on top of this may be superfluous.