Section: New Results

High-Performance Intra-node Communications

Participants : Brice Goglin, Guillaume Mercier, Stéphanie Moreaud.

We showed in [33] that the major MPI implementations had severe performance problems for large-message intra-node communication. Using the operating system assistance and the copy offload abilities of modern hardware gave a significant performance improvement that was unfortunately only available through the Open-MX stack. We thus extracted the optimized intra-node communication model out of Open-MX and created the KNem driver so as to offer the same abilities to any existing MPI stack (see Section  5.3 ).

We showed that KNem indeed improves intra-node communication performance significantly. We proposed an automatic way to determine when memory copies should be offloaded on dedicated hardware based on hardware cache sharing between the processing cores [27] . We also showed that collective MPI operations were significantly improved thanks to our model, but also required specific careful tuning due to their intensive memory requirements [45] , and we described how each communication strategy performance depends on process placement [37] .

This work was initiated in the context of our collaboration with the MPICH2 team (see Section  8.6 ) and is now also pursued with the Open MPI project (see  8.7 ).