Using MPI, I’ve just managed to get a speedup of about 6 for protein structure searches on 8 CPUs without even thinking about optimizations. If that scales across multiple nodes, the SALAMI server might soon become fast enough for interactive use…
MPI in general, and the Perl Parallel::MPI::Simple module are pretty neat APIs. However, despite its name, MPI::Simple still follows the standard MPI API quite closely and is thus not as “simple” as it could be. For many applications, a process manager would already be sufficient. Such a module would create and manage a pool of worker processes and distribute tasks in a round robin fashion.
Both apple and the Qt library offer something similar for thread pools, so why not do the same for MPI processes?
Surely, many others before me have tackled embassasingly parallel tasks before. However, CPAN seems to offer nothing like this. Is such a module simply too trivial so that nobody has bothered publishing their code? Looking at CPAN in general, this doesn’t seem to have stopped folks in the past…
I’ll probably try to cook something up tomorrow.