[maker-devel] Running mpi_maker in the background

[Carson Holt]

Yes.  This is an error thrown by mpiexec,.  Mpiexec must be run interactively or it will throw this error again and again and again.  So you must always do < /dev/null whenever you run any MPI process (not just maker) in the background.  It's actually in the CHPC and MPICH2 documentation as well.  There is unfortunately no way around it, it is an annoying part of MPI.  The error message at least gives you the solution though.

Thanks,
Carson




On 10/13/09 3:29 PM, "Barry Moore" <bmoore at genetics.utah.edu> wrote:

I just ran across an interesting problem running mpi_maker in the
background on a single machine and I thought I report the solution to
the list so that it gets in the archives.  I'm sure Carson has already
run across this, and may have come up with an better - or at least an
alternate solution.  So Carson let us know if there is a better way
here and if not we should probably get this in the MAKER docs at some
point.

The problem presented itself as our system drive filling up as MAKER
ran.  I was running MAKER (in terms of output and TMP files) on a
large partition with 100s of GB of space.  However the system drive
(65GB) kept filling up every time MAKER ran for more than a few
hours.  As I would delete files from that drive it would just fill
back up again.  Hao finally found the offending file which was /var/
log/messages - it was being filled with system messages like this:

Oct 11 20:03:12 derringer python2.4:
mpiexec_derringer.genetics.utah.edu (handle_stdin_input 1067): stdin
problem; if pgm is run in background, redirect from /dev/null
Oct 11 20:03:12 derringer python2.4:
mpiexec_derringer.genetics.utah.edu (handle_stdin_input 1068):
e.g.: mpiexec -n 4 a.out < /dev/null &

This was easily consuming 10s of GB per day of drive space.

The best explanation and solution that I found was here:

http://www.mail-archive.com/meep-discuss@ab-initio.mit.edu/msg01752.html

Basically that link tells you that "Running an MPI job in the
background is supported only when stdin is redirected" and so to run
mpi_maker in the background you should do something like this:

nohup mpiexec -n 4 mpi_maker < /dev/null 2> maker.error &

Carson, have you seen this before?  I guess this is a separate issue
from the MPI_Init issue that was solved internally within mpi_maker
with:

push @ARGV, 'null';


_______________________________________________
maker-devel mailing list
maker-devel at yandell-lab.org
http://yandell-lab.org/mailman/listinfo/maker-devel_yandell-lab.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://yandell-lab.org/pipermail/maker-devel_yandell-lab.org/attachments/20091013/e4912b6c/attachment.html>