bsddistmake – distributed extension to bsdmake, current version 0.1.2

Description

bsddistmake is a patch for OpenBSD's make(1) allowing distributed builds. The patch is last known to apply against a 19/March/2012 source update to src/usr.bin/make. This is not an official patch: if you apply it, your system is tainted!

To use, patch as noted in the Applying Patches FAQ entry. Then follow these instructions.

Whitelist distributable (and transfer-cost-efficient) Makefile targets with .DIST.
Create ~/.makehosts with lines of whitespace-delimited remote hostnames and non-zero job capacity.
Invoke the patched make(1) with the -j flag for parallel building.

The extension is only activated if -j is specified, at least one .DIST target exists, and the host file parse succeeds. Remote build hosts have no requirements beyond being accessible with ssh(1).

How does it work?

When the patched make(1) must build a .DIST target, it selects the least-loaded remote host. If all remote hosts are maxed out, it falls back to the localhost, which follows the normal build process.

For remote hosts, it distributes dependencies with scp(1), containing build files within an auto-generated build directory /tmp/make-SOURCEHOST-TIME. It then executes target commands in the build directory using ssh(1), fetching built targets with scp(1).

Caveats & Notes

You must manually remove remost hosts' staging directories in /tmp.
Interrupting builds might not interrupt remote hosts' computation.
You'll need input-less ssh(1) access (e.g., public keys) to your remote hosts.
For best performance, start up ControlMaster=yes connections (see ssh_config(5)) running to your build hosts prior to use. I might integrate this into the patch when I get tired of doing so manually.
If you have many remote jobs, you'll need to bump the remote host's MaxSessions in sshd_config(5).
Specifying -j above 30 or so will might cause fork(2) to fail and make(1) to bail.

Long Example

The following is an example scenario using bsddistmake to distribute computation. Assume I have two remot hosts, foo1 and foo2, with two possible job slots each.

% cat ~/.makehosts
foo1 2
foo2 2

For completeness, assume I have two open connections to these hosts prior to building, with connection-sharing enabled as follows.

% cat ~/.ssh/config
Host *
	ControlMaster auto
	ControlPath ~/.ssh/master-%r@%h:%p

For my project, I specify four computationally-expensive targets, bar1 through bar4, that can be off-shored. In this example, the remote hosts must have /usr/games/factor.

% cat Makefile
.DIST: bar1 bar2 bar3 bar4
all: foo
foo: bar1 bar2 bar3 bar4
	cat bar1 bar2 bar3 bar4 >$@
bar1 bar2 bar3 bar4:
	/usr/games/factor 1844674407370955161 >$@

To build with eight jobs, distributing to hosts foo1 and foo2, execute the following. The -dD simply enables debugging information for distribution.

% make -j4 -dD
Enabling distributed build.
Using build host foo1 (2 jobs).
Using build host foo2 (2 jobs).
Creating staging directory /tmp/make-ctime.hhs.se-1330786818 on foo1
Creating staging directory /tmp/make-ctime.hhs.se-1330786818 on foo2
Creating staging directory /tmp/make-ctime.hhs.se-1330786818 on foo1
Creating staging directory /tmp/make-ctime.hhs.se-1330786818 on foo2
/usr/games/factor 1844674407370955161 >bar3
/usr/games/factor 1844674407370955161 >bar1
/usr/games/factor 1844674407370955161 >bar4
/usr/games/factor 1844674407370955161 >bar2
Retrieving target bar1 from foo1:/tmp/make-ctime.hhs.se-1330786818
Retrieving target bar4 from foo2:/tmp/make-ctime.hhs.se-1330786818
Retrieving target bar3 from foo1:/tmp/make-ctime.hhs.se-1330786818
Retrieving target bar2 from foo2:/tmp/make-ctime.hhs.se-1330786818
cat bar1 bar2 bar3 bar4 >foo

At some point, you should nuke hang-around staging directories on the remote host. Obviously, don't do this during builds.

for f in foo1 foo2; do ssh $f rm -rf /tmp/make-* ; done

Contact

This patch was developed and is maintained at the Stockholm School of Economics, Department of Economics. Contact Kristaps Dzonsons with bug-reports, questions, comments, patches, and so on.