Namespaced workers and the overlay store

On Linux a worker can run in its own user namespace, with an OverlayFS mounted at /nix/store: the host store as the lower layer, a scratch directory as the upper layer. Three things follow from that.

  • A build sees every path the host store already has, and nothing is copied in.

  • The host store cannot change, because the mount is in a private mount namespace. Every other process on the machine keeps the store it had.

  • The worker owns the Nix settings of that store, including sandbox and sandbox-paths. Against the host store the daemon owns them, and a client cannot relax them.

The last point is the reason this exists. To build with an impure path in the sandbox, you previously had to copy the whole closure into a temporary store. A local overlay store gives you the host's paths and your own sandbox.

Results do not reach the host store on their own. Copy them when you want them, with an ordinary copy_closure to a daemon store.

Using it

import tempfile
import nanopynix

support = nanopynix.probe_namespace_support()
if not support:
    raise SystemExit(f"not available here: {support.reason}")

with tempfile.TemporaryDirectory() as root:
    spec = nanopynix.OverlayNamespace.under(root)
    async with (
        nanopynix.rpc.Session(namespace=spec) as session,
        session.store() as overlay,
        session.store("daemon") as host,
    ):
        # ... build in `overlay` ...
        await overlay.copy_closure([path], host, check_sigs=False)

Session makes the overlay its default store, so session.store() opens it. It also turns on the local-overlay-store experimental feature and sets the two settings the store cannot work without. Name a store explicitly, as session.store("daemon") does above, to reach past the overlay.

Use daemon by name for the host store, never auto. The worker is root inside its own user namespace, so auto resolves to a local store at /nix/store -- which is the overlay mount, not the host store.

From the command line

$ pynix build -f ./default.nix --namespaced
$ pynix build -f ./default.nix --namespaced --sandbox-path /ccache=~/.ccache
$ pynix build -f ./default.nix --overlay-dir ~/.cache/pynix/overlay

--namespaced builds in a throwaway overlay and copies the outputs into the host store when the build succeeds. --no-copy-back keeps them in the namespace, where they disappear with the worker. --overlay-dir keeps the upper layer, so a later build reuses what an earlier one produced.

Requirements

  • Linux, with unprivileged user namespaces enabled.

  • A filesystem for the layers that supports user extended attributes. An unprivileged mount cannot write the trusted.overlay.* attributes OverlayFS uses by default, so the mount asks for userxattr instead.

  • A trusted Nix user, for the copy back. The daemon refuses unsigned paths from an untrusted one.

probe_namespace_support() answers all of the first two at once. It runs the real sequence in a throwaway process rather than reading capability flags, because a host can allow user namespaces and still fail to mount. Session calls it before it starts a namespaced worker, so a host that cannot do this gives you the reason rather than a worker that died.

Why it happens at worker start

unshare(CLONE_NEWUSER) fails with EINVAL in a process that has more than one thread, and the worker has an event loop and a Nix executor thread by the time it can answer an RPC. So there is no "enter a namespace now" call. A fork keeps only the calling thread, which makes the forkserver child single-threaded, and the namespace is entered there -- inside worker_service_factory, the last point before the worker starts a thread of its own.

This makes a namespace a property of the worker, fixed when the session is created. A running worker cannot move into one.

API

class nanopynix.OverlayNamespace(upper_dir, work_dir, state_dir, log_dir, lower_store='daemon', lower_dir='/nix/store')[source]

Where the layers of one worker's overlay store live.

Every field is a plain str rather than a Path because this object is pickled through the forkserver to reach the worker.

work_dir must be on the same filesystem as upper_dir, and must not be inside it. OverlayFS requires both.

Parameters:

  • upper_dir (str)

  • work_dir (str)

  • state_dir (str)

  • log_dir (str)

  • lower_store (str)

  • lower_dir (str)

lower_store: str = 'daemon'

Store URI for the lower store. It supplies the metadata of the paths that lower_dir supplies as bytes, so the two must name one store.

lower_dir: str = '/nix/store'

Directory that OverlayFS reads the lower layer from. It defaults to the host store, which is what lower_store defaults to as well.

Nix compares the two, so they move together. local-overlay-store.cc:66 reads /proc/self/mounts, finds the overlay at the store directory, and checks that the lowerdir of that mount equals the realStoreDir of lower_store. A pair that disagrees gives overlay filesystem /nix/store mounted incorrectly.

A caller that wants a lower layer of its own gives both: a store URI such as local://?root=<root>, whose realStoreDir is <root>/nix/store, and that same directory here. Issue #208 is why the field exists, and it names what a lower layer of the host costs.

classmethod under(root, *, lower_store='daemon', lower_dir='/nix/store')[source]

Lay the four directories out under one root.

Parameters:

  • root (str | PathLike[str])

  • lower_store (str)

  • lower_dir (str)

Return type:

OverlayNamespace

store_config()[source]

Return the typed store configuration this layout names.

work_dir is absent on purpose. OverlayFS needs it, and Nix does not: it is a requirement of the mount, which is this class's concern, not a setting of the store.

Return type:

LocalOverlay

store_uri()[source]

Return the local-overlay:// URI that names this store.

Rendered by nanopynix.stores, so this URI is spelled and escaped exactly like every other store URI the library produces, and Nix's own parser is what validates it.

Return type:

str

required_settings()[source]

Return the Nix settings this store cannot work without.

build-users-group is the one that is not obvious. As the only mapped user, the worker is root inside its namespace, so Nix tries to give the store directory to the build-users group and gets EINVAL, because that group has no mapping here. An empty value tells Nix to build as itself. Nix's own tests for this store type set the same two values.

Return type:

dict[str, str]

class nanopynix.NamespaceSupport(supported, reason='')[source]

What probe_namespace_support() found on this host.

Parameters:

  • supported (bool)

  • reason (str)

reason: str = ''

Empty when supported. Otherwise one sentence that names the step that failed, ready to show to a user.

nanopynix.probe_namespace_support(root=None)[source]

Report whether this host can run an overlay-store worker.

Give root the directory the real upper layer will live under, so the probe tests the same filesystem. Without it the probe uses a temporary directory, which can give a different answer: OverlayFS needs the upper layer to support user extended attributes, and not every filesystem does.

This runs the actual sequence in a throwaway forkserver child rather than reading capability flags. A host can have user namespaces enabled and still fail to mount, so only the real calls give a reliable answer. The child's namespace dies with it, so nothing on the host changes.

Call it before starting a namespaced worker. Without it the worker fails inside the forkserver child, and the caller sees a dead worker instead of the reason.

Parameters:

root (str | PathLike[str] | None)

Return type:

NamespaceSupport

nanopynix.enter_overlay_namespace(spec, *, mount_at='/nix/store')[source]

Enter a private user and mount namespace, and mount the overlay store.

Call this on the worker's only thread, before the worker starts any other thread, and before Nix opens a store. It returns with /nix/store replaced, for this process and its children alone.

Parameters:
Return type:

None