Documentation
¶
Index ¶
- Constants
- Variables
- func BlockedInSets(ip string) (blocked, released bool)
- func FloorChainFor(subnet netip.Prefix) string
- func NodeChainsFor(subnet netip.Prefix) [2]string
- func ResetBlockSetsForTest()
- func ResetFloorsForTest()
- func SetSandboxInterconnect(allow bool)
- func WarnIfLegacyIptables(backend string, logger *slog.Logger)
- type Manager
- func (m *Manager) ApplyEgressPolicy(containerIP string, allowCIDRs, denyCIDRs []string) error
- func (m *Manager) BlockAllEgress(containerIP string) error
- func (m *Manager) BlockAllEgressReport(containerIP string) (bool, error)
- func (m *Manager) BlockAllIngress(containerIP string) error
- func (m *Manager) BridgeSubnet() netip.Prefix
- func (m *Manager) ClearAddress(ip string) error
- func (m *Manager) ClearBlockAllEgress(containerIP string) error
- func (m *Manager) ClearBlockAllIngress(containerIP string) error
- func (m *Manager) ClearEgressPolicy(containerIP string, allowCIDRs, denyCIDRs []string) error
- func (m *Manager) ClearHoldEgress(containerIP string) error
- func (m *Manager) Enabled() bool
- func (m *Manager) EnsureChain() error
- func (m *Manager) HoldEgress(containerIP string) error
- func (m *Manager) LiftBlockAllEgress(containerIP string) error
- func (m *Manager) PrepareBlockSets() error
- func (m *Manager) ReassertBlockSets() (repaired bool, err error)
- func (m *Manager) ReassertChain() error
- func (m *Manager) ReassertFloor() (repaired bool, err error)
- func (m *Manager) ReassertNodeGuard() (repaired bool, err error)
- func (m *Manager) ResetChainLatch()
- func (m *Manager) SetBridgeSubnet(subnet string)
- func (m *Manager) SetFloor(subnet netip.Prefix, cidrs []netip.Prefix) error
- func (m *Manager) SetNodeGuard(subnet netip.Prefix, allow []NodePort) error
- type NodePort
- type RuleBackend
- type RuleInsert
- type RuleRef
Constants ¶
const ( BlockTable = "aerolvm_block" BlockedSet = "blocked" ReleasedSet = "released" )
The block table and its sets.
const ( BackendExec = "exec" BackendNetlink = "netlink" ChainDockerUser = "DOCKER-USER" ChainAerolvmUser = "AEROLVM-USER" // ChainAerolvmFC holds the per-guest-IP rules for Firecracker VMs (egress // Phase 4), apart from the container engines' chains: its FORWARD // accepts are scoped to the TAP subnet. ChainAerolvmFC = "AEROLVM-FC" // ChainAerolvmInput holds per-IP drops for traffic from restricted // sandboxes to the host itself (sandboxd API, SSH gateway, cluster port). // Shared by the dockerd and containerd managers: rules are keyed by // sandbox IP and the two bridges never share an IP. ChainAerolvmInput = "AEROLVM-INPUT" )
Backend names for SB_NETRULES_BACKEND.
const ChainAerolvmFloor = "AEROLVM-FLOOR"
ChainAerolvmFloor names the deny floor (plans/egress-domain-filtering.md §5.10 PC-2): DROPs from a sandbox subnet to each floor address, jumped from the top of the user chain so they sit above every per-sandbox ACCEPT. A chain of its own because the floor is replaced as a whole when the operator file changes, and the rule backends cannot list rules to find the old ones.
Each sandbox subnet gets its own chain, FloorChainFor(subnet). Docker, containerd and Firecracker each floor their own subnet from their own user chain; when they shared this one chain, every SetFloor flushed it, and on a node running more than one of them only the last subnet kept a floor. This name is now only the legacy chain an older daemon filled, emptied at the first floor so its leftover jump is a no-op.
const ChainAerolvmNode = "AEROLVM-NODE"
ChainAerolvmNode is the node guard: sandboxes are default-denied from the node they run on. Every packet a sandbox subnet sends to the node itself (its bridge gateways, its private and any locally bound public address: whatever reaches INPUT) is dropped unless it answers a connection the host opened (sandboxd to the toolbox) or goes to a port on the allowlist the service computes (the egress gateway's DNS and proxy ports, and the few others setup/config-defaults.md lists).
It replaces an enumerated list of ports to drop (the control-port guard): a node also runs services AerolVM does not know about (an EDR agent, a log shipper, a node exporter, a SIEM forwarder), and every one of them was reachable from every default sandbox.
Layout, all in the filter table:
INPUT first rule: -j AEROLVM-NODE (kept first, like the floor's jump)
AEROLVM-NODE 1: conntrack RELATED,ESTABLISHED -j RETURN
then per sandbox subnet: -s <subnet> -j AEROLVM-NODE-<hash>-A|B
AEROLVM-NODE-<hash>-A|B
-s <subnet> [-d <addr>] -p tcp|udp --dport N -j RETURN (the allowlist)
-s <subnet> -j DROP
Its own chain, not rules in AEROLVM-INPUT, so no per-sandbox rule can sit above it: an allowlist RETURN for a CIDR that covers the node's address (AEROLVM-INPUT's inputPolicyReturnSpec) would otherwise skip the guard. RETURN, never ACCEPT, so a host firewall such as ufw still decides what it lets in.
Variables ¶
var ( // ErrNodeGuardDisabled means network rules are off on this node, so // nothing keeps sandboxes off the node's own services. ErrNodeGuardDisabled = errors.New("netrules: network rules are disabled; sandboxes can reach every service on this node") // ErrNodeGuardUnsupported means the rule backend cannot manage chains. ErrNodeGuardUnsupported = errors.New("netrules: backend cannot manage the node guard chains") )
var ErrFloorUnsupported = errors.New("netrules: backend cannot manage the egress floor chain")
ErrFloorUnsupported means the rule backend cannot manage the floor chain.
Functions ¶
func BlockedInSets ¶ added in v0.6.9
BlockedInSets reports what the mirror holds for ip, for tests and diagnostics.
func FloorChainFor ¶ added in v0.6.9
FloorChainFor is the floor chain for one sandbox subnet: the prefix plus eight hex digits of the subnet's hash, 22 characters, inside iptables' 28-character chain-name limit.
func NodeChainsFor ¶ added in v0.6.9
NodeChainsFor is the pair of chains one sandbox subnet's guard alternates between: a new allowlist is built in the one nothing jumps to, then the jump moves, so a change never leaves the subnet unguarded (a flush and refill of one chain would, briefly). 23 characters, inside iptables' 28.
func ResetBlockSetsForTest ¶ added in v0.6.9
func ResetBlockSetsForTest()
ResetBlockSetsForTest forgets the mirror. Test-only seam: the mirror is process-wide, as a node's runtimes share one process and one table.
func ResetFloorsForTest ¶ added in v0.6.9
func ResetFloorsForTest()
ResetFloorsForTest forgets every registered floor. Test-only seam: the registry is process-wide, as a node's runtimes share one process.
func SetSandboxInterconnect ¶ added in v0.6.9
func SetSandboxInterconnect(allow bool)
SetSandboxInterconnect sets whether sandboxes on one node may connect to each other; the daemon sets it from config before the first floor.
func WarnIfLegacyIptables ¶ added in v0.5.6
WarnIfLegacyIptables logs a boot-time warning when the netlink backend is selected on a host whose iptables binary runs in legacy (xtables) mode. The netlink backend writes to the nft tables, which legacy iptables cannot see — the kernel evaluates BOTH rule sets, so flipping SB_NETRULES_BACKEND with live sandboxes strands the previous backend's ACCEPT/DROP rules on recycled container IPs (TODOS.md "netrules backend switch on iptables-legacy hosts"). Best-effort: a missing binary or exec failure stays silent — no signal is not a legacy host.
Types ¶
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
func NewWithBackend ¶ added in v0.5.5
func NewWithBackend(backend RuleBackend) *Manager
NewWithBackend builds an enabled Manager over an injected backend. Test seam only — production wiring goes through New.
func NewWithOptions ¶ added in v0.5.6
NewWithOptions builds a Manager with the chosen RuleBackend. userChain selects the filter chain for per-IP rules; empty defaults to DOCKER-USER. Unknown backend names error so misconfig is loud. Enabled backends are linux-only (see newEnabledManager); other platforms always return a disabled manager.
func (*Manager) ApplyEgressPolicy ¶ added in v0.5.4
ApplyEgressPolicy installs a per-container selective egress policy in DOCKER-USER, scoped by source IP and comment-tagged (see egressPolicyComment). The mode follows the shared allow-wins precedence (plans/ egress-domain-filtering.md D4):
- allowCIDRs only → allowlist: ACCEPT each CIDR, DROP everything else.
- denyCIDRs only → blocklist: DROP each CIDR, leave the rest to Docker's default ACCEPT.
- both → allow wins, then deny, then default ACCEPT: ACCEPTs sit above the DROPs and there is no catch-all.
"allow + deny 0.0.0.0/0" is the portable allowlist spelling (E2B's allowOut + denyOut all): normalizeLists maps it to allow-only here, in the one place every apply and clear goes through, so a clear from the stored row always mirrors what the apply installed.
Re-apply is idempotent: every rule is Exists-checked before Insert, so the start/reconcile reapply paths can call this repeatedly without duplicating.
func (*Manager) BlockAllEgress ¶
BlockAllEgress installs a DROP rule for traffic originating from containerIP. The rule lives in DOCKER-USER, the chain Docker explicitly reserves for operator-defined firewall rules. DOCKER-USER is jumped from FORWARD *before* DOCKER-FORWARD, so our DROP fires before Docker's blanket "iifname docker0 accept" rule that would otherwise short-circuit any rule appended directly to FORWARD. This works on iptables-legacy and on Docker 28+/iptables-nft (which writes through to nftables) alike.
func (*Manager) BlockAllEgressReport ¶ added in v0.5.6
BlockAllEgressReport is BlockAllEgress with the idempotency outcome surfaced: inserted is true only when the rule was genuinely absent and had to be re-installed. Reconcile reapplies unconditionally (the Exists guard below is what makes that safe), so "we called Apply" is a constant rate and useless as a signal — "the rule was missing when we looked" is the actual isolation-drift event worth counting. Disabled manager and empty IP report inserted=false: nothing was installed, so nothing drifted.
func (*Manager) BlockAllIngress ¶ added in v0.1.7
BlockAllIngress installs a DROP rule for traffic destined for containerIP, the mirror of BlockAllEgress on the destination axis. Used by the network quota enforcer when net_bytes_in_limit is crossed. The honest caveat (also documented in plans/network-usage-tracking.md): host-side ingress is counted after the NIC has accepted the packet, so the meter is "what the container would have seen" rather than "bytes spent on the wire." Same chain (DOCKER-USER) and idempotency check pattern as the egress mirror.
func (*Manager) BridgeSubnet ¶ added in v0.6.6
BridgeSubnet returns the subnet set by SetBridgeSubnet (the containerd CNI bridge), or the zero prefix.
func (*Manager) ClearAddress ¶ added in v0.6.9
ClearAddress removes every per-address rule keyed by ip from this manager's user chain and AEROLVM-INPUT. Call it when the address's holder is gone or about to be, never while a sandbox still uses the address. A backend that can't list a chain clears the specs the manager knows without their lists: block-all, the ingress block and the hold.
func (*Manager) ClearBlockAllEgress ¶
ClearBlockAllEgress removes containerIP's block-all for good, when the address is let go or about to be (destroy, a failed create, a stop): on the block sets it is Forget (the delete that pays the grace period); a sandbox that keeps its address and only stops being blocked is lifted instead (LiftBlockAllEgress).
func (*Manager) ClearBlockAllIngress ¶ added in v0.1.7
func (*Manager) ClearEgressPolicy ¶ added in v0.5.4
ClearEgressPolicy removes the rules ApplyEgressPolicy would have installed for the same (containerIP, allowCIDRs, denyCIDRs). The caller passes the policy persisted on the sandbox row so cleanup is exact and comment-scoped — the blanket BlockAllEgress DROP (no comment) is left untouched.
func (*Manager) ClearHoldEgress ¶ added in v0.6.6
ClearHoldEgress removes the hold DROP. Only a successful attach lifts a hold; callers enforce that.
func (*Manager) EnsureChain ¶ added in v0.5.6
EnsureChain bootstraps the filter user chain and its FORWARD jump once per Manager lifetime. Idempotent and latched like EnsureLayer4Ready: concurrent callers single-flight through chainMu; success sets chainReady.
func (*Manager) HoldEgress ¶ added in v0.6.6
HoldEgress installs the hold DROP for containerIP. Idempotent.
func (*Manager) LiftBlockAllEgress ¶ added in v0.6.9
LiftBlockAllEgress lifts containerIP's block-all while the sandbox keeps its address (the gateway attach of a hostname-filtered create, a quota or policy that no longer blocks, a warm slot adopted without one): on the block sets it is one add to @released. The address stays in @blocked until Forget. On the per-address rules it is ClearBlockAllEgress.
func (*Manager) PrepareBlockSets ¶ added in v0.6.9
PrepareBlockSets moves this manager's block-alls into the block sets: it makes sure the table exists, reads it into the process's mirror (the first manager to prepare does), and converts the per-address block rules an older daemon left in this manager's chains (set element first, rule second, so the address is never open). A backend without sets, or a failure here, leaves the manager on the per-address rules, which stay correct, only slower; the error says why. Call once at boot, before the manager serves creates.
func (*Manager) ReassertBlockSets ¶ added in v0.6.9
ReassertBlockSets checks the table against the mirror: a missing table or element is put back (repaired reports it), and a release the kernel holds that this process never made is deleted, as nothing here would ever lift a block for it. A no-op on the per-address rules.
func (*Manager) ReassertChain ¶ added in v0.5.6
ReassertChain re-runs the chain + FORWARD-jump bootstrap WITHOUT the latch, so a caller (e.g. the containerd netns reconcile ticker) can re-assert the jump after a dockerd restart flushes/reorders FORWARD and drops it. Both steps are idempotent; a no-op when disabled or the backend cannot bootstrap.
func (*Manager) ReassertFloor ¶ added in v0.6.9
ReassertFloor puts this manager's floor back if something removed it: the chain, the user chain's jump to it, or any of its DROPs (a host firewall reload, a flush). Cheap when nothing is missing: the jump check and one rule check per floor address, no rewrite, so a floor in place never has the empty moment SetFloor's flush and refill has. repaired reports that the floor was rewritten. A no-op until SetFloor has run.
func (*Manager) ReassertNodeGuard ¶ added in v0.6.9
ReassertNodeGuard puts the guard back if something removed it: the INPUT jump (a host firewall reload that rewrote INPUT, or one that put its own rules above ours), the established-return rule, the subnet's jump, or the drop at the end of its chain (a flush). Cheap when nothing is missing: a handful of rule checks. repaired reports that the guard was rebuilt. A no-op until SetNodeGuard has succeeded once.
func (*Manager) ResetChainLatch ¶ added in v0.5.6
func (m *Manager) ResetChainLatch()
ResetChainLatch clears the bootstrap latch. Test-only seam.
func (*Manager) SetBridgeSubnet ¶ added in v0.5.6
SetBridgeSubnet records the sandbox bridge subnet whose forwarded traffic EnsureChain/ReassertChain must ACCEPT (below any per-IP DROP). Empty = no-op.
func (*Manager) SetFloor ¶ added in v0.6.6
SetFloor replaces the node-wide deny floor for sandboxes on subnet. An empty cidrs list clears the operator's part. The flush and refill are separate steps, so the floor is briefly empty while it is replaced; the egress gateway's own copy of the floor (its deny_floor set) stays in place throughout. A subnet seen for the first time also re-renders every other runtime's floor, so each drops traffic to it.
func (*Manager) SetNodeGuard ¶ added in v0.6.9
SetNodeGuard default-denies sandboxes on subnet from the node itself, except allow. Replacing the allowlist is safe while sandboxes run: the new list goes into the subnet's idle chain, the jump moves to it, then the old chain is emptied. In between, a packet passes both chains, so it gets through only if both allow it (fail closed, never open). Idempotent: the service calls it at boot and whenever the allowlist changes.
type NodePort ¶ added in v0.6.9
type NodePort struct {
// Proto is "tcp" or "udp".
Proto string
// Port is the destination port, 1-65535.
Port uint16
// Dst narrows the entry to one node address (a resolver the sandboxes
// are pointed at); the zero value allows the port on every address.
Dst netip.Addr
}
NodePort is one thing sandboxes may still reach on their own node.
type RuleBackend ¶ added in v0.5.5
type RuleBackend interface {
Exists(table, chain string, rulespec ...string) (bool, error)
Insert(table, chain string, pos int, rulespec ...string) error
Delete(table, chain string, rulespec ...string) error
}
RuleBackend is the subset of iptables operations the Manager drives. Production always wraps *go-iptables' IPTables; tests substitute an in-memory backend so rule-state semantics (which rules survive an adopt, a clear, a reapply) are assertable without root or a linux host.
func NewNetlinkBackend ¶ added in v0.5.6
func NewNetlinkBackend() (RuleBackend, error)
NewNetlinkBackend wires the per-operation nftables connection factory. Callers must be on linux; non-linux builds use the stub that always errors.
type RuleInsert ¶ added in v0.6.9
RuleInsert is a rule to ensure at an iptables-style position: 1 is the top, a position past the last rule appends (the RuleBackend.Insert contract).