sled

Sled - System Loader for Ephemeral Devices

Sled is a system software loader designed to run in the u-root initramfs. Sled is designed to support loading systems onto devices that are ephemeral, generally with a limited lifetime, and the need to reprovision the device often. The sled software consists of a client (sledc), server (sledd), API (sledapi), and controller (sledctl). The goal of sled is to allow large environments of dynamic devices to be quickly provisioned based on a centralized controller.

sledc

The primary purpose of sledc is to take a set of images stored in the sled database accessable to sledapi to be written onto the target device where sledc is running.

The sled client supports the following functionalities:

• device wiping (clean state)
• device writing (filesystem, kernel, initramfs)
• device loading (via kexec - into new kernel, initramfs, filesystem)
• daemon mode (wait as a daemon for new updates)

The process for a client (sledc) imaging itself is as follows:

1. client boots up (cmds/init/*/init/main.go) a. client loads u-root and subsequently configures base operating system b. client DHCPs for an IP address (ipv4), ipv6 untested. It is assumed that the dhcp lease will provide all necessary options. Currently sled supports dhcp options: 5,6,15,66,67,119. b. client forks sledc into the background c. client presents default shell to user in foreground

2. sledc process then begins by sending a sledd gRPC request out on all interfaces to the destination sled-server. This forces sled-server to be resolved, and sledc uses whichever interface can resolve sled-server as the primary interface.

3. sledc begins generating lldp messages over that interface.

4. sledc requests from sledd (forward to api) the command set for the mac address of the primary interface.

5. sledd responds to client with set of commands to carry out in the following order: wipe, write, kexec, daemon.

6. a daemon request is a special command which tells the client to go into a daemon mode to wait for further commands. This prevents future wait times of bootstraping into u-root and sledc. a. Daemon mode supports health checks, and requests for what the state, and status of client commands. b. Status updates are stored in a key-value store bitcask which is accessable either on the client with the daemonctl tool or through the api and sledctl.

Note: Daemon mode with prefix mac address is currently not supported.

sledd

Sledd is both the middleware and the storage software. Sledd forwards messages from sledc and sledapi to one another, while also providing the raw storage for sledc images.

Images are stored at the /var/img prefix on the sledd server.

More info on the set of commands (CommandSet): CommandSet

• Wipe
• Write
• Kexec

Example CommandSet:

		cs := &sled.CommandSet{
			Wipe: &sled.Wipe{
				Device: "sda",
			},
			Write: &sled.Write{
				ImageName:  "ubuntu18.04", // as stored in etcd
				Device:     "sda",
				KernelName: "linux-4.17", // as stored in etcd
				InitrdName: "initrd-4.17", // as stored in etcd
			},
			Kexec: &sled.Kexec{
				Append: fmt.Sprintf("root=/dev/sda1 rootfstype=ext4 rw"),
				Kernel: "/tmp/kernel",
				Initrd: "/tmp/initrd",
			},
		}

Note that the kernel and initrd strings are hard-coded on sledc to write to /tmp/kernel and /tmp/initrd.

sledapi

The sledapi is the main interface with the datastore. Sled currently only supports etcd as a datastore. The api is also the interface through which users either through the api or through the command line controller access and modify sled.

sledctl

The sled controller allows users through the command line to directly interface with sledapi. The sledctl is designed using cobra

Building a custom sledc

git submodule update --init --recursive

The base of sled is built off of u-root. If you would like to add additional linux commands you can do so in the Makefile.

If you would like to modify the linux kernel version, you can do so with the build kernel script. Many auxiliary scripts can be found in the utils directory. Currently we pull linux 4.19.20.

If you would like to add new modules, you can use the build_kernel.sh above, but modify the kernel configuration files in kconfig before building the new kernel.

For specific environments the cmds/init/*/init/main.go is a good place to start, which defines the init script.

If you have any questions, please feel free to ask or put in an issue.

Known bug when compiling with gcc-9 on ubuntu.

Testing

We currently have a minimal testing environment which supports unit and integration tests. These tests have minimal coverage. Additional information on integration tests can be found in the test/integration directory.

Currently, we use the raven framework for integration tests.

Comments / Complaints / Suggestions

Please create a github issue or PR request!