Skip to main content
Agent sessions run outside your cluster, on agent hosts: cloud instances that run each session in its own virtual machine. The chart’s host controller runs in your cluster and launches, tracks and replaces them. On AWS, agent hosts are EC2 instances started from Tembo’s agent host image, which Tembo shares with your account. They need nested virtualization, and they keep VM disks, snapshots and saved memory on the instance’s local NVMe.

​
Classes and pools

You describe agent hosts in your Helm values, with two kinds of resources the chart creates. On AWS, the Terraform module’s Helm values create one of each, named default.
  • A host class says what to launch: the image, instance types, subnets, security groups, instance profile and root volume.
  • A pool says how many to run: minHosts, maxHosts, and minIdleMemory, the free memory to keep for new sessions.
The controller creates a host resource for each instance it launches. List them with:

​
Lifecycle

  1. Launch. The controller starts an instance with the class’s settings and tags it as its own.
  2. Configure. The host reads its configuration from a Kubernetes Secret through the cluster’s API, with a short-lived token the controller gives it at launch. This is the same configuration your Helm values give the platform.
  3. Register. The host registers with your Tembo installation and becomes Ready, usually within a few minutes. The controller replaces a host that doesn’t register within 15 minutes.
  4. Run sessions. The host claims sessions and runs each one in its own virtual machine.
Hosts read their image and configuration only when they start. To roll out a new agent host image or changed configuration, replace the hosts (see Apply changes).

​
Scaling

The controller sizes each pool by memory: the memory its sessions use, plus the memory waiting sessions need, plus minIdleMemory. When that is more than the pool’s hosts provide, it launches another host, up to maxHosts. It never goes below minHosts. Removing idle hosts is coming. Until then, hosts keep running until you remove them.

​
Sessions on a host

  • Stopping. When a session goes idle, its virtual machine stops. The host saves a snapshot of its disk and memory, and uploads the disk to object storage.
  • Resuming. On the same host, a session resumes from its saved memory, with its processes still running. On another host, for example after its host was replaced, it resumes from the disk in object storage, without the processes that were running.
  • Project images. A project build creates an image of the project’s environment and stores it in object storage. Hosts download it on first use and keep a copy, so later sessions start in seconds.
  • Warm VMs. For projects with Warm pool on, hosts keep virtual machines booted from the project’s image, ready for the next session.
Saved memory only restores on a host with the same CPU, so keep all hosts on one processor generation (see Instance types).

​
Need Help?

If you run into any issues, contact support@tembo.io.