Skip to main content

Cluster Node Management

QuickStack runs on a k3s Kubernetes cluster that can be expanded by adding worker nodes. This guide covers how to add nodes to your cluster, monitor node health, and manage node scheduling.

Overview​

A QuickStack cluster consists of:

  • Master Node: The initial server where QuickStack is installed. Runs the control plane and can also run workloads.
  • Worker Nodes: Additional servers that join the cluster to provide more compute capacity.

Viewing Cluster Nodes​

  1. Navigate to QuickStack Settings.
  2. Click on the Cluster tab.
  3. View all nodes and their status.
Cluster Nodes Overview

Node Information​

For each node, you can see:

  • Node Name: The hostname or identifier of the node
  • Status Indicators:
    • 🟢 CPU: Green if CPU pressure is normal
    • 🟢 RAM: Green if memory pressure is normal
    • 🟢 Disk: Green if disk pressure is normal
  • Schedulable: Whether new workloads can be scheduled on this node

Adding a Worker Node​

Prerequisites​

Before adding a worker node, ensure:

  • The new server is running a supported Linux distribution (Ubuntu, Debian, etc.)
  • The server has network connectivity to the master node
  • Port 6443 (Kubernetes API) is accessible from the new server
  • You have SSH access to the new server

Step 1: Get the Join Command​

  1. Navigate to QuickStack Settings → Cluster.
  2. Click the Add Cluster Node button.
  3. Copy the displayed command.
Add Cluster Node Dialog

The command looks like this:

curl -sfL https://get.quickstack.dev/setup-worker.sh | K3S_URL=https://MASTER_IP:6443 JOIN_TOKEN=K10... sh -

Step 2: Configure the Command​

Replace MASTER_IP with:

  • The internal IP of your master node (for nodes in the same network)
  • The public IP of your master node (for nodes across the internet)

Step 3: Run on the New Server​

  1. SSH into your new server.
  2. Run the modified command as root or with sudo.
  3. Wait for the installation to complete.
ssh root@new-worker-server
curl -sfL https://get.quickstack.dev/setup-worker.sh | K3S_URL=https://192.168.1.100:6443 JOIN_TOKEN=K10... sh -

Step 4: Verify the Node​

  1. Return to QuickStack Settings → Cluster.
  2. The new node should appear in the list within a few minutes.
  3. Verify all status indicators are green.
Node Token

If the join token shown in the dialog is empty or invalid, you can retrieve it manually from your master node:

sudo cat /var/lib/rancher/k3s/server/node-token

Managing Node Scheduling​

You can control whether workloads are scheduled on specific nodes.

Deactivating a Node​

To prevent new workloads from being scheduled on a node (cordon):

  1. Find the node in the cluster overview.
  2. Click the node to view options.
  3. Click Deactivate Node or similar option.
  4. Confirm the action.

When deactivated:

  • Existing workloads continue running
  • No new pods will be scheduled on this node
  • The node shows Schedulable: No

Reactivating a Node​

To allow scheduling on a previously deactivated node (uncordon):

  1. Find the deactivated node.
  2. Click the node to view options.
  3. Click Activate Node.
  4. Confirm the action.
Node Deactivation

Deactivating a node does not automatically move existing workloads. To move workloads off a node, you need to:

  1. Deactivate the node (cordon)
  2. Drain the node or redeploy applications

Node Health Monitoring​

Understanding Status Indicators​

IndicatorGreenRed
CPUNormal CPU pressureHigh CPU pressure / PID exhaustion
RAMNormal memory pressureMemory pressure detected
DiskSufficient disk spaceDisk pressure detected

Troubleshooting Node Issues​

Red CPU Indicator:

  • Too many processes running
  • Consider adding more nodes or optimizing workloads

Red RAM Indicator:

  • Memory pressure detected
  • Review container memory limits
  • Consider adding more memory or nodes

Red Disk Indicator:

  • Disk space running low
  • Clean up unused images and containers (see Maintenance docs)
  • Consider expanding disk or adding nodes

Removing a Worker Node​

To remove a worker node from the cluster:

Step 1: Drain the Node​

First, move all workloads off the node:

# On the master node
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Step 2: Delete the Node​

# On the master node
kubectl delete node <node-name>

Step 3: Uninstall k3s (on the worker)​

# On the worker node
/usr/local/bin/k3s-agent-uninstall.sh

Best Practices​

Cluster Planning
  • Odd number of master nodes: For high availability, use 1, 3, or 5 master nodes
  • Consistent hardware: Similar node specifications make resource planning easier
  • Network proximity: Nodes in the same datacenter/region reduce latency
  • Resource headroom: Keep 20-30% capacity free for failover scenarios
Multi-Node Considerations
  • Storage: Use distributed storage (Longhorn) for volumes across nodes
  • Networking: Ensure all nodes can communicate on required ports
  • Firewalls: Configure firewall rules for k3s communication