A Prometheus alert fires. You trace the cause on a dashboard. You roll back. For many teams, this is the only moment they ever get to test whether their system is actually resilient — after a real production incident has already happened. Chaos engineering flips this structure around. Instead of waiting for failure to strike, it deliberately and safely injects failure so you can confirm, ahead of time, whether your system can truly withstand it. Lightweight Kubernetes distributions like K3s make this kind of verification dramatically cheaper to run. This article explains how to get started with chaos engineering on K3s using the CNCF-hosted open-source tool LitmusChaos, and walks through the judgment call every team eventually faces: what should you break first?
1. Why Teams Stay Stuck in "We Only Find Out After It Breaks"
Kubernetes operations teams have invested heavily in monitoring stacks like Prometheus, Grafana, and OpenTelemetry. But every one of these tools is designed to detect problems after they've already happened. A metric crossing a threshold, an elevated error rate, degraded latency — all of these are reactions to failures that have already occurred.
Meanwhile, questions like "will the service keep running if this pod goes down?" or "what happens if this node experiences a network partition?" can only be answered by actually triggering the failure. The official Kubernetes documentation refers to forced pod evictions caused by hardware failure or resource exhaustion as "involuntary disruptions," and explicitly states that these cannot be prevented even by a PodDisruptionBudget. In other words, systems must be designed with the assumption that unintended failures will happen eventually.
Once you accept that premise, it becomes clear that strengthening monitoring alone isn't enough. Monitoring is a mechanism for noticing that something broke — not a mechanism for confirming that your system can survive breaking. Conflating the two means every production incident becomes the first time you discover your system's actual weak points. Chaos engineering exists to run that verification proactively, in a planned and safely bounded way, instead of waiting for it to happen on its own.
A K3s cluster running on Kubo comes with Prometheus and Grafana built in by default, so the foundation for tracking the impact of the chaos experiments discussed below is already in place. With Kubo's out-of-the-box convenience, you can treat that monitoring foundation as a given and move straight to the next step: deliberate fault injection.
2. What Chaos Engineering Actually Is, and What You Can Break on K3s
The term "chaos engineering" was popularized by Netflix as part of validating its own distributed systems, and today the Principles of Chaos Engineering codifies the discipline into a formal set of principles. The key points boil down to four:
- Define a steady-state hypothesis: define "normal" using measurable indicators such as throughput, error rate, and latency
- Introduce real-world events as variables: simulate failures that can actually occur, such as server outages, network latency, or resource exhaustion
- Experiment in as close to a production environment as possible: staging and production differ in load patterns and traffic characteristics, so the closer the conditions match, the more valuable the verification
- Minimize the blast radius: localize the impact of the experiment and design it so it can be halted at any time
Within the Kubernetes ecosystem, several tools implement these principles as CRDs (Custom Resource Definitions). The leading one, and the focus of this article, is LitmusChaos, with Chaos Mesh as another major option. Chaos Mesh, a CNCF Incubating project, is known for its label-based target selection and the ability to build experiments directly from a dashboard. Both run without issue on K3s, but this article focuses on LitmusChaos for its rich library of experiment templates available through ChaosHub.
The K3s distribution that Kubo runs on is a CNCF-conformant Kubernetes distribution, meaning CNCF-ecosystem tools like LitmusChaos and Chaos Mesh work on it without modification. Not having to relearn a proprietary API on the managed service side aligns well with Kubo's design philosophy of prioritizing standards conformance.
Representative failures you can inject in a K3s environment include forced pod termination (container-kill), CPU/memory load injection (pod-cpu-hog / pod-memory-hog), and network latency injection (pod-network-latency). All of these are standard experiments introduced in LitmusChaos's practical guide, and they split into experiments that require no special permissions and higher-privilege experiments that need SecurityContext adjustments. It's safest to start with the permission-free experiments.
3. The Basic Structure of LitmusChaos and the Flow of a Pod Kill Experiment
LitmusChaos is built around three CRDs. According to the official repository, each has a clearly separated role.
- ChaosExperiment: a resource bundling the configuration parameters for a specific failure. It defines the templates and permission information needed to run it
- ChaosEngine: a resource that binds a specific failure to an application, node, or infrastructure component in Kubernetes. It's also where you configure target labels and verification conditions (probes)
- ChaosResult: a resource that records the outcome of the experiment and whether the system recovered from the injected failure
Installation, as described in the official documentation, takes just a few commands via Helm. K3s is a lightweight distribution that only needs a modern kernel and cgroup mounts to run (see the official K3s documentation), so the cost of spinning up an entire dedicated K3s cluster for verification purposes is very low. That's a real advantage: it turns chaos experiments from something you can "only try in production" into something you can "try casually, as often as you like."
A minimal Pod Kill experiment can be designed with the following flow:
- Select the
container-killexperiment template from ChaosHub - In the ChaosEngine, specify the label of the target application (e.g.
app=my-service) - Set the blast radius (the proportion of pods affected simultaneously) to a small value
- Run the experiment, then use ChaosResult to verify whether the steady-state hypothesis (error rate and latency staying within acceptable bounds) held
- If unexpected impact occurs, halt the experiment immediately and investigate the cause
Using Kubo Cloud's Captain UI, you can visualize pod restart activity and resource usage throughout the experiment instead of treating it as a black box, making it much easier to judge the results.
4. Which Workloads Are Safe to Break First, and Which Demand Caution
The judgment call teams get wrong most often in chaos engineering is choosing what to target. Break a critical production workload right out of the gate, and what you get isn't verification — it's a genuine incident. Here's a rough guide for expanding your target scope in stages.
Safe to Break First
- Stateless services running multiple replicas: Deployments with 3+ replicas where requests aren't pinned to a specific pod
- Internal batch or asynchronous jobs without strict SLAs: easy to re-run on failure, with no direct customer impact
- New versions in canary release: since only a fraction of traffic reaches them by design, the blast radius is naturally limited
Handle With Care
- Stateful workloads backed by PVCs: databases, caches, and similar workloads where a pod restart can affect data integrity
- Services running a single replica: with no redundancy, an experiment translates directly into downtime
- Components tightly coupled with network policies or ingress control: fault injection here tends to spread wider than expected
The core idea behind this roadmap is to start with targets where "breaking them still yields learning" while keeping the blast radius small, and gradually widen the scope as your team gains familiarity with the LitmusChaos experiment process. Even when choosing a tool like Chaos Mesh, which enables RBAC by default, permission design itself should be considered together with initial target selection.
If you can replicate environments cheaply on K3s, it becomes realistic to rehearse on a production-equivalent staging environment before running experiments against production, even for the "handle with care" tier of workloads. On EKS or AKS, the cost of replicating environments tends to become a psychological barrier, but with Kubo you can flexibly scale verification environments up and down by leveraging K3s's lightweight footprint.
Conclusion
Monitoring and alerting tell you that something broke. But whether your system can truly withstand failure is something you can only know by actually breaking it. Chaos engineering is the practice of running that verification proactively — "before it happens" rather than "after it happens" — and with a CNCF project like LitmusChaos, you can implement it as a Kubernetes-native, declarative resource.
What matters most is not trying to break everything at once. Start with stateless, redundant workloads, keep the blast radius small, and design a process that lets both your team and your system grow accustomed to experimentation. K3s's lightweight nature is a practical advantage that lowers the cost of running this verification cycle repeatedly.
If setting up a resilience-testing environment is itself the obstacle standing in your way, start by reaching out via our contact form to discuss a Kubo setup that lets you replicate production-equivalent environments on K3s at low cost. It's a first step toward operations that don't rely on monitoring alone — operations that verify resilience by actually breaking things and checking the results.