1. You Adopted SAST and DAST — So Why Do Incidents Still Reach Production?
Investment in CI/CD pipeline security has clearly grown over the past few years. Teams run SAST, scan container images, and check dependencies for known vulnerabilities — yet production incidents still happen, and not rarely.
The root cause is usually not the tools themselves. The design decision of which check goes where in the pipeline, and at what stage, is often never made explicit. Palo Alto Networks defines CI/CD security as "integrating security throughout the software delivery lifecycle, rather than treating it as an isolated stage" (Palo Alto Networks Cyberpedia). In other words, simply bolting on more tools one at a time does not, by itself, make security "integrated."
The more tools you add, the easier it becomes to assume "surely some stage is checking this" — while a real gap that nobody is actually watching quietly opens up. This article reframes CI/CD pipeline security not as addition, but as placement design, introducing a four-layer model and walking through concrete implementation patterns for K3s/Kubernetes environments.
As pipelines grow more complex, the operational burden of continuously maintaining this design decision grows too. We'll return to how to reduce that burden — together with how Kubo fits in — in the implementation section later in this piece.
2. Design a DevSecOps Pipeline Around Four Layers
A useful framework for designing CI/CD pipeline security is the four-layer model. Microsoft's engineering blog describes governance across four layers for DevSecOps design on AKS (Microsoft Tech Community).
What each layer does
- Pre-deployment (CI): Fast feedback and guardrails for developers. Catches problems at commit time and before merge.
- Admission Control (governance gate): The layer that forcibly blocks anything from entering the cluster. This layer is assumed to be "unskippable."
- Runtime Protection: Detects and responds to anything that slipped through. Not a substitute for the gate — the last line of defense.
- Continuous Compliance: Drift detection and auditing. Continuously verifies that configuration hasn't quietly changed after the fact.
What this article wants to emphasize in particular is the asymmetry the same blog post points out: "CI scans can be skipped, but Admission Control cannot be skipped unless the cluster's configuration itself is broken." In other words, the same check can have completely different real-world effectiveness depending on which layer it sits in. A design that relies on CI alone can collapse from a single branch-protection misconfiguration or an emergency bypass.
3. The Pre-deployment Layer: What Should Be Caught "Early"?
The purpose of the Pre-deployment layer is "fast feedback for developers," so the checks placed here need to prioritize execution speed and developer experience above all else.
OWASP's CI/CD Security Cheat Sheet is explicit that secrets should "never be hardcoded in the repository or CI/CD configuration," and recommends using an external secrets management solution along with mechanisms that detect secrets before they're committed (OWASP Cheat Sheet Series). The same guidance applies to the build execution environment itself — it should run on "properly isolated nodes," with a default-deny, least-privilege design recommended across pipeline secrets, cross-pipeline access, and OS-level permissions.
DAST (Dynamic Application Security Testing), on the other hand, targets a running application by nature, so it cannot execute at commit time. Snyk's research found that organizations running SAST and DAST together detect 40–60% more vulnerabilities than those relying on a single method alone (Snyk). But DAST is fundamentally meant to run after deployment to a staging environment — running a full DAST pass on every single commit would badly damage pipeline speed.
In short, the Pre-deployment layer isn't monolithic. The core of the design is clearly separating "what should return results instantly at commit time" (secrets scanning, dependency version pinning) from "what should be verified in a post-deployment environment" (DAST). Conflate the two, and you get one of two failure modes: developers blocked from merging while waiting for DAST to finish, or secrets scanning skipped entirely and leaks going unnoticed.
4. Admission Control Is the One Layer You Cannot Skip
No matter how thorough the Pre-deployment layer's checks are, they only work if developers actually run them. Misconfigured CI settings, pipeline bypasses during emergency patches, PRs from forked repositories — real-world paths around CI genuinely exist.
Against this, the official Kubernetes documentation defines an Admission Controller as "code that intercepts requests to the Kubernetes API server after authentication and authorization are complete, but before the object is persisted" (Kubernetes documentation). Processing happens across two phases — Mutating and then Validating — and if either phase rejects the request, the entire request is rejected on the spot. In other words, Admission Control has enforcement power on a layer independent of CI: outside the application, inside the cluster.
Two policy engines are widely used to implement this layer: Open Policy Agent (OPA) and Kyverno. According to OPA's official documentation, OPA runs as an Admission Controller for the Kubernetes API server and evaluates policy using the request type, the operation, and the full object as input (Open Policy Agent documentation). Kyverno, meanwhile, lets you write policies in the same YAML format as Kubernetes manifests, so there's no new policy language to learn (Kyverno documentation). CNCF's official blog also describes how Kyverno's four functions — Validate, Mutate, Generate, and Cleanup — help teams move away from processes that depended on manual approval gates and toward automated DevSecOps flows (CNCF blog).
If a management platform like Rancher gives you a single, unified view of policy enforcement across multiple clusters, the operational burden of tracking "what got rejected, and on which cluster" drops considerably. That visibility is one factor worth weighing when choosing a K3s-based management platform such as Kubo Cloud.
5. Implementation Patterns for CI/CD Pipelines on K3s / Kubo
If Admission Control is designed in as "the one reliably enforced layer" rather than "the last resort," the ideal is not to discover policy violations there for the first time, but to pre-validate the same policies before merge. CLI tools like Conftest (for OPA) or kyverno jp (for Kyverno) let you evaluate the exact same policies that Admission Control enforces, right inside the CI pipeline — at PR time.
Integration with a GitOps pipeline is another key design element. Argo CD's official documentation lays out the GitOps principle that "changes to the desired configuration are pushed to Git first, and the cluster state is then synced to match," and describes CI pipeline integration in three stages: "building and pushing the image to a registry," "updating manifests and committing to the configuration repository," and "triggering a sync via the CLI when needed" (Argo CD documentation). Separating the application's source repository from its manifest repository is also recommended, since it means CI has no direct write path into the cluster.
Signing container images is another important piece of protecting the supply chain. Sigstore's cosign supports keyless signing using short-lived keys backed by OIDC authentication, dramatically reducing the operational burden of key management (Sigstore documentation). Design Admission Control to reject unsigned images, and the Pre-deployment layer's build step and the Admission Control layer's enforcement become one continuous, connected line.
On K3s-based Kubo, ArgoCD/Flux integration comes built in, so there's no need to build this GitOps integration from scratch. For industries like finance or healthcare, where full supply-chain governance and audit trails are required, an option like Kubo On-Premise — which lets you run the same four-layer model entirely within an air-gapped, closed network — is also worth considering.
6. Summary
CI/CD pipeline security incidents usually stem not from a shortage of tools, but from a shortage of design. Rather than "adding" individual tools like SAST, DAST, and image scanning one at a time, deciding upfront what belongs in which of the four layers — Pre-deployment, Admission Control, Runtime Protection, and Continuous Compliance — is the shortest path to avoiding both gaps and redundancy.
Admission Control in particular is the one layer that, unlike CI, has no room to be skipped — which is exactly why placing policy enforcement there via OPA or Kyverno is the crux of the design. A good starting point is auditing your own pipeline and asking: "which layer is this check actually in right now?" and "is it truly in a layer that can't be skipped?"
You no longer need to put up with the high cost, complexity, and vendor lock-in of EKS or AKS. With K3s-based Kubo, you get the full power of Kubernetes and the same four-layer security design, at dramatically better cost efficiency. If you're rethinking your pipeline's security design, it's a good moment to rethink the operational platform underneath it too — feel free to reach out via our contact page.