Kubernetes networking is flat by default. Every pod can reach every other pod on any port — and beyond the cluster, kube-dns, the API server, and cloud metadata endpoints are all reachable unless something says otherwise. A compromised public-facing pod isn't contained to its own workload; it's a beachhead with a network view of everything.

We saw this directly during a review: a frontend pod with no business touching the data layer could open a connection to the database port, and nothing in the network path objected. The same pod could reach the cloud metadata endpoint — the classic path to credential theft in a hijacked container. None of this violated any rule, because there were no rules. The policy-as-code gates we'd built controlled what could deploy; they said nothing about who could talk to whom once it was running.

Why lockdown-by-default wasn't enough

The instinctive fix — write NetworkPolicies that allow only what we could think of, apply them everywhere — failed the first time we tried it, and failed loudly. DNS resolution broke the moment kube-dns traffic (UDP/TCP port 53) wasn't in the allow-list. Health checks started failing. Legitimate inter-service calls dropped, and the pages came in at 2 AM. Within days, teams were asking for blanket exceptions, and some policies were quietly removed to stop the bleeding — which left us worse off than before: a network policy story on paper, and a flat network in practice.

The deeper problem was that we were writing rules from guesses about traffic instead of from evidence of it. Nobody had a complete, current map of which service talks to which — and even if we had one, it would have gone stale the next sprint. There was also an uncomfortable discovery along the way: NetworkPolicies are only enforced if your CNI actually implements them. Apply a beautiful policy set on a CNI that silently ignores it, and you have the worst outcome — a false sense of containment.

The solution: default-deny, rolled out in stages

We went back at it with a staged approach: default-deny ingress and egress on every namespace, but rolled out one namespace at a time, with an observation phase before enforcement.

The rollout ran in four steps:

• Verify the CNI enforces. Before writing a single policy, we confirmed our CNI (Calico in our case) actually enforces NetworkPolicy — a quick test policy against a scratch namespace proved it. A policy that isn't enforced is a liability disguised as a control.
• Observe first, enforce second. We deployed the default-deny policies in audit/log mode first — Calico flow logs gave us a week of real traffic per namespace. That log, not our guesses, became the source of truth for the allow-lists.
• Build allow-lists from observed flows. DNS to kube-dns (port 53) was carved out everywhere on day one. Then, per workload: ingress only from the ingress controller and the specific callers the logs showed; egress only to the dependencies the logs showed — the database port, the handful of external APIs, nothing else.
• Enforce namespace by namespace. We started with the least sensitive namespaces and moved up. Each namespace got its own rollout, its own monitoring window, and a fast rollback path — so a misconfigured rule was a contained incident, never a cluster-wide outage.

None of this was one big YAML drop. The policy set is versioned alongside the rest of the platform config, and the exception process is narrow and logged: if a workload genuinely needs a new path, the request names the source, destination, port, and reason, and it expires on a timer.

Results

• Lateral movement across namespaces became structurally impossible — a compromised pod is contained to its own namespace's allow-listed paths, which are deliberately narrow.
• The metadata endpoint and database ports became unreachable from workloads that never needed them in the first place, closing off entire classes of container-escape and credential-theft paths.
• Because the allow-lists came from observed traffic, the rollout caused zero production incidents — a deliberate contrast with the first attempt.
• Exceptions stopped being silent. Every deviation from default-deny is now a logged, time-boxed, reviewed decision instead of a carve-out nobody remembers making.

Lessons learned

• DNS is the first thing that breaks. Carve out kube-dns (UDP/TCP 53) everywhere before you enforce anything else. Everyone learns this the hard way exactly once.
• Egress is where the value is. Ingress is usually already fronted by an ingress controller with its own controls. The unguarded direction is outbound — that's where data leaves and credentials get stolen.
• A week of flow logs beats a day of guessing. The observation phase was the whole difference between the failed rollout and the successful one. Traffic data is the spec.
• Roll out per namespace, not per cluster. Small blast radius per step means a bad rule is a page, not an outage — and teams build trust in the process as each namespace lands cleanly.
• A gate without an exception path gets bypassed. Same lesson as the admission gates: a narrow, logged, time-boxed override process keeps people going through policy instead of around it.
• Verify enforcement before you trust it. A NetworkPolicy on a CNI that ignores it is decoration. Test it in a scratch namespace and watch the drop happen.

What's next

IP-and-port rules get us containment, but they're still coarse: they say nothing about which workload is on the other end of an allowed connection. The next step is identity-aware policy — using workload identity (via the service mesh's mTLS identities, or Cilium's identity-based policies) so rules say "the checkout service may talk to the payments API" instead of "anything in namespace A may reach port 443 in namespace B." That's the move from network segmentation to real zero-trust inside the cluster — and the SLSA provenance work from the previous post's roadmap is still queued behind it.

If you've rolled out default-deny NetworkPolicies without taking down production, I'd like to hear how you handled the observation phase — get in touch.

Security policy at most organizations lives in a document. "Containers must not run as root." "Images must come from an approved registry." "No privileged containers in production." These rules get written once, reviewed in an audit, and then quietly ignored — because nothing in the deployment path actually checks for them. A misconfigured manifest that violates every rule in the document will deploy exactly as successfully as one that follows all of them.

We hit this directly: a security review found containers running as root, some with privileged access they didn't need, and images pulled from registries nobody had explicitly approved. None of this was malicious — it was just what happens when policy exists as prose instead of as something the pipeline actually enforces. By the time these findings surfaced, the workloads were already live.

Why manual review wasn't enough

The instinctive fix — add a manual security review step before production deploys — didn't scale and didn't match how the org actually shipped software. Reviewers became a bottleneck, review quality varied by who was on call, and the whole model depended on a human catching a YAML misconfiguration by eye, every time, under time pressure. We needed the check to happen automatically, consistently, and early enough that failing it was cheap — a rejected pull request, not an incident report.

The solution: policy gates with OPA and Kyverno

We built a layered policy enforcement setup using Open Policy Agent (OPA) for CI-time checks against Infrastructure-as-Code, and Kyverno as an admission controller enforcing the same intent at the Kubernetes cluster level — so a violation is caught whichever way it tries to reach production.

The policies we enforced were deliberately the highest-leverage ones first:

• No privileged containers — blocks any pod spec requesting privileged mode or unnecessary host access
• Non-root enforcement — rejects containers that don't explicitly set a non-root user
• Approved registry only — images must originate from our internal or explicitly allow-listed registries; anything else is rejected at admission
• Signed image verification — using Sigstore/cosign, Kyverno verifies image signatures at admission time, so an unsigned or tampered image simply cannot run, regardless of how it got built
• Mandatory resource limits — no container without CPU/memory limits defined, closing off a whole class of noisy-neighbor and cost issues alongside the security angle

OPA runs earlier — against Terraform plans and Kubernetes manifests in CI, using Conftest — so violations get caught in a pull request with a clear, specific failure message before anything is ever applied. Kyverno acts as the backstop at the cluster's admission controller, so even a change that bypasses CI (a manual kubectl apply, a hotfix, a misconfigured pipeline elsewhere) still can't land a non-compliant workload. Neither layer alone was sufficient; together they close the gap between "policy that's written" and "policy that's actually true in production."

Results

• Non-compliant manifests started getting rejected automatically at the PR stage, with the specific rule and reason surfaced directly to the engineer — no waiting on a reviewer's availability.
• Privileged and root-running containers in production dropped to effectively zero going forward, since the admission controller makes them structurally impossible to deploy, not just discouraged.
• Unsigned or unverified images became unable to run in the cluster at all, closing a real supply-chain gap rather than relying on trust in the build pipeline alone.
• Security review shifted from "find problems after the fact" to "maintain and extend the policy set" — a meaningfully different, more scalable use of the same team's time.

Lessons learned

• Enforce at more than one layer. CI-time checks are fast feedback for developers; admission-time checks are the actual guarantee. Relying on either alone leaves a gap someone will eventually find.
• Start with rules nobody can reasonably argue with. "No privileged containers" and "must run as non-root" are hard to object to, which built trust in the gate before we introduced anything more opinionated.
• Error messages are part of the policy. A rejected deploy with a vague "denied by policy" message trains people to route around the gate. A rejection that names the exact rule and the exact fix keeps the gate credible.
• Image signing was the highest-value, latest addition. It directly answers "how do we know this image is what we think it is," which matters more every year as supply-chain attacks keep growing in frequency and sophistication.
• A gate without an exception path gets bypassed. We built a narrow, logged, time-boxed override process for legitimate edge cases — without it, people find ways around policy instead of through it.

What's next

We're extending the policy set to cover network policy enforcement (default-deny with explicit allow rules) and tightening the signature verification requirement to include provenance attestation (SLSA-style build provenance), not just a valid signature — moving from "this image wasn't tampered with" to "this image was built by the pipeline we expect, from the source we expect."

If you're building out policy-as-code enforcement with OPA, Kyverno, or a similar stack, I'd like to hear how you're approaching exception handling — get in touch.

A year ago, our CI/CD security posture looked like most growing engineering orgs': inconsistent. Every team owned its own pipeline, and security controls — dependency scanning, secret detection, SAST, image scanning — were either bolted on ad hoc, copy-pasted from whichever team had last dealt with an audit finding, or missing entirely. Some pipelines ran Trivy. Others didn't scan container images at all. Secrets management practices varied wildly from team to team, and enforcing even a baseline policy meant chasing down dozens of pipeline owners individually.

The result was predictable: security findings that should have been caught in minutes were surfacing in later-stage reviews or audits, engineers were duplicating the same scanning logic across pipelines with slightly different (and often outdated) configurations, and there was no single place to push a fix when a new CVE or policy requirement came down. When a new scanning requirement landed, rolling it out meant manually touching every pipeline across every team — a process that never actually reached full coverage.

Why ad hoc fixes weren't enough

Our first instinct was the obvious one: write documentation, share example YAML, ask teams to adopt it. That failed the way it usually does. Docs went stale. Teams copied a snapshot of a pipeline config and never pulled updates. Enforcement was voluntary, so coverage was inconsistent, and every security fix had to be manually re-propagated to every consumer. We needed the fix to live in one place and update everywhere automatically — which meant we needed a platform-level primitive, not a wiki page.

The solution: account-level reusable security templates

We rebuilt our security tooling as a set of account-level (org-wide) Harness CI/CD templates — versioned, centrally maintained pipeline building blocks that any team could pull into their own pipeline with a few lines of configuration, rather than hand-rolling scanning logic themselves.

Each template encapsulated one security control as a reusable step or stage:

• Secret scanning (Gitleaks) — runs on every push, gated to block merges on verified secret exposure
• SAST — static analysis wired to fail builds on critical/high findings above an agreed threshold, tuned over time to keep false positives manageable
• Dependency/SCA scanning — flags vulnerable dependencies and, where relevant, kicks off SBOM generation
• Container image scanning (Trivy) — scans built images before they're allowed to push to the registry

Teams consume these templates the same way they'd use any shared library: reference the template version, pass a handful of parameters (severity thresholds, exclusions, notification targets), and get the full control without writing or maintaining the underlying logic themselves. The majority of our pipelines run on GitHub Actions and Harness CI/CD, with the rest on Azure DevOps for specific tracking and logging workflows — so we designed the templates to be portable in principle, keeping the enforcement logic decoupled from any one platform's quirks.

Critically, this isn't a "set it and forget it" library. We treat the template set as a living, contribution-driven system: when a new vulnerability class needs coverage, a scanning tool gets upgraded, or an audit surfaces a gap, the fix goes into the shared template once and every consuming pipeline inherits it on its next run. Engineers across the org regularly contribute enhancements back — new checks, tuned thresholds, support for a new artifact type — the same way you'd contribute to an internal open-source project.

Results

• Manual pipeline security work dropped dramatically — what used to require touching every team's pipeline individually to roll out a new control now happens in one template update.
• Time-to-rollout for a new security requirement went from weeks to days, in some cases hours, since propagation is automatic for any pipeline referencing the shared template.
• Coverage became consistent org-wide instead of dependent on which team happened to prioritize it — every pipeline built on the templates gets the same baseline controls by default, not by request.
• Audit prep time shrank significantly, since we can point to a single, versioned source of truth for what security controls are enforced, rather than reconstructing per-team configurations.

Lessons learned

• Centralizing logic beats centralizing mandates. Asking teams to comply with a policy is weaker than giving them a default that's easier to adopt than to avoid.
• Versioning matters as much as the controls themselves. Teams need the confidence to pin a version and upgrade on their own schedule, or adoption stalls.
• Treat the template set as a product, not a policy doc. Regular contributions, a changelog, and an easy path for teams to request or add new checks kept engagement high and prevented the templates from going stale the way our original documentation did.
• Start with the highest-leverage control, not the most controls. Secret scanning was our first template because it had the clearest, least controversial value — that early win built the trust needed to expand into SAST and image scanning later.

What's next

We're extending the template set to cover policy-as-code enforcement (OPA/Conftest-style checks) and working on making the templates genuinely platform-agnostic, so the same control logic can be consumed identically whether a team is on Harness, GitHub Actions, or Azure DevOps.

If you're standardizing CI/CD security across a multi-team org, I'd love to hear how you approached it — get in touch.