Skip to content
CASE STUDY / KUBERNETES SECURITY & COMPLIANCE

Kubernetes Audit Log Pipeline with Fluent Bit & S3-Compatible Archival

A regulated platform team needed a reliable way to collect, preserve, and archive Kubernetes audit logs across multiple clusters without depending on fragile node-level manual exports. The solution implemented a policy-driven audit logging pipeline using Kubernetes API server audit policies, Fluent Bit collectors, structured buffering, and S3-compatible object storage.

The result was a tamper-resistant, searchable, and retention-aware audit trail that improved compliance readiness, incident investigation, and platform accountab

Industry

Cloud-Native / Regulated Infrastructure

Engagement Type

Security Logging & Platform Automation

Primary Stack

Kubernetes, Fluent Bit, S3-Compatible Storage

Delivery Focus

Auditability, Retention & Incident Traceability

A centralized audit log pipeline was designed to capture Kubernetes control-plane activity, normalize records, buffer safely, and archive them into long-term object storage for compliance and investigation workflows.

Business Challenges​

The existing Kubernetes audit logging approach was fragmented across clusters and environments. Audit records were available only locally on control-plane nodes, retention was inconsistent, and security teams had limited ability to reconstruct administrative actions during incident reviews. The organization needed a repeatable audit pipeline that could scale across clusters while maintaining integrity, retention, and operational simplicity.

No Durable Audit Retention

Audit logs were stored locally per cluster, making investigation slow and inconsistent across environments.

Slow Incident Reconstruction

Security teams had to manually SSH into control-plane nodes to trace who made a privileged change and when.
hard-disk-line

Cluster Disk Pressure

Verbose audit logging consumed cluster-local storage with no automated rotation tied to external archival.

Unstructured Log Objects

Raw audit files lacked namespace, cluster, verb, and actor prefixes — making scoped retrieval impractical.

High-Volume Signal Noise

Audit events included excessive read operations that buried security-relevant write, delete, and auth events.

A Kubernetes audit trail is only useful when it is complete, durable, and easy to retrieve. The key challenge was not just collecting logs, but making audit evidence reliable enough for security reviews and compliance workflows.

Strategic Approach

The implementation focused on building a lightweight but durable audit logging pipeline. Instead of introducing a heavy SIEM-first dependency, the design prioritized structured log capture, safe buffering, object storage archival, and clear retention boundaries. Fluent Bit was used as the collection and routing layer, while S3-compatible storage provided scalable long-term retention.

Calibrated Audit Policy

Target Metadata and Request stages for create/update/patch/delete/auth events. Suppress high-volume reads on pods, configmaps, and endpoints that add noise without security value.

Fluent Bit at the Edge

Deploy Fluent Bit as a DaemonSet or static pod on control-plane nodes to tail the audit log file directly — no sidecar required, no log forwarding agent.

Structured Enrichment

Parse the raw JSON audit event, extract verb, user.username, sourceIPs, objectRef.namespace, and responseStatus.code, then attach cluster name as a static tag.

Partitioned S3 Archival

Write compressed NDJSON objects to S3-compatible storage under cluster/date/namespace/ prefixes — making retrieval scripting predictable without a search index.

Replay & Evidence Workflow

Use prefix-scoped S3 queries and jq filters to reconstruct administrative timelines for incident response and compliance evidence requests.

The strategy was intentionally storage-first: preserve clean, structured audit evidence in a durable location first, then enable downstream search, dashboards, or SIEM integrations as optional layers.

Solution Architecture

The pipeline runs entirely within the cluster network. Fluent Bit tails the kube-apiserver audit log file on each control-plane node, applies JSON parsing and enrichment, buffers to local disk during transient S3 outages, and flushes compressed objects to an S3-compatible bucket under structured prefixes. No SIEM, no middleware, no streaming broker required.

Unified Telemetry

Logs, metrics, traces & events

Event Correlation

Cross-system incident grouping

AI Detection

Intelligent anomaly analysis

Smart Remediation

Automated recovery actions

Operational Insights

Real-time observability context

Workflow Integration

Slack, Jira & PagerDuty

The design separated audit generation from audit storage. Kubernetes remained responsible for producing accurate API activity records, while the pipeline ensured those records were safely preserved outside the cluster.

Solution Implementation

The implementation roadmap focused on building a unified observability foundation, enabling intelligent incident correlation workflows, and orchestrating automated remediation pipelines across distributed cloud-native infrastructure.

PHASE 01

Audit Policy & Source Validation

Start with a minimal audit policy targeting RequestReceived and ResponseComplete stages for write and delete verbs across all API groups.

Add explicit rules for secrets, clusterroles, clusterrolebindings, and serviceaccounts — these are the objects attackers touch first. Suppress events for pods/status, nodes/status, and configmaps with GET/LIST/WATCH verbs to keep volume manageable.

Key Deliverables

PHASE 02

Fluent Bit Pipeline & S3 Archival

Fluent Bit is configured with a FILE input using Path /var/log/kubernetes/audit.log, Refresh_Interval 5, and DB /var/log/fluentbit-audit.db for position tracking across restarts. The Lua or RECORD_MODIFIER filter injects cluster_name, and the S3 output plugin is configured with compression gzip, s3_key_format /%Y/%m/%d/{namespace}/audit, and upload_chunk_size 10M. 

Filesystem buffering uses storage.type filesystem with storage.backlog.mem_limit 50M to prevent memory pressure during prolonged S3 unavailability.

Key Deliverables

PHASE 03

Retrieval, Replay & Compliance Evidence

Retrieval is implemented as a two-step pattern: first list objects under the target prefix using aws s3 ls –recursive, then download and decompress the relevant date/namespace range before piping through jq to extract the specific actor, verb, and objectRef fields needed for the investigation.


For compliance requests, a packaging script collects the retrieved objects, generates a manifest with SHA256 checksums, and produces a signed ZIP that serves as a defensible evidence bundle. Object lifecycle rules on the S3 bucket enforce the required retention window (90-day hot, transition to IA at 365 days, expire at 7 years) without requiring manual intervention.

Key Deliverables

Outcome & Impact

Faster Investigations

Control-plane SSH access is no longer required to trace privileged actions. Security teams retrieve scoped audit timelines in under 5 minutes using prefix queries.

Durable, Tamper-Resistant Evidence

Audit events live in immutable S3 objects outside the cluster. Object lock and lifecycle policies make deletion difficult without explicit credentials.

Repeatable Compliance Evidence

The retrieval and packaging workflow produces a consistent evidence format that satisfies SOC 2 and internal audit requests without manual reconstruction.

"The platform team moved from best-effort log collection to a defensible audit trail that supports incident response and compliance evidence without adding operational drag."

Platform Security Lead

Picout Gmbh.