Cloud-Native / Regulated Infrastructure
Security Logging & Platform Automation
Kubernetes, Fluent Bit, S3-Compatible Storage
Auditability, Retention & Incident Traceability
Looking for a resilient partner?
A Kubernetes audit trail is only useful when it is complete, durable, and easy to retrieve. The key challenge was not just collecting logs, but making audit evidence reliable enough for security reviews and compliance workflows.
Target Metadata and Request stages for create/update/patch/delete/auth events. Suppress high-volume reads on pods, configmaps, and endpoints that add noise without security value.
Parse the raw JSON audit event, extract verb, user.username, sourceIPs, objectRef.namespace, and responseStatus.code, then attach cluster name as a static tag.
Write compressed NDJSON objects to S3-compatible storage under cluster/date/namespace/ prefixes — making retrieval scripting predictable without a search index.
Use prefix-scoped S3 queries and jq filters to reconstruct administrative timelines for incident response and compliance evidence requests.
The strategy was intentionally storage-first: preserve clean, structured audit evidence in a durable location first, then enable downstream search, dashboards, or SIEM integrations as optional layers.
The pipeline runs entirely within the cluster network. Fluent Bit tails the kube-apiserver audit log file on each control-plane node, applies JSON parsing and enrichment, buffers to local disk during transient S3 outages, and flushes compressed objects to an S3-compatible bucket under structured prefixes. No SIEM, no middleware, no streaming broker required.
Logs, metrics, traces & events
Cross-system incident grouping
Intelligent anomaly analysis
Automated recovery actions
Real-time observability context
Slack, Jira & PagerDuty
The design separated audit generation from audit storage. Kubernetes remained responsible for producing accurate API activity records, while the pipeline ensured those records were safely preserved outside the cluster.
The implementation roadmap focused on building a unified observability foundation, enabling intelligent incident correlation workflows, and orchestrating automated remediation pipelines across distributed cloud-native infrastructure.
Start with a minimal audit policy targeting RequestReceived and ResponseComplete stages for write and delete verbs across all API groups.
Add explicit rules for secrets, clusterroles, clusterrolebindings, and serviceaccounts — these are the objects attackers touch first. Suppress events for pods/status, nodes/status, and configmaps with GET/LIST/WATCH verbs to keep volume manageable.
Fluent Bit is configured with a FILE input using Path /var/log/kubernetes/audit.log, Refresh_Interval 5, and DB /var/log/fluentbit-audit.db for position tracking across restarts. The Lua or RECORD_MODIFIER filter injects cluster_name, and the S3 output plugin is configured with compression gzip, s3_key_format /%Y/%m/%d/{namespace}/audit, and upload_chunk_size 10M.
Filesystem buffering uses storage.type filesystem with storage.backlog.mem_limit 50M to prevent memory pressure during prolonged S3 unavailability.
Retrieval is implemented as a two-step pattern: first list objects under the target prefix using aws s3 ls –recursive, then download and decompress the relevant date/namespace range before piping through jq to extract the specific actor, verb, and objectRef fields needed for the investigation.
For compliance requests, a packaging script collects the retrieved objects, generates a manifest with SHA256 checksums, and produces a signed ZIP that serves as a defensible evidence bundle. Object lifecycle rules on the S3 bucket enforce the required retention window (90-day hot, transition to IA at 365 days, expire at 7 years) without requiring manual intervention.
Control-plane SSH access is no longer required to trace privileged actions. Security teams retrieve scoped audit timelines in under 5 minutes using prefix queries.
Audit events live in immutable S3 objects outside the cluster. Object lock and lifecycle policies make deletion difficult without explicit credentials.
The retrieval and packaging workflow produces a consistent evidence format that satisfies SOC 2 and internal audit requests without manual reconstruction.
Platform Security Lead
Picout Gmbh.