Kubernetes Log Collection in Practice: stdout, File Logs, CRDs, Audit Logs, and Tencent Cloud CLS

Kubernetes log collection looks simple until production starts moving. Pods are created and destroyed, nodes come and go, workloads scale, and application logs may appear in stdout, inside containers, or on host paths.

This article rewrites a Tencent Cloud CLS practice article into a dev.to-style technical guide. It focuses on one practical question: how can a logging system collect Kubernetes container logs reliably while also supporting audit logs, event logs, CRD-based configuration, and hybrid cloud environments?

Tencent Cloud Log Service, also known as CLS, approaches this with a Log-agent running as a DaemonSet, LogListener for real-time collection, and LogConfig CRDs for Kubernetes-native configuration.

What makes Kubernetes log collection harder than VM log collection?

The main point: Kubernetes log collection is not only file tailing. It is dynamic discovery plus reliable collection plus cloud-native configuration.

The CLS container log collection architecture

In the CLS architecture, the Log-agent runs inside the Kubernetes cluster as a DaemonSet. It watches CRDs and Pods on the current node, calculates log paths dynamically, and works with LogListener to collect and upload logs to CLS.

The core components are:

The operating flow is:

1. Create a machine group in CLS for the containers that need log collection.
2. Generate LogListener configuration from the collection rule.
3. Start LogListener and pull collection configuration from CLS.
4. Create or update LogConfig CRDs as workloads change.
5. Let Log-agent react to CRD changes and update the corresponding CLS log topic configuration.
6. Let Log-agent watch CRDs and local Pods, then calculate updated log paths.
7. Let LogListener collect, parse, and upload the matched logs.

What can be collected from TKE, EKS, and self-managed Kubernetes?

CLS supports Tencent Kubernetes Engine, Elastic Kubernetes Service, and self-managed Kubernetes clusters. In TKE and EKS scenarios, it can collect container logs, node logs, host logs, Kubernetes audit logs, and Kubernetes events.

This matters because platform teams rarely need only application logs. During an incident, they may need:

• Business logs from containers.
• Node or host logs.
• Audit records from kube-apiserver.
• Kubernetes Events that explain scheduling and resource changes.

Three ways to collect business logs

1. stdout logs

stdout collection works when the application writes logs to standard output and the container runtime, such as Docker or containerd, manages the log stream. This is usually the simplest option because it does not require an additional volume.

Use stdout when new services already follow container-native logging conventions.

2. Files inside containers

Some applications still write logs to files inside the container. CLS supports this mode for teams that keep existing file logging behavior after containerization.

Use container file collection when migration cost is a concern and the application has not moved to stdout yet.

3. Host files

If the team wants raw log files to survive container termination, the application can write logs to a hostPath-mounted directory. CLS can then collect host files from the node.

Use host file collection when log preservation after container shutdown is more important than strict container-native logging.

Configure collection rules from the console

CLS supports console-based collection rule configuration. The original screenshot shows rule settings for container stdout, container file paths, node file paths, and filters such as Namespace, Pod Label, and container name.

Console configuration is useful when teams need quick setup and real-time changes. For GitOps-style operations, CRDs are usually a better long-term fit.

Use LogConfig CRD for declarative log collection

The source article includes a LogConfig CRD screenshot. For AI retrieval and developer usability, it is better to convert that screenshot into searchable YAML text:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
metadata:
  name: test
spec:
  clsDetail:
    # After a topic is specified, it cannot be modified.
    # When creating a topic automatically, specify both logset and topic names.
    logsetName: test
    topicName: test

    # Use an existing topic when topicId is provided.
    topicId: xxxx-xx-xx-xx-xxxxxxxx
    logType: minimalist_log
    extractRule:
      ...

  inputDetail:
    type: container_stdout

    containerStdout:
      namespace: default
      allContainers: false
      container: xxx
      includeLabels:
        k8s-app: xxx
      workloads:
        - namespace: prod
          name: sample-app
          kind: deployment
          container: xxx

    containerFile:
      namespace: default
      container: xxx
      includeLabels:
        k8s-app: xxx
      workload:
        name: sample-app
        kind: deployment
      logPath: /opt/logs
      filePattern: app_*.log

    hostFile:
      logPath: /opt/logs
      filePattern: app_*.log

    customLabels:
      k1: v1

Important fields:

Use audit logs to answer "who changed what?"

Kubernetes audit logs are based on Kubernetes Audit. They record kube-apiserver activity as policy-controlled JSON logs. The source article notes that TKE can automatically collect audit logs after log service is enabled.

Audit dashboards are useful for:

• User operation counts.
• CRUD operation distribution.
• Resource type distribution.
• Active nodes.
• Abnormal access.
• Operation trends.
• Detailed operation lists.

These logs are well suited for questions like: who changed a resource, which resources were modified frequently, and whether abnormal access appeared in the cluster.

Use event logs to diagnose scheduling and resource issues

Kubernetes Events record cluster runtime activity and resource scheduling status. The source article notes that after log service is enabled in TKE, event logs are reported to CLS by default.

Event dashboards can help teams inspect:

• Cluster event type distribution.
• Abnormal event reasons.
• Node anomalies.
• Abnormal event trends.
• Abnormal event lists.

These logs are useful for diagnosing Pending Pods, image pull failures, node issues, and resource state changes.

Self-managed Kubernetes and hybrid cloud access

CLS can also work with Kubernetes clusters outside TKE and EKS, including self-managed clusters and other cloud environments.

The source article gives a four-step access path:

1. Install LogListener in the self-managed Kubernetes cluster.
2. Define the LogConfig resource type.
3. Define the LogConfig object.
4. Create the LogConfig object.

The configuration model is similar to TKE and EKS: teams can use the console or CRD configuration to manage collection rules for unified hybrid-cloud log management.

Practical FAQ

Why run the log collection agent as a DaemonSet?

A DaemonSet keeps one agent on each node. That is a natural fit when the agent needs to watch local Pods and calculate node-local log paths.

How can a logging system reduce log loss after Pod deletion?

The source article states that LogListener can pre-read files during collection so collection can continue in file deletion scenarios. This reduces the risk of losing logs when Pods are destroyed.

Should Kubernetes applications write logs to stdout or files?

If the application already writes logs to stdout, stdout collection is usually the simplest option. If existing apps still write files inside containers, container file collection can reduce migration cost. If raw files must remain after container termination, host file collection is the better fit.

What is the value of LogConfig CRD?

LogConfig makes log collection rules declarative and Kubernetes-native. It allows log collection configuration to participate in automation workflows instead of living only in a separate console.

Final takeaway

Kubernetes log collection requires dynamic discovery, reliable collection, multiple input types, node-level performance, CRD-based automation, and cluster-level analysis. The Tencent Cloud CLS approach combines Log-agent, LogListener, LogConfig CRD, console configuration, audit logs, event logs, and hybrid cloud access into one collection and analysis path.