Logs - Control Plane

Overview

The Logs add-on, once enabled, will send logs from pods running on the cluster, and optionally, Kubernetes audit records, to Control Plane. This will allow you to query the logs and audit records from your clusters managed by the Control Plane centrally in the web UI or in Grafana.

Supported Providers

How to Enable

The Logs add-on can be enabled for your Kubernetes cluster either during the cluster creation process or at any time thereafter. The following sections outline the methods for enabling the add-on:

At Cluster Creation

Through Cluster Manifest: Add the following snippet to your cluster manifest when creating the cluster:
YAML
spec: ... addOns: logs: {} ...
Using the Console: If you’re creating the cluster through the console, navigate to Add-ons, find the Logs add-on in the list of available add-ons, and toggle it on.

After Cluster Creation

If the Logs add-on was not enabled during the cluster creation, you can still enable it using either of the following methods:

Using Manifest

Under spec.addOns in the YAML manifest of the cluster, you can edit it either by navigating to the cluster in the Console and using the Edit & Apply option for the cluster, or by applying the entire manifest using the cpln apply >_ option in the upper right corner or by using the cpln CLI. Add the following:

YAML

spec:
---
addOns:
  logs: {}

Using UI

Navigate to Control Plane Console: Visit the Control Plane Console.
Navigate to the Kubernetes cluster: In the Control Plane Console, navigate to Kubernetes in the left sidebar panel and click on the Kubernetes cluster for which you want to enable logging.
Enable Logs: Choose Add-ons and locate the Logs add-on from the list of available add-ons, then toggle it on.

Configuring Logs add-on

The configuration is optional. All settings are off or unrestricted when omitted. You can configure the following:

Field	Type	Description
`auditEnabled`	boolean	Collect the Kubernetes audit log as log events.
`includeNamespaces`	regular expression	Collect pod logs only from namespaces that match. All namespaces are collected when omitted.
`excludeNamespaces`	regular expression	Skip pod logs from namespaces that match. Nothing is excluded when omitted.
`kubelet`	boolean	Collect the logs of the `k3s-agent` service, which runs the kubelet, on each node.
`kernel`	boolean	Collect the kernel logs of each node.
`docker`	boolean	Collect the logs of containers managed by Docker, if Docker is also running on the node.

Pod logs from the cluster are always collected. Pods that belong to Control Plane workloads are not included, and neither are the logs of the collector itself.

Namespace filters

includeNamespaces and excludeNamespaces are regular expressions (RE2 syntax) that are matched against the whole namespace name, so app does not match app1. To match several namespaces, use alternation, for example default|frontend|app1. A comma separated list is not a list of names. When both are set, a namespace must match includeNamespaces and must not match excludeNamespaces. The filters apply to pod logs only. They do not affect kubelet, kernel, docker or audit logs.

Node logs

kubelet and kernel read the systemd journal in /var/log/journal on each node. Only the last 4 hours are read when collection starts.
kubelet reads the k3s-agent.service unit. The cluster’s control plane is hosted by Control Plane, so every node runs the k3s agent and is covered.
docker has no effect on nodes that do not run Docker, because it reads the Docker socket at /var/run/docker.sock.

You can make the configuration in the UI for your cluster, as displayed in the picture below, or by modifying the clusters object spec.

Logs add-on configuration UI showing namespace inclusion/exclusion filters and audit log toggle

Alternatively, directly edit the clusters object spec:

YAML

spec:
---
addOns:
  logs:
    auditEnabled: true
    excludeNamespaces: kube-system|istio-system
    includeNamespaces: default|frontend|app1
    kubelet: true
    kernel: true
    docker: false

Analyzing Logs

Once enabled, you can analyze the logs and audit events. For LogQL query examples and Grafana integration, see the Logs documentation.

Navigate to Control Plane Console: Visit the Control Plane Console.
Analyze logs: In the Control Plane Console, navigate to Logs in the left sidebar panel.
Use LogQL for query. Example query: {cluster_name="aws-us-east-2", namespace="default"}

Control Plane Console logs view with LogQL query bar and filtered log entries for a Kubernetes cluster

Click on Explore On Grafana and analyze logs in Grafana:

Grafana Explore view showing Kubernetes cluster logs with time-series graph and log detail panel

Labels

Every log line carries the cluster_name and cluster_alias labels. The source is identified by the job label:

Source	`job`	Other labels
Pod logs	`kubernetes-pods`	`namespace`, `pod`, `container`, `app`, `instance`, `component`, `owner`, `node`, `image`, `stream`
Kubelet	`kubelet`	`node`
Kernel	`kernel`	`node`, `hostname`
Docker	`docker-containers`	`node`, `container`, `container_id`, `stream`
Audit	`audit`	`verb`, `user`, `decision`, `resource`, `namespace`, `name`, `response_status`, `response_code`, `response_reason`

Audit records of Kubernetes system components and nodes, such as system:kube-scheduler and system:node:*, are not collected. Example queries:

{cluster_name="aws-us-east-2", job="kubelet"}
{cluster_name="aws-us-east-2", job="kernel"} |= "Out of memory"
{cluster_name="aws-us-east-2", job="audit", verb="delete"}

​Overview

​Supported Providers

​How to Enable

​At Cluster Creation

​After Cluster Creation

​Using Manifest

​Using UI

​Configuring Logs add-on

​Namespace filters

​Node logs

​Analyzing Logs

​Labels