Send logs from Kubernetes

This page ships container logs from a Kubernetes cluster to Rootprint over OTLP. The OpenTelemetry Collector runs as a DaemonSet — one pod per node — tailing every pod’s stdout/stderr off the kubelet. The chart’s logsCollection preset wires up the filelog receiver; the kubernetesAttributes preset enriches each record with the pod, namespace, node, and container it came from. A small transform infers severity from the message body, the same way the Docker path does.

Use this for workloads that don’t instrument themselves. A service using an OpenTelemetry SDK should send records straight to OTLP; collecting both paths stores every event twice. See Choose what to collect.

Prerequisites

A running Rootprint instance and the host it serves on — you’ll substitute it for <your-rootprint>.
A Kubernetes cluster with kubectl and helm configured against it. The DaemonSet schedules on every node whose pods you want collected.
An ingest API key scoped to your target index. In Settings → API keys, click Create ingest key, give it a name, and pick the index (otel-logs-v0_9 here; see Indexes for its schema). See API keys.

Add the OpenTelemetry Helm repo

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

Create values.yaml

Save this as values.yaml. Replace <your-rootprint> with your Rootprint host and <your-ingest-token> with the API key you copied above.

Show values.yaml

mode: daemonset

# Required: the chart ships no default image. -k8s carries every component used below.
image:
  repository: otel/opentelemetry-collector-k8s

presets:
  logsCollection:
    enabled: true
  kubernetesAttributes:
    enabled: true

# A log tailer needs no inbound ports, and the defaults claim hostPorts that
# collide with any other Collector on the node.
ports:
  otlp:
    enabled: false
  otlp-http:
    enabled: false
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false

config:
  receivers:
    filelog:
      # Skips the Collector's own logs, and Rootprint's if you self-host here.
      exclude:
        - /var/log/pods/rootprint_*/*/*.log
  processors:
    transform:
      log_statements:
        - context: log
          statements:
            - set(log.severity_number, SEVERITY_NUMBER_ERROR) where IsString(log.body) and IsMatch(log.body, "(?i)\\b(error|fatal|panic|exception)\\b")
            - set(log.severity_text, "ERROR") where log.severity_number == SEVERITY_NUMBER_ERROR
            - set(log.severity_number, SEVERITY_NUMBER_WARN) where log.severity_number == 0 and IsString(log.body) and IsMatch(log.body, "(?i)\\b(warn|warning|deprecated|retry)\\b")
            - set(log.severity_text, "WARN") where log.severity_number == SEVERITY_NUMBER_WARN
            - set(log.severity_number, SEVERITY_NUMBER_INFO) where log.severity_number == 0
            - set(log.severity_text, "INFO") where log.severity_text == ""
  exporters:
    otlp_http:
      logs_endpoint: https://<your-rootprint>/v1/logs
      compression: gzip
      headers:
        Authorization: "Bearer <your-ingest-token>"
  service:
    pipelines:
      logs:
        processors: [memory_limiter, k8s_attributes, transform, batch]
        exporters: [otlp_http]

processors and exclude are replaced, not merged. List every entry you want, including the preset-injected memory_limiter, k8s_attributes and batch.
Override the receiver as filelog, not file_log. A file_log block is ignored; Helm prints a [DEPRECATION] notice for it.
otlp_http requires Collector 0.158.0 or newer. Older images name it otlphttp.

Install the chart

helm install rootprint-otel open-telemetry/opentelemetry-collector \
  --namespace rootprint --create-namespace \
  -f values.yaml

Confirm the DaemonSet is running — one pod per node:

kubectl get pods -n rootprint -l app.kubernetes.io/name=opentelemetry-collector

Self-hosting Rootprint in this cluster? Install the Collector into Rootprint’s own namespace so the exclude above covers both. Otherwise the DaemonSet tails Rootprint and its Quickwit backend, and their logs feed back into the index.

Send a test log line

Run a throwaway pod that prints one line and exits. The Collector on that pod’s node tails it and ships it with the pod’s metadata attached.

kubectl run rootprint-smoke-test --image=alpine --restart=Never \
  -- echo "hello from rootprint"

Clean up afterwards with kubectl delete pod rootprint-smoke-test.

Verify in Rootprint

Open Search, pick otel-logs-v0_9 from the index selector, and query for hello from rootprint. Allow up to a minute for the batch to flush and Quickwit to commit. resource_attributes.k8s.pod.name reads rootprint-smoke-test.

What you get in Rootprint

The k8s.* keys are OTLP resource attributes, so query them under resource_attributes:

Field	Value
`resource_attributes.k8s.pod.name`	pod the log came from
`resource_attributes.k8s.namespace.name`	pod’s namespace
`resource_attributes.k8s.node.name`	node the pod is scheduled on
`resource_attributes.k8s.container.name`	container within the pod
`resource_attributes.k8s.pod.uid`	pod UID
`resource_attributes.k8s.deployment.name`	owning Deployment, when there is one
`service_name`	derived from the workload
`attributes.log.file.path`	the `/var/log/pods/...` file the line came from
`attributes.log.iostream`	`stdout` or `stderr`
severity	inferred from the message body

Severity inference

The transform matches two pattern families against the body:

error, fatal, panic, or exception (word-boundary) → severityText: ERROR (severity number 17).
warn / warning, deprecated, or retry → severityText: WARN (13).
Everything else → severityText: INFO (9).

DEBUG is not inferred. Emit debug records through an OpenTelemetry SDK that sets severityNumber itself.

Choose what to collect

logsCollection tails every pod on the node. Exclude what you don’t want before pointing it at a busy cluster.

Workloads that already send OTLP

A service instrumented with an OpenTelemetry SDK emits structured records carrying traceId, spanId and its own severityNumber, and usually prints the same events to stdout as well. Tail that stdout too and you store both copies, the second one flat and uncorrelated. Pick one path per workload.

Exclude the namespaces whose services report over OTLP:

config:
  receivers:
    filelog:
      exclude:
        - /var/log/pods/rootprint_*/*/*.log
        - /var/log/pods/<instrumented-namespace>_*/*/*.log

Path excludes never open the file. To filter on something the path doesn’t carry, add a filter processor after k8s_attributes and drop on resource_attributes.

Collect selectively instead

To have workloads opt in, use annotation-based discovery and drop the filelog block:

presets:
  logsCollection:
    enabled: false
  annotationDiscovery:
    logs:
      enabled: true

Annotate the pods to collect:

metadata:
  annotations:
    io.opentelemetry.discovery.logs/enabled: 'true'

Per-container variants (io.opentelemetry.discovery.logs.<container>/enabled) and per-pod filelog settings via io.opentelemetry.discovery.logs/config are documented upstream.

Multi-line logs

The preset’s container operator rejoins lines split by the container runtime, not multi-line output from the application. A stack trace or a pretty-printed object arrives as one record per line. Add a recombine operator, restating the container parser because operators is replaced wholesale:

config:
  receivers:
    filelog:
      operators:
        - type: container
          id: container-parser
        - type: recombine
          combine_field: body
          is_first_entry: body matches "^[^\\s]"

is_first_entry marks a record boundary — above, any line not starting with whitespace, which covers indented stack frames. Use a timestamp or level prefix instead if that fits your output better. source_identifier defaults to attributes["log.file.path"], so lines from different containers never merge.

Cluster events (optional)

The steps above collect pod logs. To also capture cluster-level events — scheduling, image pulls, OOMKills — deploy a second Collector as a single-replica Deployment (not a DaemonSet, to avoid duplicate events) with the kubernetesEvents preset:

mode: deployment
replicaCount: 1
image:
  repository: otel/opentelemetry-collector-k8s
presets:
  kubernetesEvents:
    enabled: true
config:
  exporters:
    otlp_http:
      logs_endpoint: https://<your-rootprint>/v1/logs
      compression: gzip
      headers:
        Authorization: "Bearer <your-ingest-token>"
  service:
    pipelines:
      logs:
        exporters: [otlp_http]

Install it under a separate release name (e.g. rootprint-otel-events) so it doesn’t collide with the DaemonSet. To keep a single workload instead, enable presets.kubernetesObjects alongside kubernetesEvents on the DaemonSet: it adds a leader election lease so only one pod watches, and also ingests the cluster’s object inventory on a schedule.

Troubleshooting

helm install fails with [ERROR] 'image.repository' must be set: add the image.repository block from the values above.
DaemonSet pod stays Pending with didn't have free ports: another Collector holds the default hostPorts. Disable them as shown above.
No pods in the DaemonSet: nodes may carry taints the Collector doesn’t tolerate. Add the matching tolerations, or check kubectl describe daemonset -n rootprint.
Records arrive as single lines of a larger message: application-level multi-line output. See Multi-line logs.
Every event appears twice: a workload is both exporting OTLP and printing to stdout. Compare attributes.log.file.path:* against NOT attributes.log.file.path:* to see the split, then exclude that namespace.
401 from Rootprint: the Authorization header is missing or the token is wrong.
403 from Rootprint: the ingest API key is invalid, unknown, or revoked. Create a new one in Settings → API keys. A key scoped to a different index does not return 403 — it writes there silently, so check the key’s index if records land somewhere unexpected.
413 from Rootprint: the batch is too large. Add send_batch_max_size to the batch processor.
415 from Rootprint: the Content-Type is not application/x-protobuf. The Collector sets this automatically; check for a proxy stripping headers.
Logs arrive without k8s.* attributes: the kubernetesAttributes preset needs RBAC to read pod metadata. The chart creates the ClusterRole automatically — if you disabled RBAC creation, grant get/list/watch on pods and namespaces.

Send logs from Docker: container logs on a single Linux host.
Send logs with the OpenTelemetry Collector: the same Collector tailing files on a host.
OTLP reference: endpoint URL, response codes, body limits.
Indexes: the otel-logs-v0_9 schema.

Get Started

Send Logs

Traces

Search Logs

Manage Rootprint

Send logs from Kubernetes

Prerequisites

What you get in Rootprint

Severity inference

Choose what to collect

Workloads that already send OTLP

Collect selectively instead

Multi-line logs

Cluster events (optional)

Troubleshooting

​Prerequisites

​What you get in Rootprint

​Severity inference

​Choose what to collect

​Workloads that already send OTLP

​Collect selectively instead

​Multi-line logs

​Cluster events (optional)

​Troubleshooting

​Related

Prerequisites

What you get in Rootprint

Severity inference

Choose what to collect

Workloads that already send OTLP

Collect selectively instead

Multi-line logs

Cluster events (optional)

Troubleshooting

Related