Send logs from Kubernetes

The OpenTelemetry Collector runs as a DaemonSet, one pod per node, and ships every pod’s stdout and stderr to Rootprint over OTLP. Each record carries the pod, namespace, node, and container it came from.

Prerequisites

A running Rootprint instance. Substitute its host for <your-rootprint>.
An ingest API key scoped to otel-logs-v0_9. Create one in Settings → API keys with Create ingest key, then substitute it for <your-ingest-api-key>. See API keys.
A Kubernetes cluster with kubectl and helm configured against it.

Add the OpenTelemetry Helm repo

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

Create values.yaml

Save this as values.yaml. Replace <your-rootprint> and <your-ingest-api-key>.

Show values.yaml

mode: daemonset

# Required: the chart ships no default image. -k8s carries every component used below.
image:
  repository: otel/opentelemetry-collector-k8s

presets:
  logsCollection:
    enabled: true
  kubernetesAttributes:
    enabled: true

# A log tailer needs no inbound ports, and the defaults claim hostPorts that
# collide with any other Collector on the node.
ports:
  otlp:
    enabled: false
  otlp-http:
    enabled: false
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false

config:
  receivers:
    file_log:
      # Skips the Collector's own logs, and Rootprint's if you self-host here.
      exclude:
        - /var/log/pods/rootprint_*/*/*.log
  processors:
    # Only fills in records with no log level, so OTLP logs keep the level their SDK set.
    transform:
      log_statements:
        - set(log.severity_number, SEVERITY_NUMBER_ERROR) where log.severity_number == SEVERITY_NUMBER_UNSPECIFIED and IsString(log.body) and IsMatch(log.body, "(?i)\\b(error|fatal|panic|exception)\\b")
        - set(log.severity_number, SEVERITY_NUMBER_WARN) where log.severity_number == SEVERITY_NUMBER_UNSPECIFIED and IsString(log.body) and IsMatch(log.body, "(?i)\\b(warn|warning|deprecated)\\b")
        - set(log.severity_number, SEVERITY_NUMBER_INFO) where log.severity_number == SEVERITY_NUMBER_UNSPECIFIED
        - set(log.severity_text, "ERROR") where log.severity_text == "" and log.severity_number == SEVERITY_NUMBER_ERROR
        - set(log.severity_text, "WARN") where log.severity_text == "" and log.severity_number == SEVERITY_NUMBER_WARN
        - set(log.severity_text, "INFO") where log.severity_text == "" and log.severity_number == SEVERITY_NUMBER_INFO
  exporters:
    otlp_http:
      logs_endpoint: https://<your-rootprint>/v1/logs
      compression: gzip
      headers:
        Authorization: "Bearer <your-ingest-api-key>"
  service:
    pipelines:
      logs:
        processors: [memory_limiter, k8s_attributes, transform, batch]
        exporters: [otlp_http]

processors and exclude are replaced, not merged. List every entry you want, including the preset-injected memory_limiter, k8s_attributes and batch.
file_log and otlp_http need Collector v0.149 or newer.

Install the chart

helm install rootprint-otel open-telemetry/opentelemetry-collector \
  --namespace rootprint --create-namespace \
  -f values.yaml

Confirm the DaemonSet runs one pod per node:

kubectl get pods -n rootprint -l app.kubernetes.io/name=opentelemetry-collector

If Rootprint runs in this cluster, install the Collector into Rootprint’s namespace so the exclude covers both. Otherwise Rootprint’s and Quickwit’s own logs feed back into the index.

Send a test log line

kubectl run rootprint-smoke-test --image=alpine --restart=Never \
  -- echo "hello from rootprint"

Clean up afterwards with kubectl delete pod rootprint-smoke-test.

Verify in Rootprint

Open Logs, pick otel-logs-v0_9 from the index selector, and search for hello from rootprint. Allow up to a minute. resource_attributes.k8s.pod.name reads rootprint-smoke-test.

What you get in Rootprint

The k8s.* keys are OTLP resource attributes, so query them under resource_attributes:

Field	Value
`resource_attributes.k8s.pod.name`	pod the log came from
`resource_attributes.k8s.namespace.name`	pod’s namespace
`resource_attributes.k8s.node.name`	node the pod is scheduled on
`resource_attributes.k8s.container.name`	container within the pod
`resource_attributes.k8s.pod.uid`	pod UID
`resource_attributes.k8s.deployment.name`	owning Deployment, when there is one
`service_name`	derived from the workload
`attributes.log.file.path`	the `/var/log/pods/...` file the line came from
`attributes.log.iostream`	`stdout` or `stderr`
`severity_text`	log level inferred from the message body

The transform sets a log level only on records that arrive without one. It matches error, fatal, panic, or exception as ERROR, warn, warning, or deprecated as WARN, and everything else as INFO. DEBUG is never inferred.

Choose what to collect

logsCollection tails every pod on the node. Exclude what you don’t want before you point it at a busy cluster.

Workloads that already send OTLP

A service instrumented with an OpenTelemetry SDK emits structured records carrying traceId, spanId and its own severityNumber, and usually prints the same events to stdout as well. Tail that stdout too and you store both copies, the second one flat and uncorrelated. Pick one path per workload.

Exclude the namespaces whose services report over OTLP:

config:
  receivers:
    file_log:
      exclude:
        - /var/log/pods/rootprint_*/*/*.log
        - /var/log/pods/<instrumented-namespace>_*/*/*.log

To filter on something the path doesn’t carry, add a filter processor after k8s_attributes and drop on resource_attributes.

Collect selectively instead

To have workloads opt in, use annotation-based discovery and drop the file_log block. The preset collects every pod unless you set its default annotation to "false":

presets:
  logsCollection:
    enabled: false
  annotationDiscovery:
    logs:
      enabled: true

config:
  receivers:
    receiver_creator/logs:
      discovery:
        default_annotations:
          io.opentelemetry.discovery.logs/enabled: "false"

Annotate the pods to collect:

metadata:
  annotations:
    io.opentelemetry.discovery.logs/enabled: 'true'

For per-container and per-pod settings, see the upstream guide.

Multi-line logs

The preset’s container operator rejoins lines split by the container runtime, not multi-line output from the application. A stack trace arrives as one record per line. Add a recombine operator. Restate the container parser too, because operators is replaced, not merged:

config:
  receivers:
    file_log:
      operators:
        - type: container
          id: container-parser
        - type: recombine
          combine_field: body
          is_first_entry: body matches "^[^\\s]"

is_first_entry marks where a record starts: here, any line that doesn’t start with whitespace, which covers indented stack frames. Match a timestamp or level prefix instead if that fits your output better.

Cluster events (optional)

To also collect cluster events such as scheduling, image pulls, and OOMKills, deploy a second Collector with the kubernetesEvents preset. Use a single-replica Deployment, not a DaemonSet, so each event arrives once:

mode: deployment
replicaCount: 1
image:
  repository: otel/opentelemetry-collector-k8s
presets:
  kubernetesEvents:
    enabled: true
config:
  exporters:
    otlp_http:
      logs_endpoint: https://<your-rootprint>/v1/logs
      compression: gzip
      headers:
        Authorization: "Bearer <your-ingest-api-key>"
  service:
    pipelines:
      logs:
        exporters: [otlp_http]

Install it under a separate release name, such as rootprint-otel-events.

Troubleshooting

helm install fails with [ERROR] 'image.repository' must be set: add the image.repository block from the values above.
A DaemonSet pod stays Pending with didn't have free ports: another Collector holds the default hostPorts. Disable them as in the values above.
Every event appears twice: a workload both exports OTLP and prints to stdout. Compare attributes.log.file.path:* with NOT attributes.log.file.path:* to see the split, then exclude that namespace.
Logs arrive without k8s.* attributes: the kubernetesAttributes preset needs RBAC to read pod metadata. The chart creates the ClusterRole. If you disabled RBAC creation, grant get, list, and watch on pods and namespaces.
401/403/413/415: see Response codes.

Send logs from Docker: container logs on a single Linux host.
Send logs with the OpenTelemetry Collector: the same Collector tailing files on a host.
OTLP reference: endpoint, response codes, limits.
Indexes: the otel-logs-v0_9 schema.

Get Started

Send Logs

Traces

Search Logs

Manage Rootprint

Send logs from Kubernetes

Prerequisites

What you get in Rootprint

Choose what to collect

Workloads that already send OTLP

Collect selectively instead

Multi-line logs

Cluster events (optional)

Troubleshooting

​Prerequisites

​What you get in Rootprint

​Choose what to collect

​Workloads that already send OTLP

​Collect selectively instead

​Multi-line logs

​Cluster events (optional)

​Troubleshooting

​Related

Prerequisites

What you get in Rootprint

Choose what to collect

Workloads that already send OTLP

Collect selectively instead

Multi-line logs

Cluster events (optional)

Troubleshooting

Related