Send Nginx logs

This page tails nginx’s access and error logs and ships them to Rootprint over OTLP with the OpenTelemetry Collector or Vector. Records land in otel-logs-v0_9 with service.name: nginx, and access-log fields arrive as OpenTelemetry HTTP attributes.

Prerequisites

A running Rootprint instance. Substitute its host for <your-rootprint>.
An ingest API key scoped to otel-logs-v0_9. Create one in Settings → API keys with Create ingest key, then substitute it for <your-ingest-api-key>. See API keys.
nginx writing its stock combined access log to /var/log/nginx/access.log and its error log to /var/log/nginx/error.log. These are the Debian/Ubuntu defaults, and no nginx.conf changes are needed.
The OpenTelemetry Collector or Vector installed on the nginx host. The agent pages cover installation.

Write the agent config

Replace <your-rootprint> and <your-ingest-api-key>.

OpenTelemetry Collector
Vector

Save this as /etc/otelcol-contrib/config.yaml.

Show config.yaml

receivers:
  file_log:
    include:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log
    start_at: end
    operators:
      - type: router
        routes:
          - expr: 'body matches "^[0-9]{4}/[0-9]{2}/[0-9]{2} "'
            output: parse_error
        default: parse_access

      - id: parse_access
        type: regex_parser
        parse_to: attributes
        regex: '^(?P<client>\S+) \S+ (?P<user>\S+) \[(?P<time>[^\]]+)\] "(?P<http_request_method>\S+) (?P<url_path>\S+) HTTP/(?P<network_protocol_version>\S+)" (?P<status>\d{3}) (?P<size>\d+) "(?P<referer>[^"]*)" "(?P<agent>[^"]*)"'
        timestamp:
          parse_from: attributes.time
          layout_type: gotime
          layout: '02/Jan/2006:15:04:05 -0700'
        output: end

      - id: parse_error
        type: regex_parser
        parse_to: attributes
        regex: '^(?P<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?P<level>\w+)\] (?P<pid>\d+)#(?P<tid>\d+): (?:\*(?P<cid>\d+) )?(?P<errmsg>.*)$'
        timestamp:
          parse_from: attributes.time
          layout_type: gotime
          layout: '2006/01/02 15:04:05'
        severity:
          parse_from: attributes.level
          overwrite_text: true
          mapping:
            debug: debug
            info: [info, notice]
            warn: warn
            error: error
            fatal: [crit, alert, emerg]
        output: end

      - id: end
        type: noop

processors:
  resource:
    attributes:
      - key: service.name
        value: nginx
        action: upsert
  transform:
    log_statements:
      - context: log
        statements:
          - set(attributes["nginx.format"], "error") where attributes["level"] != nil
          - set(attributes["nginx.format"], "access") where attributes["level"] == nil
          - set(severity_number, SEVERITY_NUMBER_INFO) where attributes["nginx.format"] == "access"
          - set(severity_text, "INFO") where attributes["nginx.format"] == "access"
          - set(attributes["http.request.method"], attributes["http_request_method"]) where attributes["http_request_method"] != nil
          - set(attributes["url.path"], attributes["url_path"]) where attributes["url_path"] != nil
          - set(attributes["network.protocol.name"], "http") where attributes["network_protocol_version"] != nil
          - set(attributes["network.protocol.version"], attributes["network_protocol_version"]) where attributes["network_protocol_version"] != nil
          - set(attributes["http.response.status_code"], Int(attributes["status"])) where attributes["status"] != nil
          - set(attributes["http.response.body.size"], Int(attributes["size"])) where attributes["size"] != nil
          - set(attributes["client.address"], attributes["client"]) where attributes["client"] != nil
          - set(attributes["user_agent.original"], attributes["agent"]) where attributes["agent"] != nil and attributes["agent"] != "-"
          - set(attributes["http.request.header.referer"], attributes["referer"]) where attributes["referer"] != nil and attributes["referer"] != "-"
          - set(attributes["enduser.id"], attributes["user"]) where attributes["user"] != nil and attributes["user"] != "-"
          - set(attributes["process.pid"], Int(attributes["pid"])) where attributes["pid"] != nil
          - set(attributes["thread.id"], Int(attributes["tid"])) where attributes["tid"] != nil
          - set(attributes["nginx.connection_id"], Int(attributes["cid"])) where attributes["cid"] != nil and attributes["cid"] != ""
          - delete_matching_keys(attributes, "^(time|level|client|user|http_request_method|url_path|network_protocol_version|status|size|referer|agent|pid|tid|cid)$")
  batch: {}

exporters:
  otlp_http:
    logs_endpoint: https://<your-rootprint>/v1/logs
    compression: gzip
    headers:
      Authorization: "Bearer <your-ingest-api-key>"

service:
  pipelines:
    logs:
      receivers: [file_log]
      processors: [resource, transform, batch]
      exporters: [otlp_http]

Save this as /etc/vector/vector.yaml.

Show vector.yaml

sources:
  nginx_logs:
    type: file
    include:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log
    read_from: end

transforms:
  parse_nginx:
    type: remap
    inputs: [nginx_logs]
    source: |
      # The file source's .host is Vector's hostname; drop it so it can't pose as the nginx host.
      del(.host)
      parsed, err = parse_nginx_log(.message, format: "combined")
      if err != null {
        parsed, err = parse_nginx_log(.message, format: "error")
      }
      if err == null && is_object(parsed) {
        . = merge(., object(parsed))
        .nginx_format = if exists(.severity) { "error" } else { "access" }
      }

      levels = {
        "debug": [5, "DEBUG"],
        "info": [9, "INFO"],
        "notice": [9, "INFO"],
        "warn": [13, "WARN"],
        "error": [17, "ERROR"],
        "crit": [21, "FATAL"],
        "alert": [21, "FATAL"],
        "emerg": [21, "FATAL"]
      }
      level = get(levels, [downcase(string(.severity) ?? "info")]) ?? [9, "INFO"]
      .severity_number = level[0]
      .severity_text = level[1]

  to_otlp:
    type: remap
    inputs: [parse_nginx]
    source: |
      msg       = string(.message) ?? ""
      file_path = string(.file) ?? ""
      fmt       = string(.nginx_format) ?? "access"

      ts_nano = to_unix_timestamp(now(), unit: "nanoseconds")
      if exists(.timestamp) && is_timestamp(.timestamp) {
        ts_nano = to_unix_timestamp!(.timestamp, unit: "nanoseconds")
      }

      sev_num  = to_int(.severity_number) ?? 9
      sev_text = string(.severity_text)   ?? "INFO"

      attrs = [
        { "key": "log.file.path",  "value": { "stringValue": file_path } },
        { "key": "nginx.format",   "value": { "stringValue": fmt } }
      ]

      if fmt == "access" {
        if exists(.request) {
          req_str = string!(.request)
          if req_str != "" && req_str != "-" {
            request_parts = split(req_str, " ")
            if length(request_parts) >= 1 {
              attrs = push(attrs, { "key": "http.request.method", "value": { "stringValue": request_parts[0] } })
            }
            if length(request_parts) >= 2 {
              attrs = push(attrs, { "key": "url.path", "value": { "stringValue": request_parts[1] } })
            }
            if length(request_parts) >= 3 {
              proto_parts = split!(request_parts[2], "/")
              if length(proto_parts) == 2 {
                attrs = push(attrs, { "key": "network.protocol.name",    "value": { "stringValue": downcase!(proto_parts[0]) } })
                attrs = push(attrs, { "key": "network.protocol.version", "value": { "stringValue": proto_parts[1] } })
              }
            }
          }
        }
        if exists(.status) {
          status_int, status_err = to_int(.status)
          if status_err == null {
            attrs = push(attrs, { "key": "http.response.status_code", "value": { "intValue": status_int } })
          }
        }
        if exists(.size) {
          size_int, size_err = to_int(.size)
          if size_err == null {
            attrs = push(attrs, { "key": "http.response.body.size", "value": { "intValue": size_int } })
          }
        }
        if exists(.client) {
          attrs = push(attrs, { "key": "client.address", "value": { "stringValue": string!(.client) } })
        }
        if exists(.agent) && string!(.agent) != "-" {
          attrs = push(attrs, { "key": "user_agent.original", "value": { "stringValue": string!(.agent) } })
        }
        if exists(.referer) && string!(.referer) != "-" {
          attrs = push(attrs, { "key": "http.request.header.referer", "value": { "stringValue": string!(.referer) } })
        }
        if exists(.user) && string!(.user) != "-" {
          attrs = push(attrs, { "key": "enduser.id", "value": { "stringValue": string!(.user) } })
        }
      } else {
        if exists(.pid) {
          pid_int, pid_err = to_int(.pid)
          if pid_err == null {
            attrs = push(attrs, { "key": "process.pid", "value": { "intValue": pid_int } })
          }
        }
        if exists(.tid) {
          tid_int, tid_err = to_int(.tid)
          if tid_err == null {
            attrs = push(attrs, { "key": "thread.id", "value": { "intValue": tid_int } })
          }
        }
        if exists(.cid) {
          cid_int, cid_err = to_int(.cid)
          if cid_err == null {
            attrs = push(attrs, { "key": "nginx.connection_id", "value": { "intValue": cid_int } })
          }
        }
        if exists(.client) {
          attrs = push(attrs, { "key": "client.address", "value": { "stringValue": string!(.client) } })
        }
        if exists(.host) {
          attrs = push(attrs, { "key": "server.address", "value": { "stringValue": string!(.host) } })
        }
        if exists(.server) {
          attrs = push(attrs, { "key": "nginx.server", "value": { "stringValue": string!(.server) } })
        }
        if exists(.upstream) {
          attrs = push(attrs, { "key": "nginx.upstream", "value": { "stringValue": string!(.upstream) } })
        }
      }

      . = {
        "resourceLogs": [{
          "resource": {
            "attributes": [
              { "key": "service.name", "value": { "stringValue": "nginx" } },
              { "key": "host.name",    "value": { "stringValue": get_hostname!() } }
            ]
          },
          "scopeLogs": [{
            "scope": { "name": "vector", "version": "" },
            "logRecords": [{
              "timeUnixNano":         ts_nano,
              "observedTimeUnixNano": to_unix_timestamp(now(), unit: "nanoseconds"),
              "severityNumber":       sev_num,
              "severityText":         sev_text,
              "body":                 { "stringValue": msg },
              "attributes":           attrs,
              "traceId":              "",
              "spanId":               "",
              "flags":                0,
              "droppedAttributesCount": 0
            }]
          }]
        }]
      }

sinks:
  rootprint:
    type: opentelemetry
    inputs: [to_otlp]
    protocol:
      type: http
      uri: https://<your-rootprint>/v1/logs
      method: post
      encoding:
        codec: otlp
      compression: gzip
      request:
        headers:
          Authorization: "Bearer <your-ingest-api-key>"
      batch:
        timeout_secs: 1
        max_bytes: 8388608

The Collector config (start_at: end) reads only lines written after each start, so it skips lines written while it was stopped. Vector (read_from: end) skips existing lines on first start only, then resumes from its checkpoint.

Grant read access to /var/log/nginx

Set AGENT_USER to the user your agent runs as, then run the lines for your distro. On RHEL, Fedora, and Amazon Linux the logs are nginx:root with mode 0640, so grant access with ACLs.

AGENT_USER=otelcol-contrib   # Vector: AGENT_USER=vector

# Debian/Ubuntu: nginx logs belong to the adm group
sudo usermod -aG adm "$AGENT_USER"

# RHEL/Fedora/Amazon Linux
sudo setfacl -m u:"$AGENT_USER":rx /var/log/nginx
sudo setfacl -m u:"$AGENT_USER":r /var/log/nginx/*.log
sudo setfacl -d -m u:"$AGENT_USER":r /var/log/nginx

The default ACL (-d) covers files logrotate creates later.

Restart the agent

Run sudo systemctl restart otelcol-contrib or sudo systemctl restart vector. The agent’s log should show no config or exporter errors.

Send a test request

Hit any site nginx serves:

curl -i http://localhost/

Verify in Rootprint

Open Logs, pick otel-logs-v0_9 from the index selector, and search for service_name:nginx. Records typically appear within 5–10 seconds.

What you get in Rootprint

Access-log entries get the HTTP attributes below, and error-log entries get process and connection attributes. nginx.format tells the two apart.

OTLP key	Source field	Format
`http.request.method`	first segment of the request line	access
`url.path`	second segment of the request line	access
`network.protocol.name`	protocol prefix of the request line, lowercased (`http`)	access
`network.protocol.version`	protocol version of the request line (e.g. `1.1`)	access
`http.response.status_code`	status code (integer)	access
`http.response.body.size`	response body size in bytes (integer)	access
`client.address`	client IP address	access
`user_agent.original`	`User-Agent` string (skipped if `-`)	access
`http.request.header.referer`	`Referer` header (skipped if `-`)	access
`enduser.id`	authenticated user (skipped if `-`)	access
`process.pid`	nginx worker PID (integer)	error
`thread.id`	nginx worker thread ID (integer)	error
`nginx.connection_id`	connection ID when present (integer)	error
`nginx.format`	`"access"` or `"error"`	both
`log.file.path`	source file path (Vector only)	both

Log level: access-log entries are always INFO, because the combined format has no level. Search on http.response.status_code to find server errors. Error-log levels map to DEBUG, INFO (info, notice), WARN, ERROR, and FATAL (crit, alert, emerg).
Timestamp: taken from each log line, with the agent’s read time as a fallback.
Error-log context: Vector also extracts client.address, server.address, nginx.server, and nginx.upstream from error lines. The Collector leaves them in body.

Useful searches

Find	Query
5xx responses	`service_name:nginx AND attributes.http.response.status_code:>=500`
4xx and 5xx under one path	`service_name:nginx AND attributes.url.path:/api/* AND attributes.http.response.status_code:>=400`
Error-log entries only	`service_name:nginx AND attributes.nginx.format:error`
Upstream failures (Vector only)	`service_name:nginx AND attributes.nginx.format:error AND attributes.nginx.upstream:*`

Troubleshooting

body is a raw nginx line and access entries have no http.* attributes: nginx.conf sets a custom log_format. The parsers expect the stock combined format. Remove the custom format or extend the parser.
401/403/413/415: see Response codes.

Send logs with the OpenTelemetry Collector and Send logs with Vector: install and run the agents.
Send Caddy logs: the same setup for Caddy.
OTLP reference: endpoint, response codes, limits.
Indexes: the otel-logs-v0_9 schema.

Get Started

Send Logs

Traces

Search Logs

Manage Rootprint

Prerequisites

What you get in Rootprint

Useful searches

Troubleshooting

​Prerequisites

​What you get in Rootprint

​Useful searches

​Troubleshooting

​Related

Prerequisites

What you get in Rootprint

Useful searches

Troubleshooting

Related