Send Nginx logs - Rootprint

nginx writes plain-text logs by default: a combined access log and a separate error log. This page tails both and ships them to Rootprint over OTLP with either the OpenTelemetry Collector or Vector. Records land in otel-logs-v0_9 tagged service.name: nginx, with access-log fields mapped to OpenTelemetry HTTP semantic conventions so they arrive as queryable attributes.

Prerequisites

A running Rootprint instance and its base URL — you’ll substitute it for <your-rootprint>.
A host running nginx that writes access and error logs to files. Stock nginx logs to /var/log/nginx/ on Debian/Ubuntu; the paths below assume those defaults.
A log agent that can read those files — the OpenTelemetry Collector or Vector.
An ingest API key scoped to your target index. See API keys.

OpenTelemetry Collector
Vector

This config assumes the agent runs where it can read /var/log/nginx.

Confirm nginx is logging to file

Stock nginx writes combined-format access logs to /var/log/nginx/access.log and error logs to /var/log/nginx/error.log. These defaults are what the config below expects. No nginx.conf changes are required.

Configure the Collector

Replace <your-rootprint> with your Rootprint base URL and <your-ingest-token> with the key from above.

Show config.yaml

receivers:
  filelog:
    include:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log
    start_at: end
    operators:
      - type: router
        routes:
          - expr: 'body matches "^[0-9]{4}/[0-9]{2}/[0-9]{2} "'
            output: parse_error
        default: parse_access

      - id: parse_access
        type: regex_parser
        parse_to: attributes
        regex: '^(?P<client>\S+) \S+ (?P<user>\S+) \[(?P<time>[^\]]+)\] "(?P<http_request_method>\S+) (?P<url_path>\S+) HTTP/(?P<network_protocol_version>\S+)" (?P<status>\d{3}) (?P<size>\d+) "(?P<referer>[^"]*)" "(?P<agent>[^"]*)"'
        timestamp:
          parse_from: attributes.time
          layout_type: gotime
          layout: '02/Jan/2006:15:04:05 -0700'
        output: end

      - id: parse_error
        type: regex_parser
        parse_to: attributes
        regex: '^(?P<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?P<level>\w+)\] (?P<pid>\d+)#(?P<tid>\d+): (?:\*(?P<cid>\d+) )?(?P<errmsg>.*)$'
        timestamp:
          parse_from: attributes.time
          layout_type: gotime
          layout: '2006/01/02 15:04:05'
        severity:
          parse_from: attributes.level
          mapping:
            debug: debug
            info: info
            warn: warn
            error: error
            fatal: [crit, alert, emerg]
        output: end

      - id: end
        type: noop

processors:
  resource:
    attributes:
      - key: service.name
        value: nginx
        action: upsert
  transform:
    log_statements:
      - context: log
        statements:
          - set(attributes["nginx.format"], "error") where attributes["level"] != nil
          - set(attributes["nginx.format"], "access") where attributes["level"] == nil
          - set(attributes["http.request.method"], attributes["http_request_method"]) where attributes["http_request_method"] != nil
          - set(attributes["url.path"], attributes["url_path"]) where attributes["url_path"] != nil
          - set(attributes["network.protocol.name"], "http") where attributes["network_protocol_version"] != nil
          - set(attributes["network.protocol.version"], attributes["network_protocol_version"]) where attributes["network_protocol_version"] != nil
          - set(attributes["http.response.status_code"], Int(attributes["status"])) where attributes["status"] != nil
          - set(attributes["http.response.body.size"], Int(attributes["size"])) where attributes["size"] != nil
          - set(attributes["client.address"], attributes["client"]) where attributes["client"] != nil
          - set(attributes["user_agent.original"], attributes["agent"]) where attributes["agent"] != nil and attributes["agent"] != "-"
          - set(attributes["http.request.header.referer"], attributes["referer"]) where attributes["referer"] != nil and attributes["referer"] != "-"
          - set(attributes["enduser.id"], attributes["user"]) where attributes["user"] != nil and attributes["user"] != "-"
          - set(attributes["process.pid"], Int(attributes["pid"])) where attributes["pid"] != nil
          - set(attributes["thread.id"], Int(attributes["tid"])) where attributes["tid"] != nil
          - set(attributes["nginx.connection_id"], Int(attributes["cid"])) where attributes["cid"] != nil
          - delete_key(attributes, "time")
          - delete_key(attributes, "status")
          - delete_key(attributes, "size")
          - delete_key(attributes, "agent")
          - delete_key(attributes, "referer")
          - delete_key(attributes, "user")
          - delete_key(attributes, "pid")
          - delete_key(attributes, "tid")
          - delete_key(attributes, "cid")
          - delete_key(attributes, "level")
          - delete_key(attributes, "http_request_method")
          - delete_key(attributes, "url_path")
          - delete_key(attributes, "network_protocol_version")
  batch: {}

exporters:
  otlphttp:
    logs_endpoint: https://<your-rootprint>/v1/logs
    compression: gzip
    headers:
      Authorization: "Bearer <your-ingest-token>"

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [resource, transform, batch]
      exporters: [otlphttp]

Grant read access to /var/log/nginx

sudo usermod -aG adm otelcol-contrib

nginx’s Debian/Ubuntu package owns /var/log/nginx/ with the adm group as the read-side convention. Add the user the Collector runs as to that adm group (the Debian/RPM Collector package creates otelcol-contrib).

Start the Collector

Start (or restart) the Collector so it loads the config. Startup logs should show no config-parse or exporter errors.

Send a test request

curl -i http://localhost/

Verify in Rootprint

Open Search, pick otel-logs-v0_9, and query service_name:nginx. Records typically appear within 5–10 seconds.

This config assumes the agent runs where it can read /var/log/nginx.

Confirm nginx is logging to file

Stock nginx already writes combined-format access logs to /var/log/nginx/access.log and error logs to /var/log/nginx/error.log. This guide assumes those defaults. No nginx.conf changes are required.

Write the Vector config

Save the following at /etc/vector/vector.yaml. Replace <your-rootprint> with your Rootprint base URL and <your-ingest-token> with the API key you copied above.

Show vector.yaml

sources:
  nginx_logs:
    type: file
    include:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log
    read_from: end

transforms:
  parse_nginx:
    type: remap
    inputs: [nginx_logs]
    source: |
      parsed, err = parse_nginx_log(.message, format: "combined")
      if err != null {
        parsed, err = parse_nginx_log(.message, format: "error")
      }
      if err == null && is_object(parsed) {
        . = merge(., object(parsed))
        .nginx_format = if exists(.severity) { "error" } else { "access" }
      }

      if .nginx_format == "error" {
        sev = downcase(string(.severity) ?? "info")
        if sev == "debug" {
          .severity_number = 5
          .severity_text   = "DEBUG"
        } else if sev == "info" {
          .severity_number = 9
          .severity_text   = "INFO"
        } else if sev == "notice" {
          .severity_number = 10
          .severity_text   = "NOTICE"
        } else if sev == "warn" {
          .severity_number = 13
          .severity_text   = "WARN"
        } else if sev == "error" {
          .severity_number = 17
          .severity_text   = "ERROR"
        } else if sev == "crit" {
          .severity_number = 19
          .severity_text   = "CRIT"
        } else if sev == "alert" {
          .severity_number = 21
          .severity_text   = "ALERT"
        } else if sev == "emerg" {
          .severity_number = 23
          .severity_text   = "EMERG"
        } else {
          .severity_number = 9
          .severity_text   = "INFO"
        }
      } else {
        .severity_number = 9
        .severity_text   = "INFO"
      }

  to_otlp:
    type: remap
    inputs: [parse_nginx]
    source: |
      msg       = string(.message) ?? ""
      file_path = string(.file) ?? ""
      fmt       = string(.nginx_format) ?? "access"

      ts_nano = to_unix_timestamp(now(), unit: "nanoseconds")
      if exists(.timestamp) && is_timestamp(.timestamp) {
        ts_nano = to_unix_timestamp!(.timestamp, unit: "nanoseconds")
      }

      sev_num  = to_int(.severity_number) ?? 9
      sev_text = string(.severity_text)   ?? "INFO"

      attrs = [
        { "key": "log.file.path",  "value": { "stringValue": file_path } },
        { "key": "nginx.format",   "value": { "stringValue": fmt } }
      ]

      if fmt == "access" {
        if exists(.request) {
          req_str = string!(.request)
          if req_str != "" && req_str != "-" {
            request_parts = split(req_str, " ")
            if length(request_parts) >= 1 {
              attrs = push(attrs, { "key": "http.request.method", "value": { "stringValue": request_parts[0] } })
            }
            if length(request_parts) >= 2 {
              attrs = push(attrs, { "key": "url.path", "value": { "stringValue": request_parts[1] } })
            }
            if length(request_parts) >= 3 {
              proto_parts = split!(request_parts[2], "/")
              if length(proto_parts) == 2 {
                attrs = push(attrs, { "key": "network.protocol.name",    "value": { "stringValue": downcase!(proto_parts[0]) } })
                attrs = push(attrs, { "key": "network.protocol.version", "value": { "stringValue": proto_parts[1] } })
              }
            }
          }
        }
        if exists(.status) {
          status_int, status_err = to_int(.status)
          if status_err == null {
            attrs = push(attrs, { "key": "http.response.status_code", "value": { "intValue": status_int } })
          }
        }
        if exists(.size) {
          size_int, size_err = to_int(.size)
          if size_err == null {
            attrs = push(attrs, { "key": "http.response.body.size", "value": { "intValue": size_int } })
          }
        }
        if exists(.client) {
          attrs = push(attrs, { "key": "client.address", "value": { "stringValue": string!(.client) } })
        }
        if exists(.agent) && string!(.agent) != "-" {
          attrs = push(attrs, { "key": "user_agent.original", "value": { "stringValue": string!(.agent) } })
        }
        if exists(.referer) && string!(.referer) != "-" {
          attrs = push(attrs, { "key": "http.request.header.referer", "value": { "stringValue": string!(.referer) } })
        }
        if exists(.user) && string!(.user) != "-" {
          attrs = push(attrs, { "key": "enduser.id", "value": { "stringValue": string!(.user) } })
        }
      } else {
        if exists(.pid) {
          pid_int, pid_err = to_int(.pid)
          if pid_err == null {
            attrs = push(attrs, { "key": "process.pid", "value": { "intValue": pid_int } })
          }
        }
        if exists(.tid) {
          tid_int, tid_err = to_int(.tid)
          if tid_err == null {
            attrs = push(attrs, { "key": "thread.id", "value": { "intValue": tid_int } })
          }
        }
        if exists(.cid) {
          cid_int, cid_err = to_int(.cid)
          if cid_err == null {
            attrs = push(attrs, { "key": "nginx.connection_id", "value": { "intValue": cid_int } })
          }
        }
        if exists(.client) {
          attrs = push(attrs, { "key": "client.address", "value": { "stringValue": string!(.client) } })
        }
        if exists(.host) {
          attrs = push(attrs, { "key": "server.address", "value": { "stringValue": string!(.host) } })
        }
        if exists(.server) {
          attrs = push(attrs, { "key": "nginx.server", "value": { "stringValue": string!(.server) } })
        }
        if exists(.upstream) {
          attrs = push(attrs, { "key": "nginx.upstream", "value": { "stringValue": string!(.upstream) } })
        }
      }

      . = {
        "resourceLogs": [{
          "resource": {
            "attributes": [
              { "key": "service.name", "value": { "stringValue": "nginx" } },
              { "key": "host.name",    "value": { "stringValue": get_hostname!() } }
            ]
          },
          "scopeLogs": [{
            "scope": { "name": "vector", "version": "" },
            "logRecords": [{
              "timeUnixNano":         ts_nano,
              "observedTimeUnixNano": to_unix_timestamp(now(), unit: "nanoseconds"),
              "severityNumber":       sev_num,
              "severityText":         sev_text,
              "body":                 { "stringValue": msg },
              "attributes":           attrs,
              "traceId":              "",
              "spanId":               "",
              "flags":                0,
              "droppedAttributesCount": 0
            }]
          }]
        }]
      }

sinks:
  rootprint:
    type: opentelemetry
    inputs: [to_otlp]
    protocol:
      type: http
      uri: https://<your-rootprint>/v1/logs
      method: post
      encoding:
        codec: otlp
      compression: gzip
      request:
        headers:
          Authorization: "Bearer <your-ingest-token>"
      batch:
        timeout_secs: 1
        max_bytes: 8388608

read_from: end skips existing content on first start, so installing Vector against an existing access.log does not replay every historical request. Flip it to beginning if you want a one-time backfill of the whole file. For a bounded backfill (capture recent history without replaying months of logs), pair read_from: beginning with ignore_older_secs: 600, which ignores any line older than 10 minutes on startup.

Grant Vector read access to /var/log/nginx

nginx’s Debian/Ubuntu package owns /var/log/nginx/ as www-data:adm with mode 640, so Vector’s vector user cannot read it. The adm group is the read-side convention for log files on those distros. Add vector to it:

sudo usermod -aG adm vector

Restart Vector

sudo systemctl restart vector
sudo systemctl status vector

The status output should show active (running) and the most recent log lines should not contain config-parse or sink-startup errors.

Send a test request

Hit any site nginx is serving so it writes a fresh access-log line:

curl -i http://localhost/

Replace http://localhost/ with whichever site URL nginx is configured for.

Verify in Rootprint

Open Search, pick otel-logs-v0_9 from the index selector, and query service_name:nginx. Records typically appear within 5–10 seconds.

What you get in Rootprint

Both agents parse nginx’s stock combined access log and error log format. Access-log fields are mapped to OpenTelemetry HTTP semantic conventions; error-log fields go to process and connection attributes. Both agents write to otel-logs-v0_9 with service.name: nginx. The nginx.format discriminator ("access" or "error") lets one config handle both files: error-log entries get process.pid, thread.id, and nginx.connection_id; access-log entries get the full HTTP attribute set.

OTLP key	Source field	Format
`http.request.method`	first segment of the request line	access
`url.path`	second segment of the request line	access
`network.protocol.name`	protocol prefix of the request line, lowercased (`http`)	access
`network.protocol.version`	protocol version of the request line (e.g. `1.1`)	access
`http.response.status_code`	status code (integer)	access
`http.response.body.size`	response body size in bytes (integer)	access
`client.address`	client IP address	access
`user_agent.original`	`User-Agent` string (skipped if `-`)	access
`http.request.header.referer`	`Referer` header (skipped if `-`)	access
`enduser.id`	authenticated user (skipped if `-`)	access
`process.pid`	nginx worker PID (integer)	error
`thread.id`	nginx worker thread ID (integer)	error
`nginx.connection_id`	connection ID when present (integer)	error
`nginx.format`	`"access"` or `"error"` discriminator	both
`log.file.path`	source file path (Vector only)	both

timeUnixNano is derived from the timestamp in each log line. If parsing fails, the agent’s read time is used as a fallback. With the OpenTelemetry Collector, error-log context fields (client.address on error lines, server.address, nginx.server, and nginx.upstream) are not broken out into attributes (they remain in the record body); the Vector path extracts them via parse_nginx_log.

Useful searches

Run these in the Rootprint search box against otel-logs-v0_9. Every 5xx response nginx returned, across all sites:

service_name:nginx AND attributes.http.response.status_code:>=500

All 4xx responses for one URL prefix, useful for spotting bad clients hammering one endpoint:

service_name:nginx AND attributes.url.path:/api/* AND attributes.http.response.status_code:>=400

Activity from a single source IP, pivot from a suspect address back to everything it touched:

attributes.client.address:1.2.3.4

Failed POSTs only:

service_name:nginx AND attributes.http.request.method:POST AND attributes.http.response.status_code:>=400

Error-log entries only (skip access logs):

service_name:nginx AND attributes.nginx.format:error

Upstream failures (the things that page someone). nginx.upstream is populated by the Vector path only (the OTEL Collector keeps error-log context in the record body):

service_name:nginx AND attributes.nginx.format:error AND attributes.nginx.upstream:*

Troubleshooting

Records arrive but attributes.http.* are missing: the entry came from error.log (no access fields), by design. Filter access-log entries with attributes.nginx.format:access.
body looks like a raw nginx line instead of a structured message: the access log is using a non-default log_format directive in nginx.conf. The parser expects nginx’s stock combined format. Either remove the custom log_format or extend the config to handle the alternate shape.
severity_text is always INFO even for 5xx responses: by design. nginx’s combined access format has no severity field; both agents trust the source. Filter on attributes.http.response.status_code:>=500 to surface server errors.
Ingest transport errors (401, 403, 413, 415): see OTLP reference → Response codes.

Send logs with Vector: generic Vector setup for tailing files on a host.
Send Caddy logs: sibling page for Caddy.
OTLP reference: endpoint URL, response codes, body limits.
Indexes: the otel-logs-v0_9 schema, so you know what you can search.

​Prerequisites

​What you get in Rootprint

​Useful searches

​Troubleshooting

​Related

Prerequisites

What you get in Rootprint

Useful searches

Troubleshooting

Related