Vector and Fluent Bit

Log shippers tail your files and post them in batches. Point their HTTP output at /v1/ingest, send NDJSON with gzip, and you're done.

Vector

transforms:
  add_ids:
    type: remap
    inputs: [app_logs]
    source: |
      .event_id = uuid_v4()
      .ts = format_timestamp!(.timestamp, format: "%+")
      del(.timestamp)

sinks:
  simpledatalogs:
    type: http
    inputs: [add_ids]
    uri: https://edge.simpledatalogs.com/v1/ingest
    method: post
    compression: gzip
    encoding:
      codec: json
    framing:
      method: newline_delimited
    request:
      headers:
        Content-Type: application/x-ndjson
      retry_attempts: 1000000
    auth:
      strategy: bearer
      token: sdl_sec_…
    batch:
      max_bytes: 262144
      timeout_secs: 5

• The ids are set in a transform, before the sink, so Vector's retries resend the same ids and never count twice.
• level, message, device_id and the other documented fields fill their columns; everything else becomes props.

Tested with Vector 0.43.1.

Fluent Bit

[OUTPUT]
    Name              http
    Match             *
    Host              edge.simpledatalogs.com
    Port              443
    tls               On
    URI               /v1/ingest
    Format            json_lines
    Header            Authorization Bearer sdl_sec_…
    Header            Content-Type application/x-ndjson
    Json_date_key     ts
    Json_date_format  iso8601
    Compress          gzip
    Retry_Limit       no_limits

Fluent Bit retries can count twice

Fluent Bit has no built-in id generator, so a retried batch is stored again. If exact counts matter, add an event_id to each record in a Lua filter.

Tested with Fluent Bit 3.2.2.

Plain text instead

A shipper that can only post raw lines can send them as text/plain. Each line is recognized on its own, stack traces join their entry, and personal data is masked. Text lines carry no event_id, so retries count twice.

Next steps

• Text log lines: the formats read without setup.
• OpenTelemetry: if you already run a Collector.
• Loki and Datadog logs: if your shipper already speaks those.