---
url: https://simpledatalogs.com/docs/integrations/log-shippers.md
description: >-
  Ship log files with Vector or Fluent Bit to /v1/ingest as gzipped NDJSON, with
  event ids so retries count once.
---

# Vector and Fluent Bit

Log shippers tail your files and post them in batches. Point their HTTP output at `/v1/ingest`, send NDJSON with gzip, and you're done.

## Vector

```yaml
transforms:
  add_ids:
    type: remap
    inputs: [app_logs]
    source: |
      .event_id = uuid_v4()
      .ts = format_timestamp!(.timestamp, format: "%+")
      del(.timestamp)

sinks:
  simpledatalogs:
    type: http
    inputs: [add_ids]
    uri: https://edge.simpledatalogs.com/v1/ingest
    method: post
    compression: gzip
    encoding:
      codec: json
    framing:
      method: newline_delimited
    request:
      headers:
        Content-Type: application/x-ndjson
      retry_attempts: 1000000
    auth:
      strategy: bearer
      token: sdl_sec_…
    batch:
      max_bytes: 262144
      timeout_secs: 5
```

* The ids are set in a transform, **before** the sink, so Vector's retries resend the same ids and never count twice.
* `level`, `message`, `device_id` and the other [documented fields](/api/events#fields) fill their columns; everything else becomes props.

Tested with Vector 0.43.1.

## Fluent Bit

```ini
[OUTPUT]
    Name              http
    Match             *
    Host              edge.simpledatalogs.com
    Port              443
    tls               On
    URI               /v1/ingest
    Format            json_lines
    Header            Authorization Bearer sdl_sec_…
    Header            Content-Type application/x-ndjson
    Json_date_key     ts
    Json_date_format  iso8601
    Compress          gzip
    Retry_Limit       no_limits
```

::: warning Fluent Bit retries can count twice
Fluent Bit has no built-in id generator, so a retried batch is stored again. If exact counts matter, add an `event_id` to each record in a Lua filter.
:::

Tested with Fluent Bit 3.2.2.

## Plain text instead

A shipper that can only post raw lines can send them as `text/plain`. Each line is [recognized on its own](/api/log-lines), stack traces join their entry, and personal data is masked. Text lines carry no `event_id`, so retries count twice.

## Next steps

* [Text log lines](/api/log-lines): the formats read without setup.
* [OpenTelemetry](/integrations/opentelemetry): if you already run a Collector.
* [Loki](/integrations/loki) and [Datadog logs](/integrations/datadog): if your shipper already speaks those.