Vector is a high-performance observability data pipeline from Datadog that collects, transforms, and routes logs, metrics, and traces across heterogeneous backends. This guide deploys Vector using Docker Compose with Traefik handling automatic HTTPS for the GraphQL API and HTTP ingest endpoint, plus a working sources → transforms → sinks pipeline. By the end, you'll have Vector accepting JSON over HTTPS and forwarding it to multiple sinks on your server.
Set Up the Directory Structure and Configuration
1. Create the project directory structure:
$ mkdir -p ~/vector/{config,data}
$ cd ~/vector
2. Create the environment file:
$ nano .env
DOMAIN=vector.example.com
LETSENCRYPT_EMAIL=admin@example.com
3. Create the Vector pipeline configuration:
$ nano config/vector.yaml
api:
enabled: true
address: "0.0.0.0:8686"
sources:
demo_logs:
type: "demo_logs"
format: "syslog"
interval: 1.0
http_input:
type: "http_server"
address: "0.0.0.0:8080"
decoding:
codec: "json"
transforms:
parse_logs:
type: "remap"
inputs:
- "demo_logs"
- "http_input"
source: |
.processed_at = now()
.pipeline = "vector-demo"
sinks:
console_output:
type: "console"
inputs:
- "parse_logs"
encoding:
codec: "json"
file_output:
type: "file"
inputs:
- "parse_logs"
path: "/var/lib/vector/logs-%Y-%m-%d.log"
encoding:
codec: "json"
http_output:
type: "http"
inputs:
- "parse_logs"
uri: "https://httpbin.org/post"
encoding:
codec: "json"
batch:
max_bytes: 1048576
timeout_secs: 10
Deploy with Docker Compose
1. Create the Docker Compose manifest:
$ nano docker-compose.yaml
services:
traefik:
image: traefik:v3.6
container_name: traefik
command:
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--entrypoints.web.http.redirections.entrypoint.to=websecure"
- "--entrypoints.web.http.redirections.entrypoint.scheme=https"
- "--certificatesresolvers.letsencrypt.acme.httpchallenge=true"
- "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
- "--certificatesresolvers.letsencrypt.acme.email=${LETSENCRYPT_EMAIL}"
- "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
ports:
- "80:80"
- "443:443"
volumes:
- "./letsencrypt:/letsencrypt"
- "/var/run/docker.sock:/var/run/docker.sock:ro"
restart: unless-stopped
vector:
image: timberio/vector:0.44.0-alpine
container_name: vector
expose:
- "8080"
- "8686"
volumes:
- "./config/vector.yaml:/etc/vector/vector.yaml:ro"
- "./data:/var/lib/vector"
labels:
- "traefik.enable=true"
- "traefik.http.routers.vector-api.rule=Host(`${DOMAIN}`) && (PathPrefix(`/playground`) || PathPrefix(`/graphql`) || PathPrefix(`/health`))"
- "traefik.http.routers.vector-api.entrypoints=websecure"
- "traefik.http.routers.vector-api.tls.certresolver=letsencrypt"
- "traefik.http.routers.vector-api.service=vector-api"
- "traefik.http.services.vector-api.loadbalancer.server.port=8686"
- "traefik.http.routers.vector-ingest.rule=Host(`${DOMAIN}`) && PathPrefix(`/ingest`)"
- "traefik.http.routers.vector-ingest.entrypoints=websecure"
- "traefik.http.routers.vector-ingest.tls.certresolver=letsencrypt"
- "traefik.http.routers.vector-ingest.service=vector-ingest"
- "traefik.http.services.vector-ingest.loadbalancer.server.port=8080"
- "traefik.http.middlewares.strip-ingest.stripprefix.prefixes=/ingest"
- "traefik.http.routers.vector-ingest.middlewares=strip-ingest"
restart: unless-stopped
2. Start the services:
$ docker compose up -d
3. Verify the services are running:
$ docker compose ps
$ docker compose logs vector
Verify the Pipeline
1. POST a JSON log to the ingest endpoint:
$ curl -X POST https://vector.example.com/ingest \
-H "Content-Type: application/json" \
-d '{"level":"error","service":"api","message":"Database connection timeout","user_id":12345}'
2. Confirm the file sink wrote the event:
$ ls -lh data/
$ grep "Database connection timeout" data/logs-*.log
3. Stream the live console sink:
$ docker compose logs -f vector
Next Steps
Vector is running with HTTPS ingest and three sinks active. From here you can:
- Add sources for files, Kafka, syslog, journald, or Kubernetes logs
- Route to production sinks (Loki, Elasticsearch, S3, Datadog, Splunk)
- Use VRL (Vector Remap Language) for richer transforms and enrichment
For the full guide with additional tips, visit the original article on Vultr Docs.
