N8N Execution Modes: Queue Mode & Worker Nodes voor Enterprise Scaling (2025)

Inhoudsopgave

• Execution Modes Overview
• Wanneer Queue Mode Gebruiken?
• Queue Mode Architecture
• Redis Setup & Configuration
• Worker Nodes Configuration
• Complete Docker Compose Setup
• Kubernetes Deployment
• Load Balancing Strategies
• Monitoring & Auto-scaling
• Performance Tuning
• Troubleshooting

N8N Execution Modes Overview

N8N kent drie execution modes, elk met specifieke use cases en schaalbaarheidseigenschappen:

Wanneer Queue Mode Gebruiken?

Queue mode is niet voor iedereen. Hier zijn de indicatoren dat je ready bent voor queue mode:

Queue Mode Architecture Deep Dive

Queue mode splitst N8N in specialized components voor optimale performance:

Redis Setup & Configuration

Redis is het hart van queue mode. Proper configuratie is cruciaal voor stability:

# Redis Configuration for N8N Queue Mode
# Version: Redis 7.2+

# Network
bind 0.0.0.0
protected-mode yes
port 6379
tcp-backlog 511
tcp-keepalive 300
timeout 0

# General
daemonize no
loglevel notice
logfile /var/log/redis/redis-server.log
databases 16

# Persistence (AOF recommended for queue)
save ""
stop-writes-on-bgsave-error no
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /data

# AOF (Append Only File)
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

# Memory Management
maxmemory 2gb
maxmemory-policy allkeys-lru
maxmemory-samples 5

# Lazy Freeing
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
lazyfree-lazy-user-del no

# Threading
io-threads 4
io-threads-do-reads yes

# Slow Log
slowlog-log-slower-than 10000
slowlog-max-len 128

# Client Management
maxclients 10000

# Security
requirepass your-strong-redis-password-here

# ACL for N8N (Redis 6+)
aclfile /etc/redis/users.acl

# /etc/redis/users.acl
# ACL rules for N8N Queue Mode

# Default user (disabled)
user default on nopass ~* &* +@all

# N8N Main instance user
user n8n-main on +@all ~* &* >main-instance-password

# N8N Worker user (limited permissions)
user n8n-worker on ~job:* ~queue:* ~worker:* +@read +@write +@list +@stream >worker-password

# Monitoring user (read-only)
user monitoring on ~* +@read +ping +info +config|get >monitoring-password

Worker Nodes Configuration

Workers zijn de execution engines. Optimale configuratie bepaalt throughput:

# Essential Worker Configuration
EXECUTIONS_MODE=queue

# Redis Connection
QUEUE_BULL_REDIS_HOST=redis.internal
QUEUE_BULL_REDIS_PORT=6379
QUEUE_BULL_REDIS_PASSWORD=your-redis-password
QUEUE_BULL_REDIS_DB=0
QUEUE_BULL_REDIS_TIMEOUT=60000

# Redis Cluster (optional)
# QUEUE_BULL_REDIS_CLUSTER_NODES=redis-1:6379,redis-2:6379,redis-3:6379

# Worker Performance
QUEUE_WORKER_PARALLELISM=10  # Concurrent jobs per worker
QUEUE_WORKER_LOCK_DURATION=30000  # Job lock timeout (ms)
QUEUE_WORKER_LOCK_RENEW_TIME=15000  # Lock renewal interval
QUEUE_WORKER_STALLED_INTERVAL=30000  # Check for stalled jobs
QUEUE_WORKER_MAX_STALLED_COUNT=1  # Max stalls before failing

# Health & Monitoring
QUEUE_HEALTH_CHECK_ACTIVE=true
QUEUE_HEALTH_CHECK_PORT=8081
QUEUE_WORKER_HEALTH_CHECK_INTERVAL=30000

# Resource Limits
NODE_OPTIONS="--max-old-space-size=4096"
EXECUTIONS_TIMEOUT=3600  # 1 hour max execution time
EXECUTIONS_TIMEOUT_MAX=7200  # 2 hours absolute max

# Concurrency Control
N8N_CONCURRENCY_PRODUCTION_LIMIT=10  # Per worker

# Database (shared with main)
DB_TYPE=postgresdb
DB_POSTGRESDB_HOST=postgres.internal
DB_POSTGRESDB_PORT=5432
DB_POSTGRESDB_DATABASE=n8n
DB_POSTGRESDB_USER=n8n
DB_POSTGRESDB_PASSWORD=db-password

# Encryption Key (MUST match main instance)
N8N_ENCRYPTION_KEY=your-32-byte-hex-encryption-key

Workers Required = (Peak Workflows/Min × Avg Execution Time) / 60 / Worker Parallelism

Example:
- Peak: 600 workflows/min
- Avg time: 30 seconds
- Parallelism: 10 jobs/worker

Workers = (600 × 30) / 60 / 10 = 30 workers needed

Complete Docker Compose Setup

Production-ready Docker Compose configuration voor queue mode:

version: '3.8'

services:
  # PostgreSQL Database
  postgres:
    image: postgres:15-alpine
    restart: unless-stopped
    environment:
      POSTGRES_DB: n8n
      POSTGRES_USER: n8n
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - n8n_internal
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U n8n"]
      interval: 10s
      timeout: 5s
      retries: 5

  # Redis Queue
  redis:
    image: redis:7-alpine
    restart: unless-stopped
    command: redis-server /usr/local/etc/redis/redis.conf
    volumes:
      - ./redis.conf:/usr/local/etc/redis/redis.conf:ro
      - redis_data:/data
    networks:
      - n8n_internal
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 5

  # N8N Main Instance
  n8n-main:
    image: n8nio/n8n:latest
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=redis
      - QUEUE_BULL_REDIS_PORT=6379
      - QUEUE_BULL_REDIS_PASSWORD=${REDIS_PASSWORD}
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=${POSTGRES_PASSWORD}
      - N8N_ENCRYPTION_KEY=${ENCRYPTION_KEY}
      - N8N_HOST=${N8N_HOST:-localhost}
      - N8N_PROTOCOL=${N8N_PROTOCOL:-https}
      - WEBHOOK_URL=${WEBHOOK_URL:-https://localhost:5678/}
      - N8N_METRICS=true
    volumes:
      - n8n_data:/home/node/.n8n
    networks:
      - n8n_internal
      - n8n_external
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.n8n.rule=Host(`${N8N_HOST}`)"
      - "traefik.http.routers.n8n.tls=true"
      - "traefik.http.services.n8n.loadbalancer.server.port=5678"

  # N8N Worker Instances
  n8n-worker:
    image: n8nio/n8n:latest
    restart: unless-stopped
    command: worker
    environment:
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=redis
      - QUEUE_BULL_REDIS_PORT=6379
      - QUEUE_BULL_REDIS_PASSWORD=${REDIS_PASSWORD}
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=${POSTGRES_PASSWORD}
      - N8N_ENCRYPTION_KEY=${ENCRYPTION_KEY}
      - QUEUE_WORKER_PARALLELISM=10
      - N8N_CONCURRENCY_PRODUCTION_LIMIT=10
      - NODE_OPTIONS=--max-old-space-size=2048
    networks:
      - n8n_internal
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
      n8n-main:
        condition: service_started
    deploy:
      replicas: 5  # Start with 5 workers
      resources:
        limits:
          cpus: '2'
          memory: 2G
        reservations:
          cpus: '1'
          memory: 1G
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
        window: 120s

  # Traefik Load Balancer
  traefik:
    image: traefik:v3.0
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"  # Dashboard
    command:
      - --api.dashboard=true
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --certificatesresolvers.letsencrypt.acme.email=${ACME_EMAIL}
      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - traefik_certs:/letsencrypt
    networks:
      - n8n_external

  # Monitoring Stack (Optional)
  prometheus:
    image: prom/prometheus:latest
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
    networks:
      - n8n_internal
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
    networks:
      - n8n_internal
    ports:
      - "3000:3000"

volumes:
  postgres_data:
  redis_data:
  n8n_data:
  traefik_certs:
  prometheus_data:
  grafana_data:

networks:
  n8n_internal:
    driver: bridge
    internal: true
  n8n_external:
    driver: bridge

Kubernetes Deployment voor Enterprise Scale

Voor enterprise deployments is Kubernetes de beste keuze met auto-scaling:

# n8n-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: n8n-production

---
# n8n-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: n8n-config
  namespace: n8n-production
data:
  EXECUTIONS_MODE: "queue"
  DB_TYPE: "postgresdb"
  N8N_METRICS: "true"
  QUEUE_WORKER_PARALLELISM: "10"

---
# n8n-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: n8n-secrets
  namespace: n8n-production
type: Opaque
stringData:
  encryption-key: "your-32-byte-hex-key"
  postgres-password: "strong-password"
  redis-password: "redis-password"

---
# n8n-main-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: n8n-main
  namespace: n8n-production
spec:
  replicas: 2  # HA setup
  selector:
    matchLabels:
      app: n8n-main
  template:
    metadata:
      labels:
        app: n8n-main
    spec:
      containers:
      - name: n8n
        image: n8nio/n8n:latest
        ports:
        - containerPort: 5678
        envFrom:
        - configMapRef:
            name: n8n-config
        env:
        - name: N8N_ENCRYPTION_KEY
          valueFrom:
            secretKeyRef:
              name: n8n-secrets
              key: encryption-key
        - name: DB_POSTGRESDB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: n8n-secrets
              key: postgres-password
        - name: QUEUE_BULL_REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: n8n-secrets
              key: redis-password
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1000m"
        livenessProbe:
          httpGet:
            path: /healthz
            port: 5678
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /healthz
            port: 5678
          initialDelaySeconds: 5
          periodSeconds: 5

---
# n8n-worker-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: n8n-worker
  namespace: n8n-production
spec:
  replicas: 10  # Initial workers
  selector:
    matchLabels:
      app: n8n-worker
  template:
    metadata:
      labels:
        app: n8n-worker
    spec:
      containers:
      - name: n8n-worker
        image: n8nio/n8n:latest
        command: ["n8n", "worker"]
        envFrom:
        - configMapRef:
            name: n8n-config
        env:
        - name: N8N_ENCRYPTION_KEY
          valueFrom:
            secretKeyRef:
              name: n8n-secrets
              key: encryption-key
        - name: DB_POSTGRESDB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: n8n-secrets
              key: postgres-password
        - name: QUEUE_BULL_REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: n8n-secrets
              key: redis-password
        resources:
          requests:
            memory: "1Gi"
            cpu: "1000m"
          limits:
            memory: "2Gi"
            cpu: "2000m"

---
# n8n-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: n8n-worker-hpa
  namespace: n8n-production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: n8n-worker
  minReplicas: 5
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Percent
        value: 100
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60

Load Balancing Strategies

Effective load balancing is cruciaal voor performance en reliability:

# Nginx config snippet
upstream n8n_main {
    least_conn;
    server n8n-main-1:5678 weight=1;
    server n8n-main-2:5678 weight=1;
    keepalive 32;
}

# Worker groups
HEAVY_WORKERS=10  # For data processing
FAST_WORKERS=20   # For quick tasks
API_WORKERS=5     # For API calls

Monitoring & Auto-scaling

Real-time monitoring en intelligent auto-scaling voor optimale resource gebruik:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'n8n-main'
    static_configs:
      - targets: ['n8n-main:5678']
    metrics_path: /metrics

  - job_name: 'n8n-workers'
    dns_sd_configs:
      - names:
          - 'n8n-worker'
        type: 'A'
        port: 5678
    metrics_path: /metrics

  - job_name: 'redis'
    static_configs:
      - targets: ['redis:9121']

  - job_name: 'postgres'
    static_configs:
      - targets: ['postgres-exporter:9187']

rule_files:
  - 'alerts.yml'

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

Performance Tuning

Fine-tune je queue mode setup voor maximale performance:

# PostgreSQL tuning
max_connections = 200
shared_buffers = 4GB
work_mem = 16MB
maintenance_work_mem = 512MB

# Redis performance
maxmemory-policy volatile-lru
io-threads 4
io-threads-do-reads yes

Troubleshooting Common Issues

Oplossingen voor veelvoorkomende problemen in queue mode:

# Debug worker connection
docker exec n8n-worker nc -zv redis 6379
docker exec n8n-worker nc -zv postgres 5432

# Clean up old jobs
redis-cli EVAL "return redis.call('del', unpack(redis.call('keys', 'bull:*:completed')))" 0

Best Practices Checklist

Gebruik deze checklist voor production-ready queue mode deployment:

Conclusie

Queue mode transformeert N8N van een simpele automation tool naar een enterprise-ready platform. Met proper configuratie van Redis, worker nodes en monitoring kun je schalen naar duizenden concurrent workflows zonder performance degradatie.

Start klein met 2-3 workers, monitor je metrics, en scale gradually based op actual usage. Met de configuraties in deze guide heb je een solid foundation voor production workloads.

Need help met enterprise N8N deployment? Contact me voor professional setup en scaling consultancy!