LiteLLM

​

Overview

​

Architecture

• LiteLLM proxy — A stateless standard workload serving the OpenAI-compatible API and Admin UI on port 4000. Runs a single replica by default; set litellm.replicas to 2 or more for an always-on tier with zero-downtime rolling restarts. All state lives in PostgreSQL and Redis, so replicas are independent — no clustering.
• PostgreSQL (postgres subchart) — The durable store for virtual keys, spend, teams, and budgets. The DB-bundled proxy image runs its Prisma migrations automatically on boot.
• Redis + Sentinel (redis subchart, optional via redis.enabled) — Shared rate-limit / budget counters and response cache across proxy replicas.

​

What Gets Created

• Standard LiteLLM Workload — One or more stateless replicas serving the API and Admin UI on port 4000.
• Stateful PostgreSQL Workload — Single-replica Postgres with a persistent volume set for keys and spend.
• Redis Workloads (optional) — A Redis master-replica set with Redis Sentinel for rate limits and cache.
• Config Secret — The model/provider list mounted into the proxy as config.yaml.
• DB-credentials Secret — Assembles the proxy’s DATABASE_URL.
• Identity & Policy — An identity bound to the proxy with least-privilege reveal on exactly its own secrets and your prerequisite secret.

​

Prerequisites

• LITELLM_MASTER_KEY — the admin key (format sk-...); gates the Admin API/UI and mints virtual keys.
• LITELLM_SALT_KEY — encrypts provider keys stored in PostgreSQL. Write-once — never rotate it (rotating corrupts every stored provider key).
• one key per name in litellm.providerEnv (default: OPENAI_API_KEY) — the actual provider API key.

cpln secret create-dictionary --name my-litellm-secrets \
  --entry LITELLM_MASTER_KEY=sk-$(openssl rand -hex 24) \
  --entry LITELLM_SALT_KEY=sk-$(openssl rand -hex 24) \
  --entry OPENAI_API_KEY=sk-...your-openai-key...

​

Configuration

litellm:
  # DB-bundled image variant — runs Prisma `migrate deploy` on boot.
  image: ghcr.io/berriai/litellm-database:v1.93.0
  replicas: 1                 # 1 = proven single-replica; 2+ = always-on, zero-downtime restarts
  resources:
    minCpu: 250m
    maxCpu: 1000m
    minMemory: 1Gi
    maxMemory: 2Gi
  storeModelInDb: true        # also manage models via the Admin UI (persisted in Postgres)
  # Model/provider list seeded into config.yaml. api_key values reference env vars
  # supplied via the prerequisite secret (never inline real keys here).
  modelList:
    - model_name: gpt-4o-mini
      litellm_params:
        model: openai/gpt-4o-mini
        api_key: os.environ/OPENAI_API_KEY
  # Env-var NAMES (non-sensitive) exposed to the proxy from the prerequisite secret.
  # Every name here must exist as a key in the secrets.name dictionary secret.
  providerEnv:
    - OPENAI_API_KEY

# Prerequisite dictionary secret — MUST EXIST BEFORE INSTALL (see Prerequisites).
secrets:
  name: my-litellm-secrets

publicAccess:
  enabled: true               # key-gated *.cpln.app endpoint (UI + API); false = internal-only
internalAccess:
  type: same-gvc              # options: none, same-gvc, same-org, workload-list
  workloads: []               # used only with workload-list

# PostgreSQL — virtual keys / spend / budgets
postgres:
  image: postgres:18
  config:
    username: litellm
    password: change-me-litellm-pg     # change before installing
    database: litellm
  resources:
    minCpu: 250m
    maxCpu: 1000m
    minMemory: 512Mi
    maxMemory: 1Gi
  volumeset:
    capacity: 10              # initial capacity in GiB (minimum is 10)

# Redis (Sentinel HA) — rate limits / cache
redis:
  enabled: true               # false = in-memory per-replica rate limiting (not shared)
  redis:
    image: redis:7.4
    replicas: 3
    auth:
      password:
        enabled: false        # true = require AUTH; wires the master password into LiteLLM's cache config
        value: change-me-litellm-redis
  sentinel:
    image: redis:7.4
    replicas: 3

​

LiteLLM proxy

• litellm.image — The DB-bundled LiteLLM proxy image (litellm-database), which runs Prisma migrations on boot. Pin a concrete tag.
• litellm.replicas — Number of stateless proxy replicas. 1 is the proven single-replica shape; 2 or more gives an always-on tier where rolling restarts cycle one replica at a time with no downtime. Keep Redis enabled at 2+ so rate limits are shared across replicas.
• litellm.resources — CPU and memory per replica.
• litellm.storeModelInDb — When true, models added through the Admin UI/API are persisted in PostgreSQL alongside the static modelList.
• litellm.modelList — The model/provider list seeded into the mounted config.yaml. Each entry’s api_key references an environment variable name (os.environ/OPENAI_API_KEY) — never a literal key.
• litellm.providerEnv — The environment-variable names exposed to the proxy from the prerequisite secret. Every name here must exist as a key in that dictionary secret.

​

Prerequisite secret

• secrets.name — Name of the user-created dictionary secret holding LITELLM_MASTER_KEY, LITELLM_SALT_KEY, and every providerEnv key. It must exist before install — see Prerequisites.

​

Access

• publicAccess.enabled — Serve the API and Admin UI on the canonical *.cpln.app HTTPS endpoint (default). The endpoint is key-gated — a valid master or virtual key is always required. Set to false for an internal-only instance.
• internalAccess.type — Internal firewall scope of the proxy workload:

​

PostgreSQL

• postgres.config.password — The bundled database password. Change it before installing.
• postgres.volumeset.capacity — Initial volume size in GiB (minimum 10).

​

Redis

• redis.enabled — When true (default), deploys Redis + Sentinel for shared rate-limit counters and response cache. When false, no Redis is created and each proxy replica rate-limits in-memory (not shared across replicas).
• redis.redis.auth.password.enabled — When false (default), Redis runs authless with the same-GVC firewall as the boundary. Set to true to require AUTH; the master password is then wired into the proxy’s cache config. Both paths are tested end to end, including Sentinel monitoring the auth-protected master.
• redis.redis.replicas / redis.sentinel.replicas — Redis and Sentinel replica counts.

​

Connecting

​

Managing Virtual Keys

curl -X POST https://<canonical>.cpln.app/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"key_alias": "team-a", "max_budget": 10}'

​

Important Notes

• The prerequisite secret must exist before install. secrets.name is a dictionary secret with LITELLM_MASTER_KEY, LITELLM_SALT_KEY, and every providerEnv key. A missing secret wedges the deployment.
• LITELLM_SALT_KEY is write-once — never rotate it. It encrypts stored provider keys; changing it corrupts every stored key. The master key may be rotated.
• The endpoint is key-gated, not open. Public access still requires a valid master or virtual key — hand out virtual keys with budgets and limits rather than the master key.
• Keep Redis enabled for replicas ≥ 2. Without it, each replica rate-limits in-memory, so the effective limit becomes N× the configured value.
• Redis ships authless by default (same-GVC firewall is the boundary). Set redis.redis.auth.password.enabled: true to require AUTH; the master password is then wired into the proxy’s cache config.
• PostgreSQL data survives reinstalls of the proxy — to reset virtual keys and spend you must also reinstall the database (which deletes its volume set).
• First install self-heals a brief DB-timing gap. On a cold install the proxy can start before PostgreSQL accepts connections and log a P1001 error with one restart; it recovers automatically once the database is ready (about 1.5–2 minutes to healthy). No action needed.

​

External References