Deployment

Moving a Tiyi Controller into production takes four decisions: where secrets live, which remote data planes you need, where logs go, and how you recover and upgrade.

Production hardening

Two config fields have dev-friendly fallbacks that explicitly do not survive a restart. Set them before going live.

KEK — at-rest encryption key

crypto:
  kek_file: "/etc/tiyi/kek.bin"     # 32-byte at-rest Key Encryption Key

All envelope-encrypted blobs (TLS private keys, ACME account keys, ACME DNS provider credentials, the bundle signing key) decrypt with this KEK. Generate it once and back it up next to the state database:

$ install -m 0600 /dev/null /etc/tiyi/kek.bin
$ head -c 32 /dev/urandom > /etc/tiyi/kek.bin

JWT signing secret

auth:
  jwt_secret: "<32+ random bytes>"     # or set via TIYI_AUTH_JWT_SECRET

HS256 signing secret for access tokens. When the field is empty Tiyi generates a 32-byte ephemeral secret on every restart and prints a WARNING line. Sessions invalidate at restart, which is usually a problem in production.

Bundle signing key

The server's ed25519 bundle signing key is stored in the singleton bundle_signing_key row, envelope-encrypted with the KEK. Agents pin the public key on first contact (TOFU) and refuse to re-pin afterwards. Rotation requires full re-enrollment — issue fresh tokens with tiyi agents issue-token and bring agents back up.

One Controller, optional remote Agents

Run exactly one Controller with tiyi run. It always owns the writable SQLite state, API, UI, and one built-in local data plane. Add remote Agents for traffic scale or placement; adding them never changes the Controller's mode.

Install a remote Agent

Open Nodes → Install, choose the URL, tags, and token TTL, then issue the token. The page shows the binary download, raw token, recommended systemd setup, foreground command, and full script separately:

$ sudo curl -fsSL -o /usr/local/bin/tiyi 'https://tiyi.example.com/download/tiyi'
$ sudo chmod 0755 /usr/local/bin/tiyi
$ sudo mkdir -p /etc/tiyi
$ printf 'TIYI_CONTROLLER_URL=https://tiyi.example.com\nTIYI_AGENT_ENROLLMENT_TOKEN=<one-use-token>\n' | sudo tee /etc/tiyi/tiyi-agent.env >/dev/null
$ sudo chmod 0600 /etc/tiyi/tiyi-agent.env
$ sudo tiyi install --mode agent --unit-name tiyi-agent --now

The Nodes page then shows the local row first and the enrolled remote Agent after it.

Controller outage boundary

If the Controller stops, remote Agents continue proxying and enforcing their last accepted signed bundle. Configuration changes, enrollment, the central UI, and aggregated control-plane views are unavailable until the same logical Controller endpoint returns. Agents reconnect with backoff and resume convergence; there is no alternate Controller URL or automatic role transition.

Client-IP trust profile

If Tiyi sits behind a CDN or L4 LB, the client IP that reaches the WAF is not the real client IP — it's the proxy in front of you. The trust profile tells Tiyi which proxies are trusted and which header carries the real IP.

Set it once via the UI (Settings → Trust Profile tab) or the CLI:

$ tiyi trust set \
    --proxy 10.0.0.0/8 \
    --proxy 172.16.0.0/12 \
    --cdn cloudflare \
    --header CF-Connecting-IP \
    --header X-Forwarded-For

# Auto-fetch CDN ranges:
$ tiyi trust show
$ tiyi trust refresh cloudflare
$ tiyi trust refresh fastly

# Trace any (peer, headers) tuple to the resolved client IP:
$ tiyi trust test --peer 10.0.0.5 --header "X-Forwarded-For: 1.2.3.4, 10.0.0.5"

The Origin Bypass Attempt alert is seeded by default and fires when a request claims a client IP via XFF but the peer isn't a trusted proxy.

SIEM egress

Tiyi forwards security, access, error, and (optionally) audit events over best-effort TCP, UDP, or unixgram in RFC 5424, CEF, or LEEF. Configure once via Settings → SIEM tab or the CLI:

$ tiyi system settings update \
  --set siem.enabled=true \
  --set siem.endpoint=tcp://siem.internal:514 \
  --set siem.transport=tcp \
  --set siem.format=RFC5424 \
  --set siem.filter.include_audit=true

SIEM forwarding is intentionally best-effort. Receiver health, durable delivery, and replay live in your SIEM pipeline, not in Tiyi. The dispatcher caches a single net.Conn per dispatcher with reconnect-on-error.

Observability

• Prometheus exporter at /metrics on the local admin socket. Scrape it from a sidecar; metrics are derived from the same telemetry pipeline that drives the dashboard.
• /healthz reports the Controller identity plus database, proxy, configuration, site, and remote-Agent facts. Use it for service health checks.
• /debug/logsink/stats on the local admin socket exposes per-kind attempted/written/dropped_full/panicked/queue_depth/last_error counters. The panicked counter is the cross-layer canary for boundary defer recover handlers.
• Telemetry explorer UI at /telemetry/explorer for ad-hoc Top-K and sample browsing without leaving the dashboard.

Remote-Agent rolling upgrades

For compatible enrolled Agents, once a binary release is imported, tiyi release apply fans an APPLY_BINARY command out to every Agent whose OS/arch matches. Each Agent downloads the replacement, verifies the SHA-256 and Ed25519 release signature, and exits so its supervisor restarts it into the new binary.

$ tiyi release import --tarball ./tiyi_3.5.2_linux_amd64.tar.gz
$ tiyi release list
$ tiyi release apply <release-id>             # all matching agents
$ tiyi release apply <release-id> --agent-id A  # wave deployments
$ tiyi release runs                          # active rollouts
$ tiyi release rollback                      # revert to previous binary

Backup

Three things to back up:

• state.db — the SQLite control-plane database. WAL-mode; sqlite3 state.db ".backup '/path/to/backup.db'" works while Tiyi is running.
• kek.bin — the at-rest encryption key. Loss is unrecoverable. Back it up to the same vault you use for other long-lived secrets.
• jwt_secret — already in your config repo, but make sure that repo is backed up too.

Use SQLite's online backup mechanism or stop the Controller for a consistent filesystem copy. Copying a live WAL database directory through a network filesystem is not a supported failover design. logs/ partition files are operational data with their own retention loop.

Going-live checklist