Operate

Troubleshooting

Diagnose from the outside in: process, listener, site match, route, WAF, then upstream. Change one layer at a time and retain the failing request ID.

First five minutes

tiyi doctor
systemctl status tiyi --no-pager
journalctl -u tiyi -n 200 --no-pager
ss -ltnp
tiyi system health

For an agent, inspect the tiyi-agent unit. For a foreground development run, inspect its terminal and command-line paths.

Dashboard does not open

Confirm the configured server.addr and host firewall.
Ports 80/443 serve proxy traffic; the dashboard listener defaults to 8080.
Run tiyi doctor for conflicts and config errors.
For reported service state ownership, review the path then run sudo tiyi doctor --fix-state-ownership.

Bootstrap password was lost

Do not delete state.db. Reset the existing account through the local admin socket; its filesystem permissions are the authentication boundary.

sudo tiyi user list
sudo tiyi user reset-password <user-id> --password '<new-strong-password>'

Request misses the site or wrong upstream answers

curl -v -H 'Host: app.example.com' http://127.0.0.1/
tiyi site list
tiyi upstream list

Check Host normalization, listener, site enabled state, longest-prefix route, upstream scheme/port, and health probe. A browser request to an IP without the configured Host is not a valid site test.

Expected attack is not blocked

Confirm the request reached the intended site and WAF is enabled.
Inspect effective policy, engine state, paranoia level, thresholds, bypasses, IP-list precedence, and path overrides.
Search Findings/Security evidence by request ID and rule ID.
Test from an untrusted source; a global/site allow or bypass may be decisive.
Preview compiled policy before changing it.

TLS or ACME fails

Check DNS, public reachability, port 80 for HTTP-01, certificate binding, clock, and ACME order details. DNS-01 needs a supported provider and correctly scoped credentials. Never include provider secrets in an issue or support bundle.

Agent is offline or will not apply

Compare server URL, enrollment expiry, clock, DNS, and network path.
Read server and agent journals around the same timestamp.
Distinguish offline, identity/protocol rejection, signature failure, and apply failure.
After reconnect, confirm applied revision/hash, not only online state.

v3.2 rejects old state

Observation v2 intentionally rejects pre-v3.2 databases, agent identity markers, and cached bundles. The project has no supported production upgrade population yet: preserve source manifests and required secrets, move the old development/test state aside, start v3.2 with a clean state, then re-enroll agents. Do not copy the old database back.

Counters exist but evidence or SIEM is late

Open Monitoring → Log Pipeline and inspect queue depth, drops, retries, and panic counters. Exact counters, retained evidence, and detail/SIEM delivery are independent. Test the destination from the Tiyi host and fix the consumer without restarting a healthy data plane unless diagnostics require it.

Collect a safe support bundle

Include version, mode, sanitized config, unit definition, health and doctor output, relevant journals, site/upstream IDs, timestamp/timezone, and request ID. Remove JWTs, passwords, enrollment tokens, private keys, DNS credentials, cookies, and sensitive bodies.