We run a small AI support bot on our own VPS. It answers product and pricing questions, streams replies into a browser widget, and costs nothing per month on the default model. That last sentence is only true because of eight controls that have nothing to do with the model.
The model is the cheap part. What produces a surprise invoice is an uncapped public endpoint, a retry loop, or a guardrail that silently became NaN three weeks ago and nobody noticed. This is the checklist we wish we had before shipping. No benchmarks, no cost percentages, no vendor endorsements — just the failure modes we hit and what we changed.
A single global daily cap is not enough. One visitor can consume the entire day's budget before lunch. We pre-check two independent ceilings on the routes that actually spend money:
Both are checked before the upstream call, not after. Only the expensive routes get the check, so a normal page load never touches the quota state and the visitor sees no difference. When a limit is exceeded the API returns a stable error code that the widget can interpret; nothing pops up on page load.
When the paid provider cap is hit, we do not show an error and we do not stop answering. The request falls back to a keyless endpoint, and the system prompt adds a short nudge toward the paid plan. The visitor still gets a useful answer; the economics still hold.
The same pattern applies to free trials. Count usage per IP per day, and once the allowance is used, switch the model rather than locking the UI. A hard wall at request five converts nobody. A slower-but-working answer plus an honest upgrade line converts some.
One subtle bug worth naming: if you increment the counter before checking it, the first request of the day can be misclassified as over-limit. Compare against the value after the read, and write the assertion as an explicit off-by-one test, not as a comment.
Caps protect the budget; these three protect the process:
| Control | What it stops |
|---|---|
| Requests per window per IP | Scripts hammering the endpoint, accidental retry storms |
| Max conversation turns | A single session growing an unbounded message array |
| Max input characters | Someone pasting a whole document into the prompt |
All three are plain environment variables. None of them need a database. A standard-library counter with a timestamp is enough for a small site, and it keeps the deployment dependency-free.
We wanted a typewriter effect. The obvious approach is to ask the upstream for stream:true and forward the deltas. On at least one keyless OpenAI-compatible endpoint, that returns HTTP 200 and emits no content deltas at all. The widget sits there empty and you spend an afternoon debugging CORS before you look at the payload.
What works reliably:
upstream : POST /chat/completions { stream: false, messages: [...] }
server : read the full text, split into small chunks
wire : push chunks as SSE `data:` events to the browser
browser : same contract as a real streamed response
Zero dependencies, deterministic, and it survives upstreams with partial streaming support. The tradeoff is time-to-first-token; for a support widget answering short questions, that is a good trade.
If you terminate TLS at Caddy and reverse-proxy the app, disable response buffering for that route or your carefully sliced SSE events arrive as one lump. A minimal site block looks like this:
handle_path /bot/* {
reverse_proxy 127.0.0.1:8795 {
flush_interval -1
}
}
Also add a redirect for the extensionless path so /bot and /bot/ do not behave differently. It is a one-line matcher, and it prevents a class of "works on my machine" reports.
This one cost us the most time because it fails silently. On systemd, an Environment=KEY=value line in the unit takes precedence over the same key loaded from EnvironmentFile=. If a loader only fills empty values, you can end up with a signing key from one place and a verification key from another, and every HMAC check quietly fails. No crash, no log line.
Rules we now follow:
.env with 600 permissions, owned by the service user..env is excluded from every deployment archive. Overwriting it is how you lose live credentials./proc/<pid>/environ.EnvironmentFile= does not parse inline comments. This line:
TRIAL_LIMIT=5 # free requests per IP per day
sets the value to the literal string 5 # free requests per IP per day. Converted with Number() it becomes NaN, and every comparison against NaN is false — so the trial limit is simply off, forever, with no error. We strip everything from the first # before parsing, and we assert the numeric guardrails at startup so a bad value fails loudly instead of quietly.
Our knowledge base is loaded once into a module-level constant when the process starts. The HTML page is read per request; the knowledge base is not. So editing the KB file and reloading the browser changes nothing, and it looks like the model ignored your instructions.
Either restart the service after a KB edit, or load it per request with a cheap mtime check. We chose the restart and documented it, because a single read is simpler than a cache invalidation strategy. The important part is that the difference is written down.
For reference, the whole thing is: one Python standard-library HTTP server on a loopback port, one systemd unit, one Caddy site block, one .env, and one text file for the system prompt. No database, no queue, no container. A keyless OpenAI-compatible endpoint handles the default model; a paid provider key is an optional upgrade selected by an environment variable, with the caps above deciding whether it is actually used on any given request.
The paid path is deliberately optional. Adding a provider key should change answer quality, not the safety properties of the system.
Two honest limits:
If you want a hosted widget with none of this operational surface, that is a different product with a different tradeoff. Self-hosting buys control and a near-zero marginal cost; it costs you the setup.
The server and a keyless or free-tier endpoint can cost nothing monthly. Real costs are an optional paid provider key and your setup time. Without caps, a paid key plus a public endpoint can produce an unplanned bill. Cap first, then upgrade.
Some free endpoints accept stream:true but never emit content deltas. Request with stream:false, take the full text, slice it server-side and push SSE chunks. The browser still gets a typewriter effect.
Usually a systemd Environment= line overriding EnvironmentFile, or an inline comment turning a numeric .env value into NaN. Parse explicitly and assert guardrails at startup.
For a public widget, degrade to a cheaper provider instead. A hard error or a modal adds friction and recovers no revenue. Pre-check quotas on expensive routes only.
A static QR code takes payment but has no callback, so activation is manual. Automatic activation needs a payment provider API and usually a business account.
Everything above is in production on this domain. You can talk to the bot for free, or take the source kit — a runnable server, deployment notes, the cost-guardrail regression tests and a compliance checklist — and deploy it on your own VPS. It is source code, not a hosted service: you need basic Python and terminal access.
Try the free assistant Get the source kit on Gumroad See all pricingDisclosure: the source kit linked above is our own paid product. Price is shown on that product page; this article does not quote it. Everything in the checklist above is published here so you can build the same controls yourself — you do not need to buy anything to use it. No performance or earnings claims are made.