Deploying an AI app

BasicsCopy Anchor Link

Keep secrets in environment variables set on the host, not in the repo. Build in a clean environment with a pinned Node or Python version. Push to GitHub and let the host build from it. After each deploy run a smoke test: one real request through the whole path, including a model call. Use a staging environment with its own keys and spend cap, and deploy to production only what passed staging.

Health and signalsCopy Anchor Link

Add a health endpoint that returns 200 when the app can reach its database and its model provider's credentials work. Platforms use it to restart unhealthy instances. Handle SIGTERM: stop accepting new requests, finish in-flight ones including streaming responses, close connections and then exit. Without this, deploys cut users off mid-answer.

AI-specificCopy Anchor Link

Set per-user and global rate limits and a daily spend cap, since a loop or an abuser can turn into a large bill. Alert on error rate, latency, token usage and cost. Keep a fallback model. Store prompts in version control and tag each deploy with the prompt and model versions. Expect provider outages and show a useful message.

Domains and dataCopy Anchor Link

Point a custom domain with DNS and HTTPS. Back up the database and test the restore. Run migrations before the new code needs them.

A worked example: a safe releaseCopy Anchor Link

1. Merge to main; the host builds.
2. Deploy to staging with its own keys and a low spend cap.
3. Smoke test: ask one real question.
4. Run the eval set against staging.
5. Promote to production.
6. Watch error rate, latency and cost for fifteen minutes.
7. If anything looks wrong, roll back to the previous version. Each step is boring and each has saved someone.

Long answers and time limitsCopy Anchor Link

A model answer can take 20 seconds or more. Hosts and proxies often cut requests that run longer than their time limit. The symptom is a feature that works on your laptop and, in production, long answers fail with a gateway timeout or stop mid-sentence. Confirm it by asking for a deliberately long answer on the deployed app, timing it, and reading the host's logs. Fixes are to stream the answer, raise the limit, or move the job to the background and notify the user.

What goes wrong: a spend cap that does not capCopy Anchor Link

Say each user gets a 5 dollar daily limit, counted in memory inside the app. You run 3 copies of the app, and each keeps its own count. A user can spend up to 3 × 5 = 15 dollars. Restarting a copy resets its count to zero as well. The symptom is a bill that is higher than the cap implies. Keep the counter in a shared database, or set a hard limit on the provider's side as well.

Rollback and the databaseCopy Anchor Link

You deploy new code that needs a new column, and the migration drops an old one. The release goes wrong and you roll back the code. The old code expects the dropped column and crashes. To keep rollback safe, change the database in two releases. First add the new column and keep the old one. Remove the old one only after the new code has been stable for a while.

Common mistakeCopy Anchor Link

No cap on spending. A bug can turn a quiet day into an expensive one.