Misuse, privacy and moderation

Privacy and securityCopy Anchor Link

Do not send data to a provider you have not vetted. Know its retention and training terms. Classify your data (public, internal, confidential, regulated) and set rules for which classes may reach which models. Redact personal data from prompts and logs. Pass an end-user identifier with each request, where the provider supports it, so abuse can be traced to one account instead of your whole product. Limit what the system can reach: least privilege applies to models as much as to people.

Misuse and contentCopy Anchor Link

Run a moderation check on inputs and outputs for the harmful categories that matter to your product. Many providers offer a free endpoint, and you can add your own classifier. Red-team the system: have people and scripts try to break the rules, including prompt injection, and add every success to your regression test set. Give users a way to report problems, and read the reports.

A worked example: a launch checklistCopy Anchor Link

For a public chat feature: a moderation call on every user message and every reply. An end-user id on every request. A daily spend cap and a rate limit per user, so an attacker cannot run up your bill. Logs with names and emails redacted, kept for 30 days. Fifty red-team prompts in the test set, rerun on every release. A visible "report this answer" button wired to a queue someone reads.

Why limits come in layersCopy Anchor Link

Invented numbers: a script sends 10 requests a second, and each request costs one cent. That is 10 cents a second, $360 an hour, $8,640 a day.

A per-user limit of 20 requests an hour caps one account at 20 cents an hour. But an attacker with 50 accounts can still spend $10 an hour. So also add a daily spend cap, for example $100, which bounds the worst case. Each layer covers a hole in the others.

What goes wrong: redaction missesCopy Anchor Link

Redaction replaces personal details with placeholders before text leaves your system. "Write to dana@example.com" becomes "Write to [EMAIL_1]". Your code keeps the mapping and restores it in the reply.

Pattern-based redaction handles emails and phone numbers well. Names are harder. Symptom: personal data shows up in your logs. To confirm, sample 100 log lines and count the leaks. Invented result: 3 of 100. Fix the patterns, repeat, and keep sampling.

What goes wrong: blocking too muchCopy Anchor Link

A moderation filter tuned to be strict also blocks a nurse asking about drug doses, or a novelist writing a crime scene. Symptom: complaints from legitimate users. Confirm by reading a sample of blocked messages. If many are harmless, loosen the setting for that category or add a human review path.

Common mistakeCopy Anchor Link

Relying on the system prompt alone to stop misuse. A prompt can be overridden. Rules that matter belong in code, permissions and monitoring.