MLOps and LLMOps basics
Getting a model or an AI feature into production and keeping it healthy is its own set of skills. It is mostly standard DevOps plus tracking things that change.
Words on this page (11)
- skill
- : A folder of instructions and scripts an agent loads only when a task needs it.
- prompt
- : The input text you give a model, including instructions, examples and data.
- eval
- : A repeatable test of AI output quality, run before and after every change.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
- git
- : The standard tool for tracking changes to code.
- weights
- : The learned numbers that define a trained model. A model whose weights are published is called open-weight.
- Docker
- : A tool that packages software and its dependencies into a portable container.
- CI
- : Continuous integration: automatically running tests on every code change. CD deploys passing changes.
- logarithm
- : The power you raise a base to in order to get a number. log base 2 of 8 is 3 because 2 to the power 3 is 8.
- trace
- : The full record of one run: inputs, model calls, tool calls, outputs, timing and cost.
- latency
- : The time you wait for a response.
What is different about AI
Three things change: code, data and the model or . Any of them can break behavior. So you version data and prompts as well as code, track experiments, and test with as well as unit tests. Common are for code, DVC for data and pipelines, MLflow or and Biases for experiment tracking, and for packaging. (continuous integration) runs tests on every change, and CD (continuous delivery) deploys the changes that pass. Dashboards watch it all in production.
A minimal setup
Pin model versions. Keep prompts in files under version control. Run the eval set in CI and block merges that regress. every call with . Add alerts on error rate, and cost. Keep a rollback path for prompt and model changes.
A worked example: tracing a regression
Pass rate fell from 92 to 84 on 14 March. The log shows: prompt v18 deployed 12 March, model upgraded 13 March, new documents indexed 14 March. Rerun the eval set with each change reverted in turn. Reverting the model alone restores 92. You now know what to fix.
How an eval gate works
- A change opens a pull request.
- CI runs the 100-case eval set against the changed prompt or model.
- It compares the pass rate with the stored result for the main branch.
- If the pass rate drops by more than a set amount, the merge is blocked.
Model output varies, so choose the amount with care. Run the unchanged main branch twice. If you get 91 and 93, a drop of two points is noise. A gate set at one point would block good changes at random. Set it above the natural wobble, such as 4 or 5 points, and look at a bigger eval set if you need to catch smaller drops.
What goes wrong: the model changed and you did not
Your code names a model by an alias that the provider moves to a newer version. The pass rate falls with no commit in the history to explain it. Confirm it by logging the exact model identifier on each call and checking whether it changed on the date the pass rate dropped. Pin the dated version, and move to a new one as a change you test.
When this is too much
A solo project with one prompt and one model does not need DVC or an experiment tracker. Git for the prompt files and a script that runs 30 test cases is enough. Add tools when you feel a specific pain, such as two people overwriting each other's data files.
Common mistake
Changing several things between releases, so you cannot tell which caused a change.
Six months after launch, accuracy drops and nobody knows why. The data changed, a prompt was edited, and a model version updated, all on different days with no record. Teams with versioned data, versioned prompts and a tracked experiment log find the cause in an hour. Others guess for a week.
Check yourself
What three things can change an AI system's behavior?
Not quite. All three need versioning and tests.
What should CI run for an LLM feature?
Not quite. The eval set catches quality regressions that unit tests miss.
Accuracy falls after a week with a new prompt, a model update and new documents. How do you find the cause?
Not quite. Versioned changes let you isolate the cause.
You got 0 of 3 right.
Go deeper
- CourseMade With MLCourse
Goku Mohandas · Free, practical MLOps course with code.
- Repo
MLflow · Experiment tracking and model registry.
- Repo
Iterative · Version control for data and pipelines.
- Repo
DataTalks.Club · Free course covering the full production flow.
Spotted a mistake or something unclear? Report it on GitHub.