Changing an agent is easy. A new instruction, a different model, one more app. Knowing the change didn't break something it used to do well is the hard part. Three things in Definable exist for exactly that, and they work best together.
1. Run evals before you publish
An eval is a set of cases: a message, and what a good answer has to do. Beside Publish there's a Run evals first switch, on by default. With it on, publishing runs every case on your draft and shows what changed before the draft goes live.
Each case is a real run, with the same model, the same cost and the same trace as production. Any tool that isn't safe to repeat, like sending an email, answers from a fixture instead, so an eval never touches the world.
A case can check that the answer:
- contains a phrase, matches a regex or is exactly a value
- is valid JSON
- used a tool, never used one, or used tools in order
- finished in fewer steps than a limit, or under a budget
- passes a judge model's reading of it
A case passes at a score of 0.7, and every eval run is compared with a baseline, so you see what got better as well as what got worse. One eval run takes up to 50 cases.
2. Put risky actions behind approval
Some actions shouldn't happen without a person: sending an email to a customer, issuing a refund, deleting a record. List those tools and app actions under Needs approval in the agent's settings.
When the agent wants to use one, the run pauses.
- In the app, the person sees a card with Approve and Deny.
- On a channel, the agent's message ends with "Reply YES to approve or NO to deny".
A waiting run has no deadline, and it carries on with the version of the agent it started on, even if you publish a new one meanwhile.
3. Read the trace
Every run has a trace, built live while it runs and kept after it ends. It shows each model call with its tokens, cost and time to first token, each tool call with what went in and what came out, and the moments the run waited on a person. When people vote on an answer, the votes land there too.
When an eval case fails, open its trace first. It usually shows the exact step where the agent went wrong, which is the step to fix.
Put together
- Make the change on a draft.
- Publish with Run evals first on, and read what changed.
- Keep anything that can't be undone behind Needs approval.
- Spot-check a few traces from real traffic after it's live.
None of this slows you down much, and it turns "I think it still works" into "I checked".