Start with the task, not a model name
A request routed by an approved spending threshold may need an explicit rule. For sales trends, first examine data definitions and statistical analysis. For free-text classification or a draft response, a model may be worth evaluating. These are proposed selection heuristics, not proof that one technology is superior. Required quality and error consequences drive the decision; adding a model to an explicit rule does not establish value.
| Illustrative task | Initial approach to investigate | Testable outcome |
|---|---|---|
| Route by amount and role | Rules and workflow | The same path for the same input and rule |
| Report sales trends | Data quality and analytics | Defined metrics reconciled with source data |
| Draft an answer from documents | Retrieval and a model with review | Grounded answers, authorized access, and refusal behavior |
| Payment or a sensitive decision | Deterministic controls and human ownership | Authorization, recorded decisions, and review before action |
Define failure consequences for the actual use case
For an illustrative support assistant, an incomplete usage answer and disclosure of another customer’s document are different failures. Before prototyping, separate content quality, data-access failures, and incorrect actions. Record an owner and a recovery path for each. Establish whether the output is only a draft or can send messages, change records, or initiate transactions. Greater authority requires additional tests and controls.
- Who is affected by an incorrect answer, and who may approve it?
- Which failures must block continuation or release?
- Without adequate evidence, should the system refuse, clarify, or escalate?
- Can users report errors to an identified resolution owner?
Enforce document access and tool authority outside the model
OWASP describes indirect prompt injection through files or pages, and notes that retrieval or fine-tuning does not fully eliminate it. Treat retrieved content as untrusted data, enforcing document permissions and action authority in application code. Least privilege and human approval for high-risk actions are among OWASP’s proposed mitigations. A source citation establishes neither factual correctness nor authorization to access the source.
Technical sources for this section: OWASP Gen AI Security Project [3]
Prepare evaluation before tuning the prototype
The NIST AI RMF calls for documented, repeatable evaluation; its Generative AI Profile is a companion for that family of risks. For a pilot, prepare a separate set of routine, ambiguous, and failing examples. If examples were repeatedly used for tuning, do not present their results as independent evaluation. The table proposes an evaluation plan for a document assistant.
| Criterion | Proposed measurement |
|---|---|
| Correctness and grounding | A domain reviewer checks claims against a specific source version |
| Access boundaries | Test unauthorized-document questions and malicious content |
| Ambiguity and missing answers | Review refusal, clarification, and escalation behavior |
| Cost and latency | Record usage and response time for stated inputs and workload |
| Material failures | Report failure types, occurrences, and the relevant sample count |
Technical sources for this section: NIST AI Resource Center [1]NIST [2]
A single score can hide material failures
Report denominators and sample counts alongside percentages, and inspect Persian and English separately. Small samples or omitted difficult cases limit the decision’s strength. Model-generated grading may provide a supplementary signal; sensitive answers need domain review and independent evidence. Record quality, latency, and cost separately so an improvement in one does not conceal a regression in another.
- Keep the model version, settings, instruction version, and data-source version for each run.
- Agree passing thresholds and release-blocking failures before inspecting results.
- Compare a rules-based option or the current manual method using the same task and measurement approach.
- Include human review and correction time in the operational evaluation.
Run a bounded pilot with a stop path
For a support assistant, a first pilot could show drafts only to an agent who retains sending authority. The interface and tool permissions must enforce that arrangement. Establish data-use permission, processing location, retention, and provider terms. Synthetic samples can support early tests but do not establish behavior on permitted real data. Before activation, name the monitoring owner and the version to which the system can revert.
- Rerun relevant evaluations after model, instruction, source, or tool changes.
- Give material failures an operational stop or disable path.
- Capture user feedback with sensitive data removed and an explicit retention period.
- Review evidence and residual risk before expanding authority or users.
What decision should the pilot enable?
Ask for a bounded use case, evaluation set, reproducible results, cost assumptions, and a decision to continue, revise, or stop. A polished demonstration is insufficient evidence. Where permitted data or acceptance criteria are missing, address that gap first. If deterministic rules cover the task, examine process automation. If a model demonstrates measurable value, operating design and controls are the next stage.