1. Define the operating constraint
Illustrative scenario: an approved order is captured in a CRM and manually entered into an accounting system. Finance needs to know whether an invoice exists and which exceptions need review.
Scoped question: how should one approved order become one accounting record, with a controlled recovery path? This example excludes warehouse, payment and inventory workflows.
2. Record the system and responsibility view
| Stage | Owner | Required decision |
|---|---|---|
| Approved order in CRM | Sales/process owner | What counts as approved and which identifier stays stable? |
| Exchange and exception handling | Integration owner | How is progress recorded and an interruption recovered? |
| Accounting record | Finance owner | What is validated and how is a duplicate detected? |
3. Make risk actionable
| Risk | Consequence | Response to evaluate | Owner |
|---|---|---|---|
| Repeated delivery of an order | A duplicate financial record | Stable external identifier and destination duplicate protection | Integration + finance |
| A rejected record is invisible | Work remains unfinished | Visible exception queue, alert and review procedure | Process owner |
| Interface ownership is unclear | Recovery depends on one person | Named operating responsibility and handover material | Business sponsor |
4. Compare options and record a decision
Option A: use a supported connector if it meets the required identifiers, validation and recovery behaviour. Option B: build a scoped integration if the connector cannot satisfy those controls.
Provisional decision: investigate the supported interface and connector before commissioning custom work. Record the interface/version evidence, gaps, support responsibilities and conditions that would change the recommendation. No vendor compatibility is assumed here.
5. Define acceptance and measurement
- One approved source order produces one destination record.
- A repeated event does not create a second record.
- A rejected record is visible to an authorised reviewer.
- An interrupted exchange can be reconciled without guessing what happened.
- Baseline and post-change manual effort and exception rates are compared over an agreed period.
These are proposed checks for the fictional scenario, not reported JVS project results.
6. Example AI evaluation worksheet
For a separate fictional document-extraction task, establish acceptance before evaluating a model. Never infer a production accuracy figure from this example.
| Synthetic test condition | Expected behaviour | Review boundary |
|---|---|---|
| Required field is absent | Return an explicit missing-field result | Human review; no invented value |
| Input conflicts with itself | Flag the conflict and show the source references | Hold any downstream action |
| Output has an invalid format | Reject it before system entry | Record the failure for investigation |
| Provider is unavailable | Use the agreed fallback or review queue | No silent completion |
What is agreed before paid work begins
The introductory conversation defines the question, available evidence, participants and constraints. A written scope then states deliverables, exclusions, workshops, analysis period, briefing date and fixed fee. Implementation is a separate commitment.
Explore the five assessment offers →
Request a 30-minute introductory conversation →
Written by Jashin Sookdhoo · Reviewed 13 October 2026 · Illustrative sample