Plan an agent evaluation when you want an AI agent to inspect evidence, apply rules, summarise findings or recommend a decision, without the agent becoming the party accountable for the outcome. The evaluation runs as a Qi Flow: a state machine that checks the agent's authority, gives it only the context it may see, and records what it found as structured data that people and services can inspect, replay and challenge.
This page helps you design that Flow: its states, the agent's UCAN capabilities, the claim under review, the rubric, the Evaluation Claim the agent writes, and the UDID that records the decision.
This is a design guide. It helps you plan, and it gives you nothing to run. The JSON on this page shows the shape of each record. It is not a schema from a released package: map each field to the Qi and IXO SDK fields your implementation uses.
| You end with | A design for one evaluation Flow: its states, the agent's UCAN delegation, a rubric, and the shapes of the Evaluation Claim and the UDID. |
| Who decides | The agent gathers evidence and recommends or proposes. It never signs the decision: a decision oracle supplies the facts, and the IXO decision engine applies the rubric and signs the decision record, the UDID. A human verifier reviews where the rubric asks for one. Treat the model's answer as working output, and treat the claims, evidence references, UCAN delegations, Flow state and UDID records as the accountable record. |
What an evaluation must answer
| Question | Where the answer lives |
|---|---|
| Who was allowed to act? | UCAN delegation and capability checks |
| What was evaluated? | Claims, entities, evidence and Flow state |
| How was it evaluated? | Protocol, rubric, tools, checks and citations |
| What was determined? | The UDID record. The Evaluation Claim holds the agent's recommendation |
An agent without these controls can act outside its authority, use stale context, take submitted evidence for verified state, summarise without citing sources, or trigger a payment, a credential or a change of state before a valid determination exists. The Flow prevents this by making the evaluation part of a governed state machine.
The pattern
A claim or task enters evaluation
A participant, service, device or agent submits a claim, completes a task or proposes a state transition in a Qi Flow.
The Flow opens an evaluation context
The Flow instance sets the subject, claim type, protocol, rubric version, allowed evidence sources, current state snapshot and decision boundary.
The Flow checks UCAN authority
The agent must hold a delegation for this exact action, resource, claim type, tool, time window and Flow instance.
The agent retrieves the context it may see
Only the claims, evidence, rooms, graph state, tools and records inside its UCAN scope.
The agent applies the rubric
It checks the claim against required fields, evidence rules, protocol constraints, scoring thresholds, disqualifiers and escalation rules.
The agent writes an Evaluation Claim
Structured data with cited evidence, the checks applied, confidence, a recommendation, limitations and proof of the agent's authority.
The Flow records a UDID when ready
The IXO decision engine signs the decision record. The UDID records what was decided, why, under which authority, with which evidence, and what state or value changed.
People or services act on the result
The Flow routes it to approval, rejection, dispute, settlement, credential issuance, a state update or a request for more evidence.
| Term | What it is |
|---|---|
| UCAN | A delegation of what an agent may do, on which resource, under which constraints |
| Claim | A record of something asserted or done: agent work, submitted evidence, a review or an evaluation |
| Evaluation Claim | A record of how a claim, task or proposed state transition was evaluated |
| UDID | The record of the final decision and impact determination, made from one or more evaluations |
| Flow state | Where the evaluation happens and which transitions are allowed |
| Rubric | The rules, scoring, thresholds, disqualifiers and escalation conditions |
| Evidence reference | A link from the evaluation to a document, measurement, observation, attestation, media file, report, sensor record or external record |
| Agentic oracle | A service that does specialised review, evidence analysis, scoring, prediction, recommendation or verification support |
Start with one narrow evaluation
Start with one review task in which the agent can recommend an outcome but not finalise it. Good first tasks:
- check whether a claim is complete
- classify evidence by type and relevance
- detect missing required fields
- compare submitted evidence with the protocol's requirements
- summarise conflicting evidence
- score one section of the rubric
- recommend whether a human verifier should approve, reject, dispute or ask for more evidence
Do not start with a task in which the agent can release funds, issue credentials, update high-value state or approve an outcome that cannot be reversed.
Design the Flow states
The minimum Flow:
| State | Purpose | Exit condition |
|---|---|---|
submitted | A claim or agent task output has entered the Flow | The claim exists and has a subject |
authority_check | Verify the UCAN delegation and credentials | The agent has the required capabilities |
context_resolved | Retrieve the current graph state, evidence and protocol rules | Every required context reference is loaded or marked missing |
evaluating | Run the agent evaluation against the rubric | A structured Evaluation Claim is produced |
review_required | A human or authorised verifier reviews the recommendation | The reviewer accepts, rejects, disputes or asks for more evidence |
determined | The UDID is created or updated | The decision and impact determination is signed or recorded |
actioned | The allowed state transition, payment, credential or next Flow is triggered | The result of the action is recorded |
closed | The evaluation loop is complete | The audit trail can be inspected |
Add explicit failure states:
| State | When |
|---|---|
unauthorized | The agent does not have the required UCAN capability |
insufficient_evidence | Required evidence is missing, stale, inaccessible or invalid |
rubric_failed | The claim fails a hard protocol rule |
conflict_detected | Evidence sources conflict or cannot be reconciled |
human_escalation | The agent is uncertain, or the rubric requires human judgement |
disputed | A participant challenges the evaluation or the determination |
The disputed Flow state is part of your Flow, not a chain status. On chain, a dispute is its own
record. See How claim disputes work on chain.
Scope the UCAN delegation
Scope each UCAN to the smallest useful evaluation task. Define:
| Part | What to set |
|---|---|
| Issuer | The person, organisation, POD or service delegating authority |
| Audience | The DID of the agent or agentic oracle receiving it |
| Resource | The POD, Flow instance, claim collection, claim, entity, evidence set, room or tool |
| Capabilities | The exact actions the agent may perform |
| Constraints | Limits on claim type, time, budget, tool use, output type, state transition and approval power |
| Expiry | When the delegation ends |
| Revocation path | How the delegation can be suspended or revoked |
| Proof chain | How the Flow verifies the delegation |
A first capability set:
| Capability | Allows |
|---|---|
claim.read | Read one submitted claim and its metadata |
evidence.read | Read only evidence linked to the claim |
entity.read | Read only entities the claim references |
rubric.read | Read the active rubric and protocol version |
evaluation.create | Create an Evaluation Claim |
state.propose | Propose a Flow transition, not execute it |
message.create | Send a structured finding to the review room |
Do not grant these at first:
| Capability | Why not |
|---|---|
payment.release | Moving value should need a UDID and stronger approval |
credential.issue | Issuing a credential should need a human or protocol-controlled determination |
state.update | A direct state update can bypass review |
claim.approve | Approval should be separate from recommendation |
policy.change | An agent should not change the rules it is evaluated against |
The shape of a delegation, to map onto the Qi and IXO SDK fields your implementation uses:
{
"ucan": {
"issuer": "did:ixo:pod:review-board",
"audience": "did:ixo:oracle:evidence-reviewer",
"resource": {
"pod": "did:ixo:pod:clean-cooking-program",
"flow": "flow:claim-review:v1",
"claimCollection": "claims:stove-usage:v1"
},
"capabilities": [
"claim.read",
"evidence.read",
"entity.read",
"rubric.read",
"evaluation.create",
"state.propose"
],
"constraints": {
"claimTypes": ["stove_usage"],
"maxEvidenceItems": 50,
"allowedTools": ["ixo.graph.query", "evidence.hash.verify", "rubric.evaluate"],
"mayApprove": false,
"mayReleasePayment": false,
"expiresAt": "2026-06-30T23:59:59Z"
}
}
}The safest default is propose-only. The agent creates an Evaluation Claim and proposes the next Flow state. The agent does not decide whether the proposal becomes a UDID-backed determination: the IXO decision engine applies the rubric and signs the decision.
Define the claim under review
| Field | What it holds |
|---|---|
claimId | The claim's unique identifier |
claimType | The type of claim under review |
issuer | The DID of the person, organisation, service, device or agent that made the claim |
subject | The entity, asset, project, person, device, service or outcome the claim is about |
data | The structured data submitted |
evidence | Linked evidence references |
proof | A cryptographic proof, signature, hash, attestation or provenance record |
submittedAt | The time of submission |
protocolId | The protocol or Blueprint used to evaluate the claim |
flowId | The Qi Flow instance handling the review |
{
"claimId": "claim:stove-usage:000123",
"claimType": "stove_usage",
"issuer": "did:ixo:org:field-operator",
"subject": "did:ixo:device:stove-7781",
"data": {
"period": "2026-04",
"reportedBurnHours": 182,
"householdId": "entity:household:442"
},
"evidence": [
{
"type": "deviceTelemetry",
"uri": "ipfs://...",
"hash": "bafy..."
},
{
"type": "fieldVisitReport",
"uri": "ipfs://...",
"hash": "bafy..."
}
],
"protocolId": "blueprint:clean-cooking-mrv:v1",
"flowId": "flow:claim-review:7781"
}Define the rubric
The rubric turns protocol rules into checks that the agent and the Flow can apply. Include the required evidence, freshness rules, source authenticity checks, data integrity checks, field completeness checks, allowed value ranges, consistency checks across evidence sources, disqualifying conditions, scoring rules, the minimum score for a recommendation, the conditions that need human review or a dispute or investigation, and the Flow transitions allowed after evaluation.
{
"rubricId": "rubric:stove-usage-review:v1",
"claimType": "stove_usage",
"requiredEvidence": [
"deviceTelemetry",
"fieldVisitReport"
],
"checks": [
{
"id": "evidence.telemetry.present",
"type": "required",
"description": "Telemetry evidence is linked and hash-verifiable"
},
{
"id": "usage.range.valid",
"type": "range",
"description": "Reported burn hours are within the protocol range",
"min": 1,
"max": 744
},
{
"id": "field.report.consistent",
"type": "consistency",
"description": "Field visit report does not contradict telemetry period or device identity"
}
],
"thresholds": {
"recommendApprove": 0.85,
"recommendRejectBelow": 0.5,
"humanReviewBelow": 0.85
},
"disqualifiers": [
"missingTelemetry",
"invalidEvidenceHash",
"deviceNotLinkedToHousehold",
"claimOutsideReportingPeriod"
]
}Define the Evaluation Claim
The agent's output is structured data, not a free-form opinion.
| Field | What it holds |
|---|---|
evaluationId | The evaluation's unique identifier |
subjectClaimId | The claim being evaluated |
evaluatorDid | The agentic oracle, agent, person or service doing the evaluation |
ucanProof | Proof that the evaluator had authority |
rubricId | The rubric used |
rubricVersion | Its exact version |
stateSnapshotRef | A reference to the Flow or graph state used at evaluation time |
evidenceRefs | The evidence items inspected |
checks | The result of each rule |
findings | Structured observations with evidence citations |
score | A numeric score, if the rubric requires one |
recommendation | The proposed next action |
confidence | Confidence in the recommendation |
limitations | Missing evidence, uncertainty, assumptions or unresolved conflicts |
proposedTransition | The Flow transition the agent proposes |
proof | A signature, hash, attestation or other proof of the evaluation record |
{
"evaluationId": "eval:claim:stove-usage:000123:oracle-01",
"type": "AgentEvaluationClaim",
"subjectClaimId": "claim:stove-usage:000123",
"evaluatorDid": "did:ixo:oracle:evidence-reviewer",
"ucanProof": "ucan:proof:...",
"rubricId": "rubric:stove-usage-review:v1",
"rubricVersion": "1.0.0",
"stateSnapshotRef": "state:flow:claim-review:7781:checkpoint:004",
"evidenceRefs": [
"evidence:telemetry:hash:bafy...",
"evidence:field-report:hash:bafy..."
],
"checks": [
{
"checkId": "evidence.telemetry.present",
"result": "pass",
"evidenceRef": "evidence:telemetry:hash:bafy..."
},
{
"checkId": "usage.range.valid",
"result": "pass",
"observedValue": 182
},
{
"checkId": "field.report.consistent",
"result": "needs_review",
"reason": "Field report date is one day outside the telemetry period"
}
],
"score": 0.82,
"recommendation": "request_more_evidence",
"confidence": 0.76,
"limitations": [
"Field report date mismatch requires human review"
],
"proposedTransition": "human_escalation"
}Do not store the model's private scratchpad as the audit trail. Store the evidence references, extracted facts, tool calls, checks applied, rule outcomes, recommendation, limitations and the final rationale that reviewers can inspect.
Define the UDID
Create or update a UDID when the Flow reaches a determination point, not for every intermediate model output.
| Field | What it holds |
|---|---|
udid | The determination's unique identifier |
decisionType | Approval, rejection, request for evidence, dispute, settlement, credential issuance, state update or no-op |
subjectClaims | The claims considered |
evaluationClaims | The evaluations used |
authority | UCANs, credentials, verifier role or governance authority |
rubric | The rubric and protocol version applied |
evidence | The evidence references used in the determination |
determination | The final decision |
impact | What changed, or will change, because of the decision |
stateTransition | The Flow transition or graph update the determination authorises |
approver | The person, service, governance process or authorised verifier |
timestamp | The time of the determination |
proof | A signature, attestation, transaction hash or other proof |
disputeWindow | The period or condition in which the determination can be challenged |
{
"udid": "udid:flow:claim-review:7781:determination:001",
"type": "UniversalDecisionAndImpactDetermination",
"decisionType": "request_more_evidence",
"subjectClaims": [
"claim:stove-usage:000123"
],
"evaluationClaims": [
"eval:claim:stove-usage:000123:oracle-01"
],
"authority": {
"verifier": "did:ixo:person:human-verifier-17",
"agentUcan": "ucan:proof:...",
"protocol": "blueprint:clean-cooking-mrv:v1"
},
"rubric": {
"id": "rubric:stove-usage-review:v1",
"version": "1.0.0"
},
"determination": {
"status": "more_evidence_required",
"reason": "Telemetry is present and valid, but the field report date conflicts with the reporting period."
},
"impact": {
"paymentReleased": false,
"credentialIssued": false,
"claimStatus": "evidence_requested"
},
"stateTransition": {
"from": "review_required",
"to": "insufficient_evidence"
},
"disputeWindow": "P14D"
}Decide what the agent may do
| Mode | The agent can | Use when |
|---|---|---|
Recommend | Inspect context and create an Evaluation Claim | A first implementation, a high-stakes review, a new rubric |
Propose | Create an Evaluation Claim and propose a Flow transition | The rubric is stable and human review is still required |
Act | Execute a permitted transition after the checks pass | Low-risk actions with a strict UCAN scope and protocol guardrails |
Start with Recommend, move to Propose, and allow Act only for narrow, reversible, low-risk
transitions. In every mode the agent recommends or carries out a transition. It does not sign the
decision record.
Plan the tests
Run a test set before the evaluation touches real state.
| Case | Expected behaviour |
|---|---|
| Valid claim with complete evidence | The agent recommends approval |
| Missing required evidence | The agent detects the incomplete submission |
| Invalid evidence hash | The agent does not trust evidence it cannot verify |
| Stale evidence | The agent applies the freshness rules |
| Conflicting evidence | The agent escalates instead of forcing a decision |
| Unauthorised agent | The UCAN gate blocks the evaluation |
| Wrong claim type | The agent cannot work outside its scope |
| Revoked UCAN | The Flow refuses access that was authorised before |
| Prompt injection in evidence | The agent treats evidence content as untrusted input |
| Score below threshold | The agent does not recommend approval |
| Boundary score | The rubric's threshold behaviour is correct |
| Human disagreement | The review and the correction are captured |
| Disputed determination | The Flow can route to dispute handling |
| Payment-triggering decision | No payment happens without valid UDID authority |
Measure the operation, not only the model:
| Metric | What good looks like |
|---|---|
| Authorisation accuracy | The agent acts only when its UCAN scope permits |
| Evidence citation coverage | Every finding links to evidence or state |
| Rubric adherence | Every required check is applied |
| False approval rate | Bad claims are not recommended for approval |
| False rejection rate | Valid claims are not rejected without cause |
| Escalation quality | Ambiguous cases go to people |
| UDID completeness | Determinations include authority, evidence, decision, impact and proof |
| Human override rate | A high rate triggers work on the rubric or the agent |
| Audit replay success | A reviewer can rebuild the evaluation from the records |
| State transition accuracy | Proposed transitions match the Flow's rules |
| Time to determination | Review gets faster without losing accountability |
When something goes wrong
| Problem | What to do |
|---|---|
| The agent invents evidence | Require an evidence reference for every finding. Reject evaluations that cite documents, measurements, claims or state that are not in the permitted context. |
| The agent has too much authority | Replace broad API keys with UCAN capability delegation. Scope authority by Flow instance, resource, claim type, tool, time window and allowed action. |
| The agent uses stale context | Include a state snapshot reference in the Evaluation Claim. If the graph state changes, require a new evaluation or an explicit refresh. |
| The rubric is too vague | Turn policy language into checks, thresholds, disqualifiers and escalation rules. Send ambiguity to human review. |
| The model's output is treated as truth | Write structured Evaluation Claims and UDID records. A chat response is not the record for settlement, credentials or state changes. |
| The agent approves its own work | Separate doing the task from evaluating it. Use independent evaluators or human review for high-stakes decisions. |
| The Flow cannot explain a decision | Require the UDID to reference the claims, evaluations, evidence, rubric version, authority, decision, impact and proof. |
Check the design
Your first build should be one agent-assisted evaluation that cannot approve, pay, issue or update state by itself, with one of each: claim type, claim collection, Flow, agentic oracle or agent DID, UCAN delegation, rubric, Evaluation Claim schema, UDID schema, human review step, dispute path and production metric dashboard. Run at least 20 representative claims through the Flow before you enable any automated state transition.
Before launch, confirm that:
- the agent has a DID
- every evaluation action requires UCAN authority
- UCAN scopes are narrow and expire
- claims have typed schemas and evidence references
- evidence can be resolved and verified
- the rubric is versioned
- the Flow has explicit states and failure paths
- the agent emits structured Evaluation Claims
- the UDID records authority, evidence, decision, impact, state transition and proof
- irreversible actions need human, protocol or governance approval
- disputes can be submitted and resolved
- reviewers can replay the evaluation from stored records
- revoked authority blocks future actions
- payment, credential and state update actions cannot run without valid determination authority
Example: agent-assisted evidence review
A field operator submits a claim that a clean cooking device was used during a reporting period. The Qi Flow:
- receives the claim
- checks that the Evidence Review Oracle has UCAN authority to inspect this claim type
- retrieves the linked telemetry, field report, device entity, household entity and active protocol rules
- asks the agent to apply the usage review rubric
- records an Evaluation Claim with findings, evidence references, score, recommendation and limitations
- routes the recommendation to a human verifier, because the score is below the automatic threshold
- records a UDID after the verifier decides to request more evidence
- moves the Flow to
insufficient_evidence - tells the claimant about the missing or conflicting evidence
The agent helped evaluate the claim, and it did not become the final authority. The accountable record is the UCAN delegation, the claim, the evidence, the Evaluation Claim, the human review, the Flow transition and the UDID, taken together.
Next steps
Where each step of a claim workflow runs, and the chain's evaluation statuses.
How claim disputes work on chainWho can file a dispute, the stake, and how it is decided.
Agentic oraclesAgent services for verification, decision support and evidence analysis.
Model Context ProtocolHow agents connect to IXO services through capability-scoped tools.