Design an agent evaluation

Plan a Qi Flow in which an AI agent reviews a claim within UCAN authority, applies a rubric and writes an Evaluation Claim that recommends a decision.

Plan an agent evaluation when you want an AI agent to inspect evidence, apply rules, summarise findings or recommend a decision, without the agent becoming the party accountable for the outcome. The evaluation runs as a Qi Flow: a state machine that checks the agent's authority, gives it only the context it may see, and records what it found as structured data that people and services can inspect, replay and challenge.

This page helps you design that Flow: its states, the agent's UCAN capabilities, the claim under review, the rubric, the Evaluation Claim the agent writes, and the UDID that records the decision.

You end withA design for one evaluation Flow: its states, the agent's UCAN delegation, a rubric, and the shapes of the Evaluation Claim and the UDID.
Who decidesThe agent gathers evidence and recommends or proposes. It never signs the decision: a decision oracle supplies the facts, and the IXO decision engine applies the rubric and signs the decision record, the UDID. A human verifier reviews where the rubric asks for one. Treat the model's answer as working output, and treat the claims, evidence references, UCAN delegations, Flow state and UDID records as the accountable record.

What an evaluation must answer

QuestionWhere the answer lives
Who was allowed to act?UCAN delegation and capability checks
What was evaluated?Claims, entities, evidence and Flow state
How was it evaluated?Protocol, rubric, tools, checks and citations
What was determined?The UDID record. The Evaluation Claim holds the agent's recommendation

An agent without these controls can act outside its authority, use stale context, take submitted evidence for verified state, summarise without citing sources, or trigger a payment, a credential or a change of state before a valid determination exists. The Flow prevents this by making the evaluation part of a governed state machine.

The pattern

A claim or task enters evaluation

A participant, service, device or agent submits a claim, completes a task or proposes a state transition in a Qi Flow.

The Flow opens an evaluation context

The Flow instance sets the subject, claim type, protocol, rubric version, allowed evidence sources, current state snapshot and decision boundary.

The Flow checks UCAN authority

The agent must hold a delegation for this exact action, resource, claim type, tool, time window and Flow instance.

The agent retrieves the context it may see

Only the claims, evidence, rooms, graph state, tools and records inside its UCAN scope.

The agent applies the rubric

It checks the claim against required fields, evidence rules, protocol constraints, scoring thresholds, disqualifiers and escalation rules.

The agent writes an Evaluation Claim

Structured data with cited evidence, the checks applied, confidence, a recommendation, limitations and proof of the agent's authority.

The Flow records a UDID when ready

The IXO decision engine signs the decision record. The UDID records what was decided, why, under which authority, with which evidence, and what state or value changed.

People or services act on the result

The Flow routes it to approval, rejection, dispute, settlement, credential issuance, a state update or a request for more evidence.

TermWhat it is
UCANA delegation of what an agent may do, on which resource, under which constraints
ClaimA record of something asserted or done: agent work, submitted evidence, a review or an evaluation
Evaluation ClaimA record of how a claim, task or proposed state transition was evaluated
UDIDThe record of the final decision and impact determination, made from one or more evaluations
Flow stateWhere the evaluation happens and which transitions are allowed
RubricThe rules, scoring, thresholds, disqualifiers and escalation conditions
Evidence referenceA link from the evaluation to a document, measurement, observation, attestation, media file, report, sensor record or external record
Agentic oracleA service that does specialised review, evidence analysis, scoring, prediction, recommendation or verification support

Start with one narrow evaluation

Start with one review task in which the agent can recommend an outcome but not finalise it. Good first tasks:

  • check whether a claim is complete
  • classify evidence by type and relevance
  • detect missing required fields
  • compare submitted evidence with the protocol's requirements
  • summarise conflicting evidence
  • score one section of the rubric
  • recommend whether a human verifier should approve, reject, dispute or ask for more evidence

Do not start with a task in which the agent can release funds, issue credentials, update high-value state or approve an outcome that cannot be reversed.

Design the Flow states

The minimum Flow:

StatePurposeExit condition
submittedA claim or agent task output has entered the FlowThe claim exists and has a subject
authority_checkVerify the UCAN delegation and credentialsThe agent has the required capabilities
context_resolvedRetrieve the current graph state, evidence and protocol rulesEvery required context reference is loaded or marked missing
evaluatingRun the agent evaluation against the rubricA structured Evaluation Claim is produced
review_requiredA human or authorised verifier reviews the recommendationThe reviewer accepts, rejects, disputes or asks for more evidence
determinedThe UDID is created or updatedThe decision and impact determination is signed or recorded
actionedThe allowed state transition, payment, credential or next Flow is triggeredThe result of the action is recorded
closedThe evaluation loop is completeThe audit trail can be inspected

Add explicit failure states:

StateWhen
unauthorizedThe agent does not have the required UCAN capability
insufficient_evidenceRequired evidence is missing, stale, inaccessible or invalid
rubric_failedThe claim fails a hard protocol rule
conflict_detectedEvidence sources conflict or cannot be reconciled
human_escalationThe agent is uncertain, or the rubric requires human judgement
disputedA participant challenges the evaluation or the determination

The disputed Flow state is part of your Flow, not a chain status. On chain, a dispute is its own record. See How claim disputes work on chain.

Scope the UCAN delegation

Scope each UCAN to the smallest useful evaluation task. Define:

PartWhat to set
IssuerThe person, organisation, POD or service delegating authority
AudienceThe DID of the agent or agentic oracle receiving it
ResourceThe POD, Flow instance, claim collection, claim, entity, evidence set, room or tool
CapabilitiesThe exact actions the agent may perform
ConstraintsLimits on claim type, time, budget, tool use, output type, state transition and approval power
ExpiryWhen the delegation ends
Revocation pathHow the delegation can be suspended or revoked
Proof chainHow the Flow verifies the delegation

A first capability set:

CapabilityAllows
claim.readRead one submitted claim and its metadata
evidence.readRead only evidence linked to the claim
entity.readRead only entities the claim references
rubric.readRead the active rubric and protocol version
evaluation.createCreate an Evaluation Claim
state.proposePropose a Flow transition, not execute it
message.createSend a structured finding to the review room

Do not grant these at first:

CapabilityWhy not
payment.releaseMoving value should need a UDID and stronger approval
credential.issueIssuing a credential should need a human or protocol-controlled determination
state.updateA direct state update can bypass review
claim.approveApproval should be separate from recommendation
policy.changeAn agent should not change the rules it is evaluated against

The shape of a delegation, to map onto the Qi and IXO SDK fields your implementation uses:

{
  "ucan": {
    "issuer": "did:ixo:pod:review-board",
    "audience": "did:ixo:oracle:evidence-reviewer",
    "resource": {
      "pod": "did:ixo:pod:clean-cooking-program",
      "flow": "flow:claim-review:v1",
      "claimCollection": "claims:stove-usage:v1"
    },
    "capabilities": [
      "claim.read",
      "evidence.read",
      "entity.read",
      "rubric.read",
      "evaluation.create",
      "state.propose"
    ],
    "constraints": {
      "claimTypes": ["stove_usage"],
      "maxEvidenceItems": 50,
      "allowedTools": ["ixo.graph.query", "evidence.hash.verify", "rubric.evaluate"],
      "mayApprove": false,
      "mayReleasePayment": false,
      "expiresAt": "2026-06-30T23:59:59Z"
    }
  }
}

Define the claim under review

FieldWhat it holds
claimIdThe claim's unique identifier
claimTypeThe type of claim under review
issuerThe DID of the person, organisation, service, device or agent that made the claim
subjectThe entity, asset, project, person, device, service or outcome the claim is about
dataThe structured data submitted
evidenceLinked evidence references
proofA cryptographic proof, signature, hash, attestation or provenance record
submittedAtThe time of submission
protocolIdThe protocol or Blueprint used to evaluate the claim
flowIdThe Qi Flow instance handling the review
{
  "claimId": "claim:stove-usage:000123",
  "claimType": "stove_usage",
  "issuer": "did:ixo:org:field-operator",
  "subject": "did:ixo:device:stove-7781",
  "data": {
    "period": "2026-04",
    "reportedBurnHours": 182,
    "householdId": "entity:household:442"
  },
  "evidence": [
    {
      "type": "deviceTelemetry",
      "uri": "ipfs://...",
      "hash": "bafy..."
    },
    {
      "type": "fieldVisitReport",
      "uri": "ipfs://...",
      "hash": "bafy..."
    }
  ],
  "protocolId": "blueprint:clean-cooking-mrv:v1",
  "flowId": "flow:claim-review:7781"
}

Define the rubric

The rubric turns protocol rules into checks that the agent and the Flow can apply. Include the required evidence, freshness rules, source authenticity checks, data integrity checks, field completeness checks, allowed value ranges, consistency checks across evidence sources, disqualifying conditions, scoring rules, the minimum score for a recommendation, the conditions that need human review or a dispute or investigation, and the Flow transitions allowed after evaluation.

{
  "rubricId": "rubric:stove-usage-review:v1",
  "claimType": "stove_usage",
  "requiredEvidence": [
    "deviceTelemetry",
    "fieldVisitReport"
  ],
  "checks": [
    {
      "id": "evidence.telemetry.present",
      "type": "required",
      "description": "Telemetry evidence is linked and hash-verifiable"
    },
    {
      "id": "usage.range.valid",
      "type": "range",
      "description": "Reported burn hours are within the protocol range",
      "min": 1,
      "max": 744
    },
    {
      "id": "field.report.consistent",
      "type": "consistency",
      "description": "Field visit report does not contradict telemetry period or device identity"
    }
  ],
  "thresholds": {
    "recommendApprove": 0.85,
    "recommendRejectBelow": 0.5,
    "humanReviewBelow": 0.85
  },
  "disqualifiers": [
    "missingTelemetry",
    "invalidEvidenceHash",
    "deviceNotLinkedToHousehold",
    "claimOutsideReportingPeriod"
  ]
}

Define the Evaluation Claim

The agent's output is structured data, not a free-form opinion.

FieldWhat it holds
evaluationIdThe evaluation's unique identifier
subjectClaimIdThe claim being evaluated
evaluatorDidThe agentic oracle, agent, person or service doing the evaluation
ucanProofProof that the evaluator had authority
rubricIdThe rubric used
rubricVersionIts exact version
stateSnapshotRefA reference to the Flow or graph state used at evaluation time
evidenceRefsThe evidence items inspected
checksThe result of each rule
findingsStructured observations with evidence citations
scoreA numeric score, if the rubric requires one
recommendationThe proposed next action
confidenceConfidence in the recommendation
limitationsMissing evidence, uncertainty, assumptions or unresolved conflicts
proposedTransitionThe Flow transition the agent proposes
proofA signature, hash, attestation or other proof of the evaluation record
{
  "evaluationId": "eval:claim:stove-usage:000123:oracle-01",
  "type": "AgentEvaluationClaim",
  "subjectClaimId": "claim:stove-usage:000123",
  "evaluatorDid": "did:ixo:oracle:evidence-reviewer",
  "ucanProof": "ucan:proof:...",
  "rubricId": "rubric:stove-usage-review:v1",
  "rubricVersion": "1.0.0",
  "stateSnapshotRef": "state:flow:claim-review:7781:checkpoint:004",
  "evidenceRefs": [
    "evidence:telemetry:hash:bafy...",
    "evidence:field-report:hash:bafy..."
  ],
  "checks": [
    {
      "checkId": "evidence.telemetry.present",
      "result": "pass",
      "evidenceRef": "evidence:telemetry:hash:bafy..."
    },
    {
      "checkId": "usage.range.valid",
      "result": "pass",
      "observedValue": 182
    },
    {
      "checkId": "field.report.consistent",
      "result": "needs_review",
      "reason": "Field report date is one day outside the telemetry period"
    }
  ],
  "score": 0.82,
  "recommendation": "request_more_evidence",
  "confidence": 0.76,
  "limitations": [
    "Field report date mismatch requires human review"
  ],
  "proposedTransition": "human_escalation"
}

Define the UDID

Create or update a UDID when the Flow reaches a determination point, not for every intermediate model output.

FieldWhat it holds
udidThe determination's unique identifier
decisionTypeApproval, rejection, request for evidence, dispute, settlement, credential issuance, state update or no-op
subjectClaimsThe claims considered
evaluationClaimsThe evaluations used
authorityUCANs, credentials, verifier role or governance authority
rubricThe rubric and protocol version applied
evidenceThe evidence references used in the determination
determinationThe final decision
impactWhat changed, or will change, because of the decision
stateTransitionThe Flow transition or graph update the determination authorises
approverThe person, service, governance process or authorised verifier
timestampThe time of the determination
proofA signature, attestation, transaction hash or other proof
disputeWindowThe period or condition in which the determination can be challenged
{
  "udid": "udid:flow:claim-review:7781:determination:001",
  "type": "UniversalDecisionAndImpactDetermination",
  "decisionType": "request_more_evidence",
  "subjectClaims": [
    "claim:stove-usage:000123"
  ],
  "evaluationClaims": [
    "eval:claim:stove-usage:000123:oracle-01"
  ],
  "authority": {
    "verifier": "did:ixo:person:human-verifier-17",
    "agentUcan": "ucan:proof:...",
    "protocol": "blueprint:clean-cooking-mrv:v1"
  },
  "rubric": {
    "id": "rubric:stove-usage-review:v1",
    "version": "1.0.0"
  },
  "determination": {
    "status": "more_evidence_required",
    "reason": "Telemetry is present and valid, but the field report date conflicts with the reporting period."
  },
  "impact": {
    "paymentReleased": false,
    "credentialIssued": false,
    "claimStatus": "evidence_requested"
  },
  "stateTransition": {
    "from": "review_required",
    "to": "insufficient_evidence"
  },
  "disputeWindow": "P14D"
}

Decide what the agent may do

ModeThe agent canUse when
RecommendInspect context and create an Evaluation ClaimA first implementation, a high-stakes review, a new rubric
ProposeCreate an Evaluation Claim and propose a Flow transitionThe rubric is stable and human review is still required
ActExecute a permitted transition after the checks passLow-risk actions with a strict UCAN scope and protocol guardrails

Start with Recommend, move to Propose, and allow Act only for narrow, reversible, low-risk transitions. In every mode the agent recommends or carries out a transition. It does not sign the decision record.

Plan the tests

Run a test set before the evaluation touches real state.

CaseExpected behaviour
Valid claim with complete evidenceThe agent recommends approval
Missing required evidenceThe agent detects the incomplete submission
Invalid evidence hashThe agent does not trust evidence it cannot verify
Stale evidenceThe agent applies the freshness rules
Conflicting evidenceThe agent escalates instead of forcing a decision
Unauthorised agentThe UCAN gate blocks the evaluation
Wrong claim typeThe agent cannot work outside its scope
Revoked UCANThe Flow refuses access that was authorised before
Prompt injection in evidenceThe agent treats evidence content as untrusted input
Score below thresholdThe agent does not recommend approval
Boundary scoreThe rubric's threshold behaviour is correct
Human disagreementThe review and the correction are captured
Disputed determinationThe Flow can route to dispute handling
Payment-triggering decisionNo payment happens without valid UDID authority

Measure the operation, not only the model:

MetricWhat good looks like
Authorisation accuracyThe agent acts only when its UCAN scope permits
Evidence citation coverageEvery finding links to evidence or state
Rubric adherenceEvery required check is applied
False approval rateBad claims are not recommended for approval
False rejection rateValid claims are not rejected without cause
Escalation qualityAmbiguous cases go to people
UDID completenessDeterminations include authority, evidence, decision, impact and proof
Human override rateA high rate triggers work on the rubric or the agent
Audit replay successA reviewer can rebuild the evaluation from the records
State transition accuracyProposed transitions match the Flow's rules
Time to determinationReview gets faster without losing accountability

When something goes wrong

ProblemWhat to do
The agent invents evidenceRequire an evidence reference for every finding. Reject evaluations that cite documents, measurements, claims or state that are not in the permitted context.
The agent has too much authorityReplace broad API keys with UCAN capability delegation. Scope authority by Flow instance, resource, claim type, tool, time window and allowed action.
The agent uses stale contextInclude a state snapshot reference in the Evaluation Claim. If the graph state changes, require a new evaluation or an explicit refresh.
The rubric is too vagueTurn policy language into checks, thresholds, disqualifiers and escalation rules. Send ambiguity to human review.
The model's output is treated as truthWrite structured Evaluation Claims and UDID records. A chat response is not the record for settlement, credentials or state changes.
The agent approves its own workSeparate doing the task from evaluating it. Use independent evaluators or human review for high-stakes decisions.
The Flow cannot explain a decisionRequire the UDID to reference the claims, evaluations, evidence, rubric version, authority, decision, impact and proof.

Check the design

Your first build should be one agent-assisted evaluation that cannot approve, pay, issue or update state by itself, with one of each: claim type, claim collection, Flow, agentic oracle or agent DID, UCAN delegation, rubric, Evaluation Claim schema, UDID schema, human review step, dispute path and production metric dashboard. Run at least 20 representative claims through the Flow before you enable any automated state transition.

Before launch, confirm that:

  • the agent has a DID
  • every evaluation action requires UCAN authority
  • UCAN scopes are narrow and expire
  • claims have typed schemas and evidence references
  • evidence can be resolved and verified
  • the rubric is versioned
  • the Flow has explicit states and failure paths
  • the agent emits structured Evaluation Claims
  • the UDID records authority, evidence, decision, impact, state transition and proof
  • irreversible actions need human, protocol or governance approval
  • disputes can be submitted and resolved
  • reviewers can replay the evaluation from stored records
  • revoked authority blocks future actions
  • payment, credential and state update actions cannot run without valid determination authority

Example: agent-assisted evidence review

A field operator submits a claim that a clean cooking device was used during a reporting period. The Qi Flow:

  1. receives the claim
  2. checks that the Evidence Review Oracle has UCAN authority to inspect this claim type
  3. retrieves the linked telemetry, field report, device entity, household entity and active protocol rules
  4. asks the agent to apply the usage review rubric
  5. records an Evaluation Claim with findings, evidence references, score, recommendation and limitations
  6. routes the recommendation to a human verifier, because the score is below the automatic threshold
  7. records a UDID after the verifier decides to request more evidence
  8. moves the Flow to insufficient_evidence
  9. tells the claimant about the missing or conflicting evidence

The agent helped evaluate the claim, and it did not become the final authority. The accountable record is the UCAN delegation, the claim, the evidence, the Evaluation Claim, the human review, the Flow transition and the UDID, taken together.

Next steps