Agent Evaluations

Build this when an AI agent should inspect evidence, apply rules, summarize findings, or recommend a decision without becoming the final source of accountability.

Agent evaluations in Qi are not ordinary model evaluations. You are not only asking whether an answer is fluent, correct, or useful. You are checking whether an agent acted within delegated authority, used the right evidence, applied the right rubric, and produced a verifiable decision record that people, services, regulators, or communities can inspect.

The problem

Teams need agents to help with evidence review, claim processing, decision support, fulfillment checks, and workflow routing.

Ordinary agents create several risks:

• they may act outside their authority
• they may use stale or incomplete context
• they may confuse submitted evidence with verified state
• they may summarize without citing sources
• they may recommend actions that the workflow does not allow
• they may produce outputs that cannot be replayed, challenged, or audited
• they may trigger payments, credentials, or state changes before a valid determination exists

In Qi, the Flow engine solves this by making evaluation part of a governed state machine.

An agent evaluation should answer four questions:

What you build

You build a Qi Flow that evaluates agent work or agent-assisted reviews against verified context.

The Flow should:

1. receive a Claim, task output, or Flow event
2. check the agent’s UCAN authority
3. resolve the current state of the relevant entities, Claims, credentials, and evidence
4. run the evaluation against a declared rubric
5. produce a structured Evaluation Claim
6. produce or update a UDID when a decision and impact determination is ready
7. route the result to a human, service, payment step, credential step, dispute path, or next Flow state

Core pattern

Key concepts

Start with one evaluation

Do not begin with a fully autonomous approval workflow.

Start with one narrow review task where the agent can recommend, but not finalize, the outcome.

Good first tasks:

• check whether a Claim is complete
• classify evidence by type and relevance
• detect missing required fields
• compare submitted evidence against protocol requirements
• summarize conflicting evidence
• score one rubric section
• recommend whether a human verifier should approve, reject, dispute, or request more evidence

Avoid first tasks where the agent can directly release funds, issue credentials, update high-value state, or approve irreversible outcomes.

Design the evaluation Flow

Use this minimum Flow shape:

Add explicit failure states:

Configure UCAN authority

A UCAN should be scoped to the smallest useful evaluation task.

Define:

• issuer: the human, organization, POD, or service delegating authority
• audience: the agent or Agentic Oracle DID receiving the authority
• resource: the POD, Flow instance, Claim Collection, Claim, entity, evidence set, room, or tool
• capabilities: the exact actions the agent may perform
• constraints: limits on claim type, time, budget, tool use, output type, state transition, and approval power
• expiry: when the delegation ends
• revocation path: how the delegation can be suspended or revoked
• proof chain: how the Flow verifies the delegation

A useful first UCAN capability set:

Avoid granting these at first:

Example UCAN design shape

Use this as a design shape, then map it to the canonical Qi and IXO SDK fields used in your implementation.

{
  "ucan": {
    "issuer": "did:ixo:pod:review-board",
    "audience": "did:ixo:oracle:evidence-reviewer",
    "resource": {
      "pod": "did:ixo:pod:clean-cooking-program",
      "flow": "flow:claim-review:v1",
      "claimCollection": "claims:stove-usage:v1"
    },
    "capabilities": [
      "claim.read",
      "evidence.read",
      "entity.read",
      "rubric.read",
      "evaluation.create",
      "state.propose"
    ],
    "constraints": {
      "claimTypes": ["stove_usage"],
      "maxEvidenceItems": 50,
      "allowedTools": ["ixo.graph.query", "evidence.hash.verify", "rubric.evaluate"],
      "mayApprove": false,
      "mayReleasePayment": false,
      "expiresAt": "2026-06-30T23:59:59Z"
    }
  }
}

Define the Claim under review

Each evaluation should start with a clear Claim.

Minimum Claim inputs:

Example:

{
  "claimId": "claim:stove-usage:000123",
  "claimType": "stove_usage",
  "issuer": "did:ixo:org:field-operator",
  "subject": "did:ixo:device:stove-7781",
  "data": {
    "period": "2026-04",
    "reportedBurnHours": 182,
    "householdId": "entity:household:442"
  },
  "evidence": [
    {
      "type": "deviceTelemetry",
      "uri": "ipfs://...",
      "hash": "bafy..."
    },
    {
      "type": "fieldVisitReport",
      "uri": "ipfs://...",
      "hash": "bafy..."
    }
  ],
  "protocolId": "blueprint:clean-cooking-mrv:v1",
  "flowId": "flow:claim-review:7781"
}

Define the rubric

The rubric converts protocol rules into checks that the agent and Flow can apply.

A practical rubric should include:

• required evidence
• evidence freshness rules
• source authenticity checks
• data integrity checks
• field completeness checks
• allowed value ranges
• consistency checks across evidence sources
• disqualifying conditions
• scoring rules
• minimum score for recommendation
• conditions that require human review
• conditions that require dispute or investigation
• allowed Flow transitions after evaluation

Example rubric shape:

{
  "rubricId": "rubric:stove-usage-review:v1",
  "claimType": "stove_usage",
  "requiredEvidence": [
    "deviceTelemetry",
    "fieldVisitReport"
  ],
  "checks": [
    {
      "id": "evidence.telemetry.present",
      "type": "required",
      "description": "Telemetry evidence is linked and hash-verifiable"
    },
    {
      "id": "usage.range.valid",
      "type": "range",
      "description": "Reported burn hours are within the protocol range",
      "min": 1,
      "max": 744
    },
    {
      "id": "field.report.consistent",
      "type": "consistency",
      "description": "Field visit report does not contradict telemetry period or device identity"
    }
  ],
  "thresholds": {
    "recommendApprove": 0.85,
    "recommendRejectBelow": 0.5,
    "humanReviewBelow": 0.85
  },
  "disqualifiers": [
    "missingTelemetry",
    "invalidEvidenceHash",
    "deviceNotLinkedToHousehold",
    "claimOutsideReportingPeriod"
  ]
}

Run the evaluation

The agent evaluation should produce structured output, not a free-form opinion.

Minimum Evaluation Claim output:

Example Evaluation Claim:

{
  "evaluationId": "eval:claim:stove-usage:000123:oracle-01",
  "type": "AgentEvaluationClaim",
  "subjectClaimId": "claim:stove-usage:000123",
  "evaluatorDid": "did:ixo:oracle:evidence-reviewer",
  "ucanProof": "ucan:proof:...",
  "rubricId": "rubric:stove-usage-review:v1",
  "rubricVersion": "1.0.0",
  "stateSnapshotRef": "state:flow:claim-review:7781:checkpoint:004",
  "evidenceRefs": [
    "evidence:telemetry:hash:bafy...",
    "evidence:field-report:hash:bafy..."
  ],
  "checks": [
    {
      "checkId": "evidence.telemetry.present",
      "result": "pass",
      "evidenceRef": "evidence:telemetry:hash:bafy..."
    },
    {
      "checkId": "usage.range.valid",
      "result": "pass",
      "observedValue": 182
    },
    {
      "checkId": "field.report.consistent",
      "result": "needs_review",
      "reason": "Field report date is one day outside the telemetry period"
    }
  ],
  "score": 0.82,
  "recommendation": "request_more_evidence",
  "confidence": 0.76,
  "limitations": [
    "Field report date mismatch requires human review"
  ],
  "proposedTransition": "human_escalation"
}

Create the UDID

A UDID is created when the Flow has enough information to record a decision and impact determination.

Do not create a UDID for every intermediate model output. Create or update a UDID when the Flow reaches a determination point.

A UDID should record:

Example UDID shape:

{
  "udid": "udid:flow:claim-review:7781:determination:001",
  "type": "UniversalDecisionAndImpactDetermination",
  "decisionType": "request_more_evidence",
  "subjectClaims": [
    "claim:stove-usage:000123"
  ],
  "evaluationClaims": [
    "eval:claim:stove-usage:000123:oracle-01"
  ],
  "authority": {
    "verifier": "did:ixo:person:human-verifier-17",
    "agentUcan": "ucan:proof:...",
    "protocol": "blueprint:clean-cooking-mrv:v1"
  },
  "rubric": {
    "id": "rubric:stove-usage-review:v1",
    "version": "1.0.0"
  },
  "determination": {
    "status": "more_evidence_required",
    "reason": "Telemetry is present and valid, but the field report date conflicts with the reporting period."
  },
  "impact": {
    "paymentReleased": false,
    "credentialIssued": false,
    "claimStatus": "evidence_requested"
  },
  "stateTransition": {
    "from": "review_required",
    "to": "insufficient_evidence"
  },
  "disputeWindow": "P14D"
}

Decide what the agent may do

Use three evaluation modes.

For most production systems, start with Recommend, move to Propose, and only allow Act for narrow, reversible, low-risk transitions.

Test the evaluation

Use a test set before connecting the evaluation to real state changes.

Create cases for:

Evaluation metrics

Track operational quality, not only model quality.

Common failure modes

First implementation move

Build one agent-assisted evaluation that cannot directly approve, pay, issue, or update state.

Define:

• one Claim type
• one Claim Collection
• one Flow
• one Agentic Oracle or agent DID
• one UCAN delegation
• one rubric
• one Evaluation Claim schema
• one UDID schema
• one human review step
• one dispute path
• one production metric dashboard

Then run at least 20 representative Claims through the Flow before enabling any automated state transition.

Production checklist

Before launch, confirm:

• the agent has a DID
• every evaluation action requires UCAN authority
• UCAN scopes are narrow and expire
• Claims have typed schemas and evidence references
• evidence can be resolved and verified
• the rubric is versioned
• the Flow has explicit states and failure paths
• the agent emits structured Evaluation Claims
• the UDID records authority, evidence, decision, impact, state transition, and proof
• irreversible actions require human, protocol, or governance approval
• disputes can be submitted and resolved
• reviewers can replay the evaluation from stored records
• revoked authority blocks future actions
• payment, credential, and state update actions cannot execute without valid determination authority

Example: agent-assisted evidence review

A field operator submits a Claim that a clean cooking device was used during a reporting period.

The Qi Flow:

1. receives the Claim
2. checks that the Evidence Review Oracle has UCAN authority to inspect this claim type
3. retrieves linked telemetry, field report, device entity, household entity, and active protocol rules
4. asks the agent to apply the usage review rubric
5. records an Evaluation Claim with findings, evidence references, score, recommendation, and limitations
6. routes the recommendation to a human verifier because the score is below the automatic threshold
7. records a UDID after the verifier decides to request more evidence
8. updates the Flow state to insufficient_evidence
9. notifies the claimant about the missing or conflicting evidence

The agent helped evaluate the Claim, but it did not become the final authority. The accountable record is the combination of UCAN delegation, Claim, evidence, Evaluation Claim, human review, Flow transition, and UDID.

Next steps