An evaluator checks claims against written rules, and each decision is signed. On IXO it is called a decision oracle, and on asks.world it appears as a verifier. This page helps you design one on paper. When you have the design, the decision oracle guide is where you build, test and deploy it.
This is a design guide. It helps you plan and gives you nothing to run. The commands are in the build guide.
| Available now | Yes. A decision oracle runs on devnet, testnet and mainnet. Design and test on testnet, which uses play money. |
| You need | Knowledge of the work you want to judge. To build afterwards, a free Cloudflare account. |
| Cost | Design costs nothing. When it runs, the decision engine bills each decision to the account that registered the oracle, and each AI check spends model tokens. |
| You end with | A rubric design: the rules, the facts each rule needs, what happens when a fact is missing, the questions for a human reviewer, and test cases. |
| Who decides | The party commissioning the work owns the rules. Your oracle produces facts. The decision engine applies the rules. A person answers what the rules send to review. |
What you are designing
A decision oracle never judges anything by itself. The rubric asks for named facts, your oracle produces them, and the decision engine decides from those facts. That split is the main design constraint, so keep it in view.
| Part | What it is |
|---|---|
| Claim form | What the person submitting fills in. Their answers are the claim. |
| Rubric | The rule document: gates, scored criteria, the outside sources and AI checks it needs, a duplicate rule, a submission-rate rule, and tasks for a human reviewer. |
| Fact | One named answer your oracle produces: true or false, a score from 0 to 10,000, or an explicit "could not establish". |
| Decision | What the engine reaches from the facts, with a signed receipt. |
Design steps
Write the decision in one sentence
Start with the sentence a careful human reviewer would use. For example: "Approve a delivery claim when the courier's system confirms the delivery, the photo was taken within 100 metres of the address, and the parcel in the photo matches the order."
If you cannot write the sentence, the work is not ready for an evaluator yet.
Split it into gates and scores
A gate is a rule that is true or false. A scored criterion takes a score from 0 to 10,000. Put each condition from your sentence into one or the other.
Prefer gates. A gate is easier to test and easier to explain to someone whose claim failed.
Name the fact behind each rule
For every gate and score, write down the one fact that answers it and where the fact comes from. The package has stock ways to produce the common ones, and every fact you leave to them is a fact you do not have to write or debug.
| Where the fact comes from | Stock, or yours to write |
|---|---|
| A field on the claim form | Read from the claim by the engine. Nothing for you to write |
| An outside system that answers yes or no, or a score | Stock when the system speaks the package's contract. Otherwise a handler you write |
| Where or when a photo was taken | Stock geo handlers |
| A judgment a model can make from written instructions | Stock AI check. A handler when you need a different model or a pre-check |
| Whether this is a duplicate of an earlier claim | Stock |
| How often this person has submitted | Stock |
Decide what happens when a fact is missing
Every fact needs a fallback for the case where it cannot be established. The stock handlers are strict about this, and your design should be too:
- A photo with no location or capture time is an error. It is never treated as "false".
- Evidence a model cannot read (too dark, blurred, cropped) is an error. It is never a failed rule.
- An outage in an outside system is retried. It is never frozen into a decision.
For each fact, choose where an error goes. Sending it to a human reviewer is the usual choice.
Write the review tasks
When the rules cannot decide, the decision is held for a person and names the tasks they must answer. Write those tasks as questions with a fixed answer type: yes or no, one of a list, a number, or text.
A reviewer's answers are folded into the same decision. The facts already produced are replayed, with no new AI calls and no new charges. The reviewer answers exactly the tasks that were named, and nothing else.
Write the test cases
Before you build, write at least these, each with the decision you expect:
- A claim that should be approved.
- One claim for each gate that should fail that gate.
- A claim that should go to review.
- A claim with missing or unreadable evidence.
The build guide runs these two ways: as unit tests with no network, and as a dry run against the live rubric that asks the engine what it would decide, with nothing saved and nothing billed.
Pin the version
A rubric's id is the SHA-256 of its published bytes. Change one word and it is a different rubric. Record the id with your test results, so a later reader can tell which rules a decision was made under.
Decide who has authority
Settle these before you deploy:
- Who lets your oracle sign. The owner of a claim collection grants your oracle the right to record decisions on it. Without the grant, the chain refuses them.
- Whether it signs unattended. Turn automatic signing on only when you mean it.
- Who reviews. The engine never lets the account that registered an oracle review that oracle's claims. Register with an account you will not use for reviewing.
- Who pays for decisions. Charge an evaluation fee, or limit whose work you take. Otherwise anyone who hires your oracle spends your credits.
What an evaluator does not do
- It does not decide. It produces facts, and the engine applies the commissioner's rules to them.
- It does not make a result true. A stock AI check is as strict or as lenient as its instructions say, and no more.
- It does not authorise payment. A decision can carry a payment instruction, and a payer's own service decides whether to act on it. See What each record proves.
Build, test and get hired
The ixo-oracle-builder plugin walks you through it, starts on testnet, and asks before every
chain write.
The full guide for @ixo/decision-oracle, from an empty directory to a deployed oracle.
Unit tests in Node, then a dry run against a real claim with a preview of the decision.
Get hiredThe four checks a collection passes before your oracle takes its claims, and how asks pick a verifier.