> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ixo.world/llms.txt
> Use this file to discover all available pages before exploring further.

# Test your oracle

> Two layers — unit tests against a stub runtime, and integration tests that boot the real Nest app against real Matrix and a real LLM.

## Recipe — wire vitest in two modes

The example oracle ships a single config that runs unit tests by default and switches to integration mode with `--mode int`. Copy it.

File: [`vitest.config.ts`](https://github.com/ixoworld/ixo-oracles-boilerplate/blob/main/apps/qiforge-example/vitest.config.ts)

```ts theme={"system"}
import { defineConfig, mergeConfig } from 'vitest/config';
import nestConfig from '@ixo/vitest-config/nest';

export default defineConfig(({ mode }) => {
  if (mode === 'int') {
    const merged = mergeConfig(nestConfig, {});
    merged.test = {
      ...merged.test,
      include: ['test/**/*.int.test.ts'],
      exclude: ['node_modules', 'dist'],
      testTimeout: 120_000,
      hookTimeout: 120_000,
      setupFiles: ['./test/integration/setup.ts'],
      fileParallelism: false,
    };
    return merged;
  }
  return mergeConfig(nestConfig, {});
});
```

Then in `package.json`:

```json theme={"system"}
{
  "scripts": {
    "test": "vitest run",
    "test:integration": "vitest run --mode int"
  }
}
```

## The two layers

| Layer                    | What it boots                                           | When to use it                                                       |
| ------------------------ | ------------------------------------------------------- | -------------------------------------------------------------------- |
| **Unit**                 | Nothing — `createTestRuntime` fakes a `RuntimeContext`  | Tool input parsing, middleware hooks, sub-agent prompts in isolation |
| **Integration (Tier A)** | A test runtime — invoke tools directly, no LLM, no HTTP | Verifying upstream-API integration deterministically and for \$0     |
| **Integration (Tier B)** | Full `createOracleApp` + real Matrix + real LLM         | End-to-end: discovery, routing, multi-turn behaviour                 |

All three live under `apps/qiforge-example/test/integration/` — read [`weather.int.test.ts`](https://github.com/ixoworld/ixo-oracles-boilerplate/blob/main/apps/qiforge-example/test/integration/weather.int.test.ts) for a complete example.

## Setup file

File: [`test/integration/setup.ts`](https://github.com/ixoworld/ixo-oracles-boilerplate/blob/main/apps/qiforge-example/test/integration/setup.ts)

```ts theme={"system"}
import 'reflect-metadata';
import { fileURLToPath } from 'node:url';
import { dirname, resolve } from 'node:path';
import { config as dotenvConfig } from 'dotenv';
import { expect } from 'vitest';
import { langchainMatchers } from '@langchain/core/testing';
import { Logger } from '@nestjs/common';

const __dirname = dirname(fileURLToPath(import.meta.url));
const appRoot = resolve(__dirname, '../..');

dotenvConfig({ path: resolve(appRoot, '.env') });
dotenvConfig({ path: resolve(appRoot, '.env.integration'), override: true });

expect.extend(langchainMatchers);
process.env.LOG_LEVEL ??= 'warn';
Logger.overrideLogger(['error', 'warn']);
```

The setup file loads `.env` first (runtime config), then layers `.env.integration` on top with `override: true` (test-only credentials). Then it registers LangChain matchers and quiets logs.

<Note>
  Each `.int.test.ts` file declares its required env up front and throws on missing values — see the next step. No silent skips.
</Note>

## Per-file env gate — fail loud

Every integration test file lists the env it needs and throws at module load if anything is missing.

<Steps>
  <Step title="Declare required env at the top of the test file">
    ```ts theme={"system"}
    const REQUIRED_ENV = [
      'MATRIX_BASE_URL',
      'MATRIX_ORACLE_ADMIN_USER_ID',
      'MATRIX_ORACLE_ADMIN_ACCESS_TOKEN',
      'TEST_USER_MNEMONIC',
      'TEST_USER_DID',
      'ORACLE_DID',
      'OPEN_ROUTER_API_KEY',
    ] as const;

    const missing = REQUIRED_ENV.filter((k) => !process.env[k]);
    if (missing.length > 0) {
      throw new Error(
        `weather.int.test.ts requires the following env vars: ${missing.join(', ')}`,
      );
    }
    ```

    Source: [`weather.int.test.ts` lines 80-94](https://github.com/ixoworld/ixo-oracles-boilerplate/blob/main/apps/qiforge-example/test/integration/weather.int.test.ts).
  </Step>

  <Step title="Do not use describe.skipIf for env gates">
    Silent skips hide broken setups. A throw at file load surfaces immediately when `.env.integration` is missing or incomplete.
  </Step>
</Steps>

## Unit test — createTestRuntime

`createTestRuntime` returns a `TestRuntime` directly (no `{ runtime }` wrapper). Drive it through that object's methods — `listTools`, `invokeTool`, `invokeMiddleware`, `invokeSubAgent`, `listCapabilities`, `getManifest`, `assertNoCollisions`, `assertManifestValid`.

```ts theme={"system"}
import { describe, expect, it } from 'vitest';
import { createTestRuntime } from '@ixo/oracle-runtime/testing';
import { WeatherPlugin } from '../src/plugins/weather/index.js';

describe('WeatherPlugin', () => {
  it('registers get_current_weather at boot', async () => {
    const runtime = await createTestRuntime({
      plugins: [new WeatherPlugin()],
      config: { WEATHER_DEFAULT_UNITS: 'celsius' },
    });
    expect(runtime.listTools().map((t) => t.name)).toContain('get_current_weather');
  });

  it('has valid manifests and no registry collisions', async () => {
    const runtime = await createTestRuntime({ plugins: [new WeatherPlugin()] });
    expect(() => runtime.assertManifestValid()).not.toThrow();
    expect(() => runtime.assertNoCollisions()).not.toThrow();
  });
});
```

`createTestRuntime` resolves plugins, populates the six registries, and builds a `RuntimeContext` it hands to tool handlers — but does not boot Nest, talk to Matrix, or call the LLM (every ambient service is a mock). Pass plugin env through `config`. Use for fast, focused tests: tool registration, manifest validity, middleware hooks, and sub-agent prompts in isolation.

<Note>
  `createTestRuntime` options take `config` (plugin env, read as `ctx.config`) — not `env`. Its `mocks.fetch` is exposed for handlers that read it explicitly; it is **not** auto-installed onto `globalThis.fetch`. To stub a plugin's upstream HTTP calls deterministically, use the Tier A `createIntegrationRuntime`, whose `fetch` option does replace `globalThis.fetch`. The full `TestRuntime` surface lives in [`create-test-runtime.ts`](https://github.com/ixoworld/ixo-oracles-boilerplate/blob/main/packages/oracle-runtime/src/testing/create-test-runtime.ts).
</Note>

## Tier A integration — direct invoke

```ts theme={"system"}
import { afterAll, beforeAll, describe, expect, test } from 'vitest';
import {
  createIntegrationRuntime,
  type IntegrationRuntime,
} from '@ixo/oracle-runtime/testing/integration';
import { WeatherPlugin } from '../../src/plugins/weather/index.js';

describe('Tier A — direct invoke', () => {
  let runtime: IntegrationRuntime | undefined;

  beforeAll(async () => {
    runtime = await createIntegrationRuntime({
      plugins: [new WeatherPlugin()],
      user: { did: process.env.TEST_USER_DID! },
    });
  }, 60_000);

  afterAll(async () => {
    if (runtime) await runtime.close();
  });

  test('get_current_weather({ city: "Berlin" }) returns numeric temperature', async () => {
    const raw = await runtime!.invokeTool('get_current_weather', { city: 'Berlin' });
    const result = JSON.parse(raw as string) as { temp: number; city: string };
    expect(result.city.toLowerCase()).toContain('berlin');
    expect(Number.isFinite(result.temp)).toBe(true);
  });
});
```

Tier A boots the runtime registries against your plugin list but skips the agent loop entirely. Use it to verify env wiring, upstream-API contracts, and config threading.

`createIntegrationRuntime` accepts more than `plugins` + `user`: `config` (plugin env, defaults to `process.env`), `features`, `delegation`, `capabilities`, `session`, `state`, `identity`, `fetch` (stub upstreams), and `ucan`.

<Warning>
  The default `ucan` stub **throws** on `mintInvocation` / `createInvocationFromDelegation`. If the plugin under test mints downstream invocations (memory, sandbox, skills, composio, …), pass a real adapter — boot an oracle with `createIntegrationOracle()` and hand its `bootedOracle.app.ambient.ucan` through as `ucan`, after seeding the user's delegation into that oracle's `UcanService` cache.
</Warning>

## Tier B integration — full agent loop

```ts theme={"system"}
import {
  ChatClient,
  allCaps,
  createIntegrationOracle,
  mintAuthInvocation,
  mintUserDelegation,
  waitForMatrixLoaded,
  type IntegrationOracle,
} from '@ixo/oracle-runtime/testing/integration';
import * as sdk from 'matrix-js-sdk';

describe('Tier B — agent loop', () => {
  let oracle: IntegrationOracle | undefined;
  let client: ChatClient | undefined;

  beforeAll(async () => {
    const matrixClient = sdk.createClient({
      baseUrl: process.env.MATRIX_BASE_URL!,
      userId: process.env.MATRIX_ORACLE_ADMIN_USER_ID!,
      accessToken: process.env.MATRIX_ORACLE_ADMIN_ACCESS_TOKEN!,
    });
    oracle = await createIntegrationOracle({
      config: oracleConfig,
      plugins: [new EditorPlugin({ matrixClient }), new WeatherPlugin()],
    });
    await waitForMatrixLoaded(oracle, 90_000);

    // Primary auth: a user-signed invocation (Authorization: Bearer +
    // X-Auth-Type: ucan). Downstream authorization: the delegation.
    const invocation = await mintAuthInvocation({
      userMnemonic: process.env.TEST_USER_MNEMONIC!,
      oracleDid: process.env.ORACLE_DID!,
      userDid: process.env.TEST_USER_DID,
    });
    const delegation = await mintUserDelegation({
      userMnemonic: process.env.TEST_USER_MNEMONIC!,
      oracleDid: process.env.ORACLE_DID!,
      userDid: process.env.TEST_USER_DID,
      capabilities: allCaps,
    });
    client = new ChatClient(oracle.baseUrl, { invocation, delegation });
  }, 180_000);

  afterAll(async () => {
    if (oracle) await oracle.close();
  });

  test('weather request triggers get_current_weather', async () => {
    const sid = await client!.createSession();
    const stream = client!.stream(sid, "What's the weather in Berlin?");
    const calls = [];
    for await (const evt of stream) {
      if (evt.event === 'tool_call') calls.push(evt.data);
    }
    expect(calls.some((c) => c.toolName === 'get_current_weather')).toBe(true);
  });
});
```

<Note>
  `mintAuthInvocation` (sent as `Authorization: Bearer` + `X-Auth-Type: ucan`) is the **primary** auth artifact the runtime expects; `mintUserDelegation` (sent as `x-ucan-delegation`) carries the capabilities plugins use for downstream authorization. `ChatClient` sends both when supplied — that's the production path. A delegation-only `new ChatClient(url, { delegation })` still authenticates via the migration fallback, which is why older tests pass without an invocation.
</Note>

Full reference, including multi-turn scenarios: [`weather.int.test.ts`](https://github.com/ixoworld/ixo-oracles-boilerplate/blob/main/apps/qiforge-example/test/integration/weather.int.test.ts). Cross-plugin chains: [`agent-scenarios.int.test.ts`](https://github.com/ixoworld/ixo-oracles-boilerplate/blob/main/apps/qiforge-example/test/integration/agent-scenarios.int.test.ts). Boot smoke: [`boot.int.test.ts`](https://github.com/ixoworld/ixo-oracles-boilerplate/blob/main/apps/qiforge-example/test/integration/boot.int.test.ts).

## Why the vitest config looks like that

<AccordionGroup>
  <Accordion title="fileParallelism: false" icon="ban">
    Integration tests share a single Matrix admin user. Two test files booting in parallel collide on Matrix's one-time key uploads at the homeserver. Run sequentially.
  </Accordion>

  <Accordion title="120s timeouts" icon="clock">
    Real Nest boot, Matrix sync, and LLM round-trips run 5-30s each. 120s leaves headroom for retries; pushing higher means cutting scope, not raising the cap.
  </Accordion>

  <Accordion title="Share one session across tests in a describe" icon="share-nodes">
    `client.createSession()` is a server-side round-trip. Create once in `beforeAll`, reuse for every test. Only mint per-test sessions when the test's whole point is session isolation (first-contact, cross-session recall).
  </Accordion>

  <Accordion title="Assert structurally on streamed events" icon="list-check">
    Tier B uses a real model — response wording drifts. Collect `tool_call` events from the stream and assert which tools fired with which args, not on text output.
  </Accordion>

  <Accordion title="createIntegrationOracle swaps in cheaper test models" icon="dollar-sign">
    `createIntegrationOracle` installs a default `resolveModel` hook that overrides the `main` and `subagent` roles with cheaper test models; every other role falls through to the production resolver. Pass your own `hooks` to override — caller hooks merge on top per key, so a caller-supplied `resolveModel` wins.
  </Accordion>
</AccordionGroup>

## What not to do

<Warning>
  **Don't loosen assertions to mask failures.** Broadening a regex, adding "or" clauses, or raising tolerances to make a flaky test pass discards the check that catches the bug. Investigate the real failure.
</Warning>

<Warning>
  **Don't edit plugin code to make tests pass.** Two test-side retry attempts max per failure, then stop and ask. Plugins are presumed-working production code; tests describe behaviour, not dictate it.
</Warning>

<Warning>
  **Don't add skip-real-services flags** (`skipMatrixInit`, `skipGracefulShutdown`) to integration tests as a speed-up. Integration tests must boot the same way production does — that's their point.
</Warning>

## Where to read next

<CardGroup cols={2}>
  <Card title="Write a plugin" icon="puzzle-piece" href="/build-an-oracle/develop/write-a-plugin">
    The Weather plugin tests live next to its source.
  </Card>

  <Card title="Environment variables" icon="key" href="/build-an-oracle/reference/environment-variables">
    The `.env.integration` requirements per plugin.
  </Card>

  <Card title="Runtime context" icon="diagram-next" href="/build-an-oracle/reference/runtime-context">
    The object handed to your tool handlers in both unit and integration tests.
  </Card>
</CardGroup>