Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.ixo.world/llms.txt

Use this file to discover all available pages before exploring further.

​
Recipe — wire vitest in two modes

The example oracle ships a single config that runs unit tests by default and switches to integration mode with --mode int. Copy it. File: vitest.config.ts 
import { defineConfig, mergeConfig } from 'vitest/config';
import nestConfig from '@ixo/vitest-config/nest';

export default defineConfig(({ mode }) => {
  if (mode === 'int') {
    const merged = mergeConfig(nestConfig, {});
    merged.test = {
      ...merged.test,
      include: ['test/**/*.int.test.ts'],
      exclude: ['node_modules', 'dist'],
      testTimeout: 120_000,
      hookTimeout: 120_000,
      setupFiles: ['./test/integration/setup.ts'],
      fileParallelism: false,
    };
    return merged;
  }
  return mergeConfig(nestConfig, {});
});
Then in package.json: 
{
  "scripts": {
    "test": "vitest run",
    "test:integration": "vitest run --mode int"
  }
}

​
The two layers

LayerWhat it bootsWhen to use it
UnitNothing — createTestRuntime fakes a RuntimeContextTool input parsing, middleware hooks, sub-agent prompts in isolation
Integration (Tier A)A test runtime — invoke tools directly, no LLM, no HTTPVerifying upstream-API integration deterministically and for $0
Integration (Tier B)Full createOracleApp + real Matrix + real LLMEnd-to-end: discovery, routing, multi-turn behaviour
All three live under apps/qiforge-example/test/integration/ — read weather.int.test.ts for a complete example.

​
Setup file

File: test/integration/setup.ts 
import 'reflect-metadata';
import { fileURLToPath } from 'node:url';
import { dirname, resolve } from 'node:path';
import { config as dotenvConfig } from 'dotenv';
import { expect } from 'vitest';
import { langchainMatchers } from '@langchain/core/testing';
import { Logger } from '@nestjs/common';

const __dirname = dirname(fileURLToPath(import.meta.url));
const appRoot = resolve(__dirname, '../..');

dotenvConfig({ path: resolve(appRoot, '.env') });
dotenvConfig({ path: resolve(appRoot, '.env.integration'), override: true });

expect.extend(langchainMatchers);
process.env.LOG_LEVEL ??= 'warn';
Logger.overrideLogger(['error', 'warn']);
The setup file loads .env first (runtime config), then layers .env.integration on top with override: true (test-only credentials). Then it registers LangChain matchers and quiets logs.
Each .int.test.ts file declares its required env up front and throws on missing values — see the next step. No silent skips.

​
Per-file env gate — fail loud

Every integration test file lists the env it needs and throws at module load if anything is missing.
1

Declare required env at the top of the test file

const REQUIRED_ENV = [
  'MATRIX_BASE_URL',
  'MATRIX_ORACLE_ADMIN_USER_ID',
  'MATRIX_ORACLE_ADMIN_ACCESS_TOKEN',
  'TEST_USER_MNEMONIC',
  'TEST_USER_DID',
  'ORACLE_DID',
  'OPEN_ROUTER_API_KEY',
] as const;

const missing = REQUIRED_ENV.filter((k) => !process.env[k]);
if (missing.length > 0) {
  throw new Error(
    `weather.int.test.ts requires the following env vars: ${missing.join(', ')}`,
  );
}
Source: weather.int.test.ts lines 80-94.
2

Do not use describe.skipIf for env gates

Silent skips hide broken setups. A throw at file load surfaces immediately when .env.integration is missing or incomplete.

​
Unit test — createTestRuntime

import { describe, expect, it } from 'vitest';
import { createTestRuntime } from '@ixo/oracle-runtime/testing';
import { WeatherPlugin } from '../src/plugins/weather/index.js';

describe('WeatherPlugin', () => {
  it('registers get_current_weather at boot', async () => {
    const { runtime } = await createTestRuntime({
      plugins: [new WeatherPlugin()],
      env: { WEATHER_DEFAULT_UNITS: 'celsius' },
    });
    expect(runtime.toolRegistry.toolNames()).toContain('get_current_weather');
  });
});
createTestRuntime resolves plugins, populates registries, and builds a RuntimeContext you can hand straight to tool handlers — but does not boot Nest, talk to Matrix, or call the LLM. Use for fast, focused tests.

​
Tier A integration — direct invoke

import { afterAll, beforeAll, describe, expect, test } from 'vitest';
import {
  createIntegrationRuntime,
  type IntegrationRuntime,
} from '@ixo/oracle-runtime/testing/integration';
import { WeatherPlugin } from '../../src/plugins/weather/index.js';

describe('Tier A — direct invoke', () => {
  let runtime: IntegrationRuntime | undefined;

  beforeAll(async () => {
    runtime = await createIntegrationRuntime({
      plugins: [new WeatherPlugin()],
      user: { did: process.env.TEST_USER_DID! },
    });
  }, 60_000);

  afterAll(async () => {
    if (runtime) await runtime.close();
  });

  test('get_current_weather({ city: "Berlin" }) returns numeric temperature', async () => {
    const raw = await runtime!.invokeTool('get_current_weather', { city: 'Berlin' });
    const result = JSON.parse(raw as string) as { temp: number; city: string };
    expect(result.city.toLowerCase()).toContain('berlin');
    expect(Number.isFinite(result.temp)).toBe(true);
  });
});
Tier A boots the runtime registries against your plugin list but skips the agent loop entirely. Use it to verify env wiring, upstream-API contracts, and config threading.

​
Tier B integration — full agent loop

import {
  ChatClient,
  allCaps,
  createIntegrationOracle,
  mintUserDelegation,
  waitForMatrixLoaded,
  type IntegrationOracle,
} from '@ixo/oracle-runtime/testing/integration';
import * as sdk from 'matrix-js-sdk';

describe('Tier B — agent loop', () => {
  let oracle: IntegrationOracle | undefined;
  let client: ChatClient | undefined;

  beforeAll(async () => {
    const matrixClient = sdk.createClient({
      baseUrl: process.env.MATRIX_BASE_URL!,
      userId: process.env.MATRIX_ORACLE_ADMIN_USER_ID!,
      accessToken: process.env.MATRIX_ORACLE_ADMIN_ACCESS_TOKEN!,
    });
    oracle = await createIntegrationOracle({
      config: oracleConfig,
      plugins: [new EditorPlugin({ matrixClient }), new WeatherPlugin()],
    });
    await waitForMatrixLoaded(oracle, 90_000);

    const delegation = await mintUserDelegation({
      userMnemonic: process.env.TEST_USER_MNEMONIC!,
      oracleDid: process.env.ORACLE_DID!,
      userDid: process.env.TEST_USER_DID,
      capabilities: allCaps,
    });
    client = new ChatClient(oracle.baseUrl, { delegation });
  }, 180_000);

  afterAll(async () => {
    if (oracle) await oracle.close();
  });

  test('weather request triggers get_current_weather', async () => {
    const sid = await client!.createSession();
    const stream = client!.stream(sid, "What's the weather in Berlin?");
    const calls = [];
    for await (const evt of stream) {
      if (evt.event === 'tool_call') calls.push(evt.data);
    }
    expect(calls.some((c) => c.toolName === 'get_current_weather')).toBe(true);
  });
});
Full reference, including multi-turn scenarios: weather.int.test.ts. Cross-plugin chains: agent-scenarios.int.test.ts. Boot smoke: boot.int.test.ts.

​
Why the vitest config looks like that

Integration tests share a single Matrix admin user. Two test files booting in parallel collide on Matrix’s one-time key uploads at the homeserver. Run sequentially.
Real Nest boot, Matrix sync, and LLM round-trips run 5-30s each. 120s leaves headroom for retries; pushing higher means cutting scope, not raising the cap.
client.createSession() is a server-side round-trip. Create once in beforeAll, reuse for every test. Only mint per-test sessions when the test’s whole point is session isolation (first-contact, cross-session recall).
Tier B uses a real model — response wording drifts. Collect tool_call events from the stream and assert which tools fired with which args, not on text output.

​
What not to do

Don’t loosen assertions to mask failures. Broadening a regex, adding “or” clauses, or raising tolerances to make a flaky test pass discards the check that catches the bug. Investigate the real failure.
Don’t edit plugin code to make tests pass. Two test-side retry attempts max per failure, then stop and ask. Plugins are presumed-working production code; tests describe behaviour, not dictate it.
Don’t add skip-real-services flags (skipMatrixInit, skipGracefulShutdown) to integration tests as a speed-up. Integration tests must boot the same way production does — that’s their point.

Write a plugin

The Weather plugin tests live next to its source.

Environment variables

The .env.integration requirements per plugin.

Runtime context

The object handed to your tool handlers in both unit and integration tests.