AI Agent Evaluation: How to Test Tool Calls and Trajectories

Consider an ERP helpdesk agent that handles credit note requests. A customer service officer types that PT Sinar Jaya wants a credit note for two damaged units on an invoice. The agent replies with a polite, correct-looking summary and a draft number. A final-answer eval scores it as a pass. The trace tells a different story: the agent searched for the customer by a partial name, picked the wrong PT Sinar Jaya out of three, and created the draft against an invoice it never fetched. The answer read well because the model is good at writing; the work underneath was wrong.

That gap is why AI agent evaluation needs a different unit than prompt evaluation. An existing post on this site covers LLM evals in CI for single prompts: a golden dataset, assertions and a pass-rate gate. This one goes one level down, to the multi-step path an agent takes. It covers what to grade in a trajectory, how to build the dataset from production traces, how to make a NestJS agent emit the spans an eval can read, and how to write promptfoo trajectory assertions against them, with OpenAI trace grading and Google ADK evaluation compared at the end. Every assertion name and option below comes from the vendors' current documentation.

Why output-only evals miss agent failures

A final-answer check compares one string to an expectation. An agent run is a sequence of decisions, and most of the expensive failures happen in the middle of it, where a fluent final message hides them. The table maps common agent failures to what each style of check can see.

The pattern is that the final message is the least informative artefact of an agent run. It is written last, by the component most skilled at sounding right. A trajectory check moves the assertion to where the decision was made, which is also where the fix will go.

What to grade in an agent trajectory

Five properties cover nearly every agent failure worth blocking a merge for. Each maps to a different kind of check, and keeping them separate tells you which part of the agent regressed instead of handing you one blended score.

• Tool selection. Did the agent call the tool the task required, and did it avoid tools it must never call for this request? This is the cheapest check and the one that catches routing and prompt regressions first.
• Arguments. Did it pass the right identifiers, amounts and filters? A partial match is enough for most cases; an exact match is worth it on write tools, where an invented extra argument is a real risk.
• Ordering. Did reads happen before the writes that depend on them? Most sequences only need the expected steps in order with other calls allowed between them; a strict exact sequence is for short, safety-critical flows.
• Goal success. Did the run actually achieve what the user asked? This is the one property that usually needs a model grader, because a correct path can still end in a wrong conclusion.
• Cost per run. How many steps and tokens did the run take? A change that keeps the answer right but doubles the tool calls is a regression your invoice will notice before your users do.

Grade the deterministic properties with deterministic checks and reserve the model grader for goal success. A model grader asked whether the tool calls were sensible will be inconsistent where a string comparison would have been exact and free.

Build the eval dataset from production traces

The best trajectory test cases are runs that already went wrong. Invented edge cases test what you imagined; traces test what users actually typed, including the ambiguous company names and the half-remembered invoice numbers that send agents down the wrong path.

1. Sample traces from production weekly, weighted towards runs that ended in a correction, a human handover or a user complaint.
2. For each failing run, write down the trajectory a competent operator would have taken: the tools, their key arguments and their order.
3. Replace personal and customer data with stable fictional values, keeping the shape that caused the failure, such as two customers sharing a name.
4. Freeze the case as a test file in the repository, so every dataset change shows up in a pull request diff next to the code that fixed it.
5. Keep a handful of passing runs as well, so a fix for one failure cannot quietly break the common path.

A case derived from the opening scenario looks like this. The assertions describe the path, not the prose, so the test survives any rewording of the agent's reply.

# evals/helpdesk/credit-note-ambiguous-customer.yaml
# Derived from a production trace. Names and numbers are fictional, but the
# shape that broke the agent is kept: three customers share "Sinar Jaya".
- description: 'Credit note, ambiguous customer name, partial damage'
  vars:
    query: >-
      PT Sinar Jaya Abadi wants a credit note for 2 damaged units
      on INV-2026-08812
  assert:
    # The disambiguating lookup must carry the full legal name, not "Sinar Jaya".
    - type: trajectory:tool-args-match
      value:
        name: find_customer
        args:
          legal_name: 'PT Sinar Jaya Abadi'

    # Read the invoice before drafting against it. in_order (the default)
    # tolerates extra calls in between, e.g. a stock-return lookup.
    - type: trajectory:tool-sequence
      value:
        steps:
          - find_customer
          - get_invoice
          - create_credit_note_draft

    # On the write tool, exact mode rejects invented extras such as force: true.
    # ignore drops the key the agent generates fresh on every call.
    - type: trajectory:tool-args-match
      value:
        name: create_credit_note_draft
        mode: exact
        args:
          invoice_no: 'INV-2026-08812'
          lines:
            - line_no: 1
              qty: 2
          reason: damaged
        ignore:
          - idempotency_key

Instrument a NestJS agent so its trajectory is visible

promptfoo reads trajectories from OpenTelemetry spans. Its tracing documentation says it includes a traceparent header when it calls an HTTP target, and the application behind the target can add child spans under that trace. It recognises tool calls from span attributes such as tool.name, and arguments from attributes such as tool.arguments, parsing string values as JSON when possible. So a NestJS-hosted agent needs two things: OpenTelemetry started before the app, with HTTP instrumentation to pick up the incoming traceparent, and a span around every tool execution.

The tracer file goes first in main.ts. The wrapper is the only place tool calls are executed, which also gives you one place to enforce timeouts and audit logging later.

// src/tracing.ts — import this on the FIRST line of main.ts, before NestFactory,
// or the HTTP module is loaded un-instrumented and the traceparent is ignored.
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import {
  BatchSpanProcessor,
  SimpleSpanProcessor,
} from '@opentelemetry/sdk-trace-base';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { resourceFromAttributes } from '@opentelemetry/resources';

const exporter = new OTLPTraceExporter({
  // promptfoo's built-in receiver listens on 127.0.0.1:4318 by default.
  url: process.env.OTEL_TRACES_URL ?? 'http://127.0.0.1:4318/v1/traces',
});

const sdk = new NodeSDK({
  resource: resourceFromAttributes({ 'service.name': 'erp-helpdesk-agent' }),
  // Under eval, flush every span immediately so it lands before the response.
  spanProcessors: [
    process.env.AGENT_EVAL_MODE === '1'
      ? new SimpleSpanProcessor(exporter)
      : new BatchSpanProcessor(exporter),
  ],
  // Extracts the incoming W3C traceparent, so our spans join promptfoo's trace.
  instrumentations: [new HttpInstrumentation()],
});

sdk.start();

// src/agent/traced-tool.ts — every tool call in the agent loop goes through here.
import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('erp-helpdesk-agent');

export function tracedTool<T>(
  name: string,
  args: Record<string, unknown>,
  run: () => Promise<T>,
): Promise<T> {
  // startActiveSpan parents this span under the request span that
  // HttpInstrumentation created from promptfoo's traceparent.
  return tracer.startActiveSpan(`execute_tool ${name}`, async (span) => {
    // These two keys are what trajectory:tool-used and tool-args-match read.
    span.setAttribute('tool.name', name);
    span.setAttribute('tool.arguments', JSON.stringify(args));
    try {
      const result = await run();
      // Truncate: an invoice with 400 lines should not become a 2 MB attribute.
      span.setAttribute('tool.output', JSON.stringify(result).slice(0, 2000));
      return result;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

// Usage inside the agent loop, for each tool call the model requests:
// const invoice = await tracedTool('get_invoice', call.args, () =>
//   this.erp.getInvoice(call.args.invoice_no as string),
// );

Tool arguments often carry customer names and amounts. promptfoo's receiver accepts a redactAttributes list that replaces matching attribute values before storage, and a retentionDays setting that prunes old traces. Its documentation warns that short patterns over-match, since the key token also matches gen_ai.usage.input_tokens, so name the exact attribute keys you want hidden.

AI agent evaluation with promptfoo trajectory assertions

With spans flowing, the config points an HTTP provider at the NestJS endpoint, enables tracing with the built-in OTLP receiver on its default port 4318, and asserts on the path. The assertion family is trajectory:tool-used, trajectory:tool-args-match, trajectory:tool-sequence, trajectory:step-count and the model-graded trajectory:goal-success.

# promptfooconfig.yaml
description: ERP helpdesk agent — trajectory evals

tracing:
  enabled: true
  failOnReceiverStartFailure: true   # no receiver means no trajectories: fail loudly
  otlp:
    http:
      redactAttributes: ['authorization', 'customer_npwp']
  storage:
    type: sqlite
    retentionDays: 14

prompts:
  - '{{query}}'

providers:
  - id: http
    config:
      url: http://localhost:3000/agent/run
      method: POST
      headers:
        Content-Type: application/json
      body:
        message: '{{query}}'
      transformResponse: json.reply

defaultTest:
  assert:
    # Policy for every request: the agent drafts, a human posts.
    - type: not-trajectory:tool-used
      value: post_credit_note
    # Loop guard: more than 12 tool steps is a regression even if the answer is right.
    - type: trajectory:step-count
      value:
        type: tool
        max: 12

tests:
  - file://evals/helpdesk/*.yaml

A few options do most of the work. tool-used accepts a name, a list of names, or a glob pattern with min and max counts. tool-args-match defaults to partial mode, where expected properties are matched recursively as a subset; exact mode rejects any extra argument, and the defaults and ignore lists let exact mode tolerate pagination defaults and volatile ids such as an idempotency key. tool-sequence defaults to in_order, which allows other calls between the expected steps, while exact requires the traced sequence to match step for step. Every promptfoo assertion can be negated with a not- prefix, which is how a forbidden tool is expressed.

Goal success, custom rules and the cost budget

trajectory:goal-success is model-graded and takes an explicit provider in the documented examples. Give it a goal written as an outcome the grader can verify, such as which customer and invoice the draft must reference, rather than a vague instruction to be helpful. Because it is a model judgement, treat a failure here as a prompt to read the trace, and let the deterministic checks be the ones that block a merge.

Rules the built-in assertions cannot express go in a javascript assertion, which receives the trace as context.trace.spans. The same mechanism handles the cost dimension: sum token usage across spans and fail the run above a ceiling. The ceiling below is an example to calibrate from your own baseline runs, not a recommendation.

# Appended to defaultTest.assert in promptfooconfig.yaml
    # Goal success: model-graded, so it reports; the deterministic checks block.
    - type: trajectory:goal-success
      value: >-
        Create a credit note DRAFT for the customer and invoice named in the
        request, for the damaged quantity only, and tell the user the draft
        number without claiming it has been posted.
      provider: openai:gpt-6-luna
      metric: goal_success

    # Business rule for every case: no draft without having fetched the
    # invoice it credits. Per-case tool-sequence checks then pin the order.
    - type: javascript
      value: |
        const names = context.trace.spans
          .filter((s) => s.attributes['tool.name'])
          .map((s) => s.attributes['tool.name']);
        if (!names.includes('create_credit_note_draft')) return true;
        return names.includes('get_invoice');

    # Cost ceiling from our own GenAI spans. 40k is an example: set it from
    # the p95 of your baseline runs, then tighten.
    - type: javascript
      value: |
        const used = context.trace.spans.reduce((total, s) =>
          total
            + Number(s.attributes['gen_ai.usage.input_tokens'] ?? 0)
            + Number(s.attributes['gen_ai.usage.output_tokens'] ?? 0), 0);
        return used <= 40000;

promptfoo also ships a cost assertion, but its documentation notes it needs the provider to return cost information, which a plain HTTP target usually does not. Counting tokens from your own spans works with any backend, and the step-count check gives a second, model-independent signal for loops.

promptfoo, OpenAI trace grading and Google ADK evals compared

promptfoo is not the only way to grade trajectories. OpenAI and Google both offer trajectory-aware evaluation, each tied to its own agent stack. The differences matter more than the feature lists suggest.

Two caveats change the choice. OpenAI's graders documentation states that it is deprecating graders as part of the evals and fine-tuning workflows they support, and points to its deprecations page for the timeline, so a dashboard-only eval suite is a migration waiting to happen. ADK's documentation notes that tool_trajectory_avg_score, response_match_score and final_response_match_v2 are not supported with user simulation, so simulated multi-turn runs fall back to rubric-based criteria such as rubric_based_tool_use_quality_v1. Checks that live in your repository, against spans you emit yourself, are the ones that survive a vendor change.

The rule to carry is that an agent is graded by what it did, not by what it said. Instrument every tool call as a span, turn the runs that went wrong into frozen trajectories, check selection, arguments and order deterministically, keep the model grader for goal success, and put a ceiling on steps and tokens. The final answer still matters, but it is the last thing to check, not the only one.