Core concepts

Trace

One top-level operation in your application. A trace is the billing unit and the primary object in the UI.

A RAG pipeline that rewrites the query, retrieves, reranks and generates is one trace containing four spans, not four traces.

Span

A step inside a trace. Spans nest arbitrarily. Typical span kinds:

• llm — a model call, with prompt, completion, tokens and cost
• retrieval — a vector or keyword search, with query and returned chunks
• tool — a function the model chose to call, with arguments and return value
• custom — anything else you wrap

Prompt

A named, versioned template. Every save creates an immutable revision. Traces record the exact revision that served them, so you can always answer "which prompt produced this?"

Revisions carry labels such as production or staging. Code fetches by label; you move labels between revisions to deploy.

Dataset

A collection of examples, each with an input and an expected output. Datasets are versioned. The usual way to build one is promoting real traces from production.

Evaluation

A grader plus a dataset. Graders come in three kinds:

• Judge — an LLM scores the output against a rubric
• Assertion — deterministic code returns pass or fail
• Human — a teammate labels it in a review queue

An eval run applies an evaluation to a dataset and produces per-example scores plus an aggregate.

Monitor

A rule over production data that fires an alert. Monitors watch eval scores on sampled traffic, token cost, latency percentiles and error rates.

How they fit together

production traffic
      │
      ▼
   traces ──── promote ────▶ datasets
      │                         │
      │                         ▼
      │                   evaluations ──▶ eval runs ──▶ CI gate
      ▼
   monitors ──▶ alerts

The important property is that it is one data model. You instrument once, and every capability above works off the same traces.