# Datasheet — VEB Canonical-135 (13-model board)

A *Datasheet for Datasets* (Gebru et al., 2021) for the **VEB Canonical-135**
sales-negotiation trajectory dataset. This document is intended to travel with
the data so that anyone evaluating, buying, or building on it can judge fitness
for their use without contacting the authors first.

- **Dataset:** `veb-canonical-135`
- **Version / freeze:** canonical-135 (13-model board), grid frozen 2026-08-20
- **Full size:** 3,510 datapoints (one self-contained JSON object per line)
- **Free evaluation sample:** `veb-canonical-135-preview.jsonl` — 78 rows, same schema, full fidelity
- **Benchmark:** The Value Engine Benchmark (VEB)

---

## Motivation

**For what purpose was the dataset created?**
To measure and improve how large language models conduct *multi-turn B2B sales
negotiations* — a setting that stresses long-horizon planning, value framing,
price discipline, and honesty under pressure in ways single-turn benchmarks do
not. Each episode is graded by an LLM judge against an explicit sales-methodology
rubric, producing both a scalar reward (for RLVR / offline RL) and a structured
diagnostic grade (for reward-model training and error analysis).

**Who created the dataset?**
The Value Engine Benchmark (VEB) project.

**Intended uses:** RLHF / RLVR, reward-model training, offline RL, preference-pair
construction, SFT target mining, and negotiation/sales methodology research.

---

## Composition

**What does each instance represent?**
One complete negotiation *episode* between an LLM **seller** (the model under
test) and a stochastic LLM **buyer**, paired with a full judge **grade** and a
scalar **reward**. One instance = one JSONL line, fully self-contained.

**How many instances are there?**
3,510 in the full set. The grid is exact and balanced:

| Axis | Values |
|------|--------|
| Models (seller under test) | 13 (11 closed-weight + 2 open-weight) |
| Scenarios | 9 |
| Seeds | 15 |
| Tracks | 2 (`oob`, `pack`) |
| Per model per track | 135 |
| **Total** | **13 × 9 × 15 × 2 = 3,510** |

Each of the 13 models has exactly 135 rows in `oob` and 135 in `pack`; each of the
9 scenarios has exactly 390 rows.

**What data does each instance consist of?**
Top-level keys: `id`, `env`, `model`, `buyer_sim`, `seed`, `reward`, `resolved`,
`cleared_bar`, `reward_breakdown`, `trajectory`, `format_retries`, `cost`,
`transcript_ref`, `provenance`, `generated_at`, `episode`, `grade`.

- `episode` — the full trajectory: `turns`, `events`, `signals`,
  `internalChannel` (the seller's private reasoning channel), `finalState`, `outcome`.
- `grade` — the full rubric: `meddpicc`, `threeWhys`, `ebEngagement`,
  `mapDatesConfirmedPct`, `champion`, `conditionalCommitmentBeforeProof`,
  `priceIntegrity`, `dvi`, `saleQualityScore`, `failureModes`, `walkAways`,
  `internalChannelReveal`, `notes`.

**Is there a label or target?**
Yes — two, embedded in every row: `reward` (scalar in [0,1]) and `grade`
(structured rubric, including the `outcome` and `saleQualityScore`).

**Outcome distribution (full set):**

| Outcome | Count | Share |
|---------|-------|-------|
| no_decision | 2,768 | 78.9% |
| won | 330 | 9.4% |
| lost | 193 | 5.5% |
| walked_away | 176 | 5.0% |
| buyer_dark | 43 | 1.2% |

The reward signal is dominated by hard negotiations that end without a deal —
this is realistic for enterprise B2B and is *why* the rubric grade matters:
it discriminates quality among the large no-decision mass.

**Is any information missing?**
Integrity is verified: 0 rows missing `episode`, 0 rows missing `grade`, 0
zero-turn rows. The off-roster `gpt-5.5-pro` arm (reasoning-only exploratory) is
intentionally excluded from this release.

**Are relationships between instances made explicit?**
Yes — the paired design. For a fixed (scenario, seed), the `oob` and `pack` rows
of the same model form a matched pair, enabling within-pair methodology-lift
estimation.

**Recommended splits?**
None imposed. For leakage-free evaluation, split by `scenario_id` (hold out whole
scenarios) rather than by row, since seeds within a scenario share the task.

---

## Collection Process

**How was the data collected?**
Generated by executing the VEB harness: each rollout instantiates a scenario, a
fresh seeded stochastic LLM buyer, and the seller under test, then runs the
multi-turn episode to termination and grades it with an LLM judge.

**Stochastic buyer:** every rollout faces a fresh, seeded stochastic LLM buyer
(not a frozen replay), averaged over 15 seeds per scenario for realism.

**Tracks:** `oob` = out-of-the-box seller prompt; `pack` = seller augmented with a
structured sales-methodology pack.

**Over what timeframe?** Rollouts accumulated through the grid freeze on 2026-08-20.

**Provenance:** each row carries `provenance` and `generated_at`; routing/cost
metadata is in `cost`.

**Provenance caveat — dirty generating tree.** 3,510 of 3,510 rows in this
freeze record `provenance.git.dirty = true`. Rollouts were generated inside the
private development repo while that working tree had uncommitted changes, so the
recorded `provenance.git.sha` identifies the checkout each run *started from*,
not the exact code bytes that produced the row. What *is* pinned per row:
`provenance.taskChecksum` (sha256 over the canonical scenario JSON, so the world
the agent negotiated in is exactly identified), `provenance.buyerSim` (frozen
buyer-sim version string), the seed, and the SHA-256 of the released file itself.
This is not retroactively fixable for canonical-135; it is disclosed here rather
than papered over.

---

## Preprocessing / Cleaning / Labeling

**What preprocessing was done?**
Rows are deduplicated by `(scenario_id, model, pack, seed)`, keeping the latest
`generated_at`. The release is filtered to the 13-model live roster. No content
of the episode or grade is altered, truncated, or redacted — rows are emitted
verbatim.

**Was raw data saved?** Yes; the released rows *are* the raw self-contained
datapoints (episode + grade embedded). No lossy transformation stands between the
rollout and the row.

**Labeling:** the `grade` and `reward` are produced by the VEB LLM judge against
the methodology rubric — they are model-generated labels, not human annotations.
Treat them accordingly (see Limitations).

---

## Row status fields

Every row of `veb-canonical-135-preview.jsonl` carries four machine-readable
quality/status fields. These make the quality caveats disclosed in the paper
directly filterable — a downstream RL/fine-tuning user can select only
`training_ready=true` rows without reading the prose.

| Field | Type | Derivation |
|-------|------|------------|
| `judge_type` | `string` (`"llm"` or `"heuristic"`) | Copied verbatim from `grade.judge`. |
| `format_retries` | `int` | Copied verbatim from the top-level `format_retries` field. |
| `flags` | `list[string]` | `"heuristic_fallback"` if `judge_type == "heuristic"`; `"format_retry"` if `format_retries > 0`. Empty list means no degradation. |
| `training_ready` | `bool` | `true` iff `judge_type == "llm"` AND `flags` is empty. |

**Exact derivation (binding):**
```python
judge_type     = row["grade"]["judge"]          # verbatim
format_retries = int(row["format_retries"])     # verbatim
flags = []
if judge_type == "heuristic":
    flags.append("heuristic_fallback")
if format_retries > 0:
    flags.append("format_retry")
training_ready = (judge_type == "llm") and (len(flags) == 0)
```

**Observed counts in the 78-row preview:**

| Metric | Count |
|--------|-------|
| `judge_type == "heuristic"` | 0 |
| `format_retries > 0` | 29 |
| flagged (any flag) | 29 |
| `training_ready == True` | 49 |
| `training_ready == False` | 29 |

**Same derivation over the full 3,510-row release:**

| Metric | Count |
|--------|-------|
| `judge_type == "heuristic"` | 51 |
| `format_retries > 0` | 1,002 |
| flagged (any flag) | 1,046 |
| `training_ready == True` | 2,464 |

Rationale: `flags` is populated only from real, observable degradations — no
thresholds, no inference, no fabrication. Every reason a row is quarantined is
visible in `flags`. The `training_ready` gate is the conservative filter
recommended for RLVR/SFT use.

---

## Uses

**What can the dataset be used for?**
- **RLVR / offline RL** — `reward` is a ready scalar target.
- **Reward-model training** — the structured `grade` supplies rich supervision
  beyond the scalar.
- **Preference pairs** — within a (scenario, seed), rank rows by `reward` or
  `saleQualityScore` to mint pairs.
- **SFT** — mine high-reward `episode.turns` as demonstrations.
- **Methodology research** — the paired `oob`/`pack` design isolates the effect
  of a sales-methodology prompt.

**What should it NOT be used for?**
- Do not treat the judge `grade`/`reward` as ground truth about real-world sales
  outcomes — they are an LLM judge's assessment of simulated negotiations.
- Do not use `internalChannel` (the seller's private reasoning) as if it were
  user-facing content; it is included for analysis of honesty/consistency.
- Not a source of real customer, PII, or proprietary commercial data (there is none).

**Training-readiness gate.** Prefer the `training_ready` flag over assuming every
row is training-clean: rows that fell back to the heuristic judge, or that needed
a format retry, are marked *not* training-ready so downstream training can filter
them. See *Row status fields* above for the exact derivation.

---

## Distribution & Licensing

**Free evaluation sample:** `veb-canonical-135-preview.jsonl` (78 rows) is
released under **CC BY-NC 4.0** so evaluators can inspect real, full-fidelity
rows before any purchase. It is stratified across all 13 models, both tracks, and
8 of the 9 scenarios, and deliberately over-samples the rare
high-signal outcomes (won / lost / walked_away) so the judge rubric is visibly
doing work.

**Full dataset:** `veb-canonical-135.jsonl` (3,510 rows) and its
compressed copy are distributed out-of-band (they exceed platform file limits);
the repository tracks only `manifest.json`, `DATASHEET.md`, and `*.sha256`
records. Full-dataset license is negotiated separately with the dataset owner.

**Integrity:** verify any file against the shipped `*.sha256` before use
(`shasum -a 256 -c <file>.sha256`).

---

## Limitations

- **Simulated, not real.** Buyers and outcomes are LLM-simulated; findings
  transfer to real sales only as far as the simulation is faithful.
- **Judge-labeled.** `grade`/`reward` inherit the judge model's biases; 51 cells
  fell back to the deterministic heuristic judge and are flagged as such.
- **Outcome imbalance.** ~78.9% of rollouts end `no_decision`; downstream training
  should account for this (e.g., reweighting, or using `saleQualityScore` rather
  than raw outcome).
- **English, B2B enterprise framing.** Nine scenarios in specific verticals; not a
  general negotiation corpus.

---

## Maintenance

- **Owner/contact:** the VEB dataset owner (contact before redistribution).
- **Versioning:** this release is the frozen `canonical-135` 13-model board (2026-08-20).
  Enrichment layers (AI-derived features) ship as additive fields with their own
  provenance, leaving these columns unchanged.
- **Erratum policy:** integrity checksums are authoritative; any correction ships
  as a new freeze with an updated `manifest.json`.