# Benchmark: Hệ 1 → Hệ 2 cascade vs classic rule vs vision LLMs (100 real 1 km² tiles)

Run on 2026-10-18 (UTC 15:29–16:45). Every number below comes from `benchmark.json` (same folder), which also holds every per-tile answer, the prompt, the pre-registration, timestamps and the software versions. Scripts are `data/scripts/bench_01_prep.py` … `bench_05_report.py`, and the raw CLI logs are in `data/bench/logs/`. Total measured LLM spend for the whole experiment was **US$2.86** (320 calls, 0 failures).

## 1. Set-up

- **Data (identical for every contender):**
  - Imagery: Sentinel-2 L2A, 2019-01-28 and 2026-01-26, over the 10×10 km Đắk Glong AOI (see `README.md`).
  - Tiles: 100 tiles of 1 km², each 100×100 px at 10 m, with ids `T<row><col>` and row 0 in the north.
- **Reference per tile:** the fraction of the tile with Hansen GFC v1.13 treecover2000 > 30% and lossyear 2019–2025.
  - Primary label, "significant loss": fraction ≥ 10%. **Prevalence is 76/100.**
  - Because of that prevalence, an "always yes" classifier already scores F1 = 0.86, so **MCC is the honest headline**, not F1.
  - Secondary label: fraction ≥ 20%, prevalence 25/100.
  - Reference mean is 0.158 and median 0.148.
- **Pre-registration:** `data/bench/preregistration.json` was written at 15:29:36 UTC, before any LLM call. It fixes:
  - the reference and the labels;
  - the 0.10 decision threshold, which is the same for all contenders and not tuned on these tiles;
  - the cascade's escalation rule, combination rule and LLM.
- **Contenders:**
  1. **Classic ΔNDVI rule:** a pixel is lost if ΔNDVI ≤ the fold threshold tuned on the training half (east 0.000, west −0.015), inside the forest-2019 mask.
  2. **Hệ 1:** LightGBM, **out-of-fold only**. The W-trained model predicts the east tiles and the E-trained model predicts the west tiles; the 300 m buffer strip uses the averaged model.
     - A pixel is lost if p ≥ its fold-tuned threshold (0.41 / 0.28 / 0.345), inside the forest-2019 mask.
     - Tile fraction = lost pixels / 10,000.
     - Held-out check: re-predicting with the saved model reproduces `prob_oof.npy` (max |Δp| = 3e-8).
  3. **Vision LLMs:** Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5, **all on all 100 tiles**.
     - Call: `claude -p --model … --input-format stream-json --output-format stream-json --tools "" --strict-mcp-config --system-prompt <13 words>`.
     - Each call is one user turn: the prompt plus two base64 PNG chips (512×512, upsampled LANCZOS from 100×100, with the same stretch for both dates).
     - The prompt states the scale (1 km), the dates and the exact Hansen-style definition, including that plantation harvest counts and that already non-forest land does not.
     - It asks for strict JSON `{"loss_fraction","significant_loss","reason"}`.
     - Concurrency 4, with up to 4 attempts per call (every call succeeded on the first attempt). Effort was left at the CLI default, which is generous on quality.
  4. **Cascade (our proposal, pre-registered):**
     - Hệ 1 runs on all tiles.
     - Tiles with 0.05 ≤ Hệ 1 fraction < 0.20 (half to double the decision threshold) are escalated to **Opus 5.5**, which was chosen a priori.
     - For escalated tiles, the final label is the LLM's `significant_loss` and the final fraction is the mean of Hệ 1 and the LLM.
     - Tiles where the LLM and Hệ 1 disagree go to the human queue.
- **Cost, fair "API-only":**
  - Input tokens per call: 2 images × round(512·512/750) = 700, plus 600 prompt tokens (measured by a text-only calibration call), plus 16 system tokens = **1,316**.
  - Output: the measured mean output tokens, **including thinking** (Haiku 421 incl. 355 thinking, Sonnet 66, Opus 253 incl. 194 thinking).
  - List prices: Opus $4/$20, Sonnet $2/$10, Haiku $0.10/$0.50 per M tokens (https://platform.claude.com/docs/en/about-claude/pricing). No Batch discount is applied (Batch would halve it).
- **Cost, measured CLI:** `total_cost_usd`, reported alongside.
  - It includes about 545 tokens of CLI overhead per call and the 1-hour cache-write premium.
  - The default Claude Code configuration (default system prompt and tools) would add about 27k tokens per call: measured at 15.7k cache-write plus 11.2k cache-read for a trivial prompt.
- **Hệ 1 cost:** measured batch time (features on 1 M px plus prediction, 36 threads) × US$1.428/h (AWS c7i.8xlarge on-demand).
  - Imagery download and I/O are excluded for **every** contender, since the LLMs need the same imagery plus chip rendering.
- **National extrapolation:** 14,971,553 ha ≈ 149,716 tiles of 1 km² per pass, × 73 passes per year (5-day revisit). A 1 km² tile is much coarser than Hệ 1's native 10 m pixels.
- **Statistics:** 95% CIs from a bootstrap over tiles (2,000 resamples); paired bootstrap for the differences.

## 2. Results

| Contender | F1 ≥10% | **MCC ≥10%** [95% CI] | Confusion TP/FP/FN/TN | MCC ≥20% | **MAE** [95% CI] | **Spearman ρ** [95% CI] | s / tile | **$ / tile (API-only)** | $ / tile (CLI measured) | $ / national pass | $ / year (73 passes) | Repeat consistency (label, 20 tiles) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Trivial "always yes" | 0.86 | 0.00 | 76/24/0/0 | 0.00 | – | – | – | – | – | – | – | – |
| Trivial constant = ref median (oracle) | – | – | – | – | 0.063 | – | – | – | – | – | – | – |
| Classic ΔNDVI rule | 0.86 | 0.23 [−0.02, 0.45] | 72/19/4/5 | 0.23 | 0.074 [0.062, 0.087] | 0.39 [0.20, 0.56] | 0.0002 | 6e-8 | – | $0.01 | $1 | 100% (deterministic) |
| Claude Haiku 5.5 | 0.83 | 0.16 [−0.06, 0.37] | 67/18/9/6 | 0.17 | 0.107 [0.091, 0.123] | 0.26 [0.06, 0.43] | 4.1 (median latency) | 0.00034 | 0.00058 | $51 | $3,738 | **75%** (5/20 labels flipped) |
| Claude Sonnet 5.5 | 0.87 | 0.44 [0.21, 0.64] | 67/11/9/13 | 0.02 | 0.084 [0.070, 0.099] | 0.32 [0.12, 0.50] | 2.3 | 0.0033 | 0.0081 | $492 | $35,935 | not measured |
| Claude Opus 5.5 | 0.90 | 0.51 [0.30, 0.69] | 74/14/2/10 | 0.09 | 0.110 [0.093, 0.128] | 0.30 [0.11, 0.48] | 5.3 | 0.0103 | 0.0199 | $1,544 | $112,735 | not measured |
| **Hệ 1 (LightGBM, held-out)** | **0.92** | **0.59 [0.42, 0.74]** | 76/14/0/10 | **0.64** | **0.048 [0.041, 0.054]** | **0.90 [0.85, 0.93]** | **0.031** (36 thr) / 0.64 (1 thr, 1 tile) | **0.000012** | – | **$1.8** | **$133** | **100%** (deterministic) |
| **Cascade Hệ 1 → Opus (pre-registered)** | 0.92 | 0.58 [0.38, 0.75] | 75/13/1/11 | 0.46 | 0.065 [0.056, 0.074] | 0.79 [0.68, 0.86] | 2.8 (avg) | 0.0054 | 0.0104 | $805 | $58,756 (52% escalated, hotspot) · **≈ $332–493 at national change rates** | Hệ 1 part 100%; escalated part inherits LLM variability |
| Cascade Hệ 1 → Sonnet (post-hoc) | 0.90 | 0.53 | 71/11/5/13 | 0.53 | 0.053 | 0.85 | 1.2 | 0.0017 | 0.0042 | $258 | $18,820 | – |
| Cascade Hệ 1 → Haiku (post-hoc) | 0.88 | 0.37 | 72/16/4/8 | 0.46 | 0.063 | 0.80 | 2.2 | 0.00019 | 0.00031 | $28 | $2,077 | – |
| ΔNDVI, no forest mask (sensitivity) | 0.86 | 0.00 | 76/24/0/0 | 0.08 | 0.275 | 0.02 | 0.0002 | 6e-8 | – | $0.01 | $1 | 100% |

### Time per national pass

| Contender | Time per pass | Note |
|---|---|---|
| Hệ 1 | **1.3 h** | one 36-thread CPU box |
| Haiku | 171 h serial (≈108 h at concurrency 4) | |
| Sonnet | 96 h serial (≈ 87 h at concurrency 4) | |
| Opus | 222 h serial (≈ 89 h at concurrency 4) | |

A 5-day revisit is 120 h, so at concurrency 4 an LLM-only screen barely keeps up, and only if rate limits allow it.

### Paired comparisons (bootstrap)

- **Hệ 1 vs each LLM, MAE:** Hệ 1 is better than every LLM with probability 1.00.
  - ΔMAE −0.060 vs Haiku, −0.037 vs Sonnet, −0.063 vs Opus.
  - Every CI excludes 0.
- **Hệ 1 vs each LLM, MCC ≥10%:**
  - vs Haiku: +0.43 [0.18, 0.69], significant.
  - vs Sonnet: +0.15 [−0.06, 0.37], p(better) 0.91.
  - vs Opus: +0.08 [−0.12, 0.30], p(better) 0.78.
  - **Only the Haiku difference is significant at n = 100.**
- **Cascade vs Opus-only:**
  - MAE −0.045 [−0.059, −0.031].
  - MCC +0.07 [0.00, 0.19].
  - It costs **48% less** per tile here (52% of tiles escalated).

### Area estimate for the AOI

Hansen reference: 1,582 ha.

| Contender | Estimate | Error |
|---|---|---|
| Hệ 1 | 2,032 ha | **+28%**: thresholds were tuned for recall |
| Sonnet | 1,642 ha | +4%, but its errors cancel out (ρ 0.32) |
| Opus | 2,190 ha | |
| Haiku | 2,120 ha | |
| ΔNDVI | 1,830 ha | |

### National escalation rate

This uses Hansen data across Viet Nam between 10° and 20°N: 96,344 forest cells of ~1 km² (`data/bench/national_tile_change.json`).

- Only **18.9%** of forest tiles had ≥ 10% loss over 2019–2025, against 76% in this AOI.
- In any one year, **12.9–22.9%** of tiles lose ≥ 1% and **1.5–6.5%** lose ≥ 5%.
- This AOI is therefore a severe hotspot, and its 52% escalation rate is a pessimistic upper bound for the cascade.

### What the experiment actually shows

1. **Hệ 1 alone is the most accurate contender on F1, MCC, MAE and Spearman, and it is 28× (Haiku) to 850× (Opus) cheaper per tile and about 75–170× faster than the vision LLMs.**
   - Sonnet has slightly higher precision (0.86 vs 0.84). The classic ΔNDVI rule is even cheaper than Hệ 1, but barely better than "always yes" (MCC 0.23).
   - Among the pre-registered contenders, it is the only one whose MAE beats a constant guess.
   - It is the only one that ranks tiles well (ρ 0.90 vs 0.26–0.32).
   - It is the only one that still discriminates at the ≥ 20% level (MCC 0.64 vs 0.02–0.17).
   - **Caveat:** Hệ 1 is trained on Hansen labels, so "agreement with Hansen" structurally favours it.
2. **Vision LLMs on RGB chips are poor tile-level loss estimators.**
   - Their MAE is worse than the constant guess, and they barely beat "always yes" on F1.
   - Opus is the best LLM on the binary label (MCC 0.51) but the worst on the fraction (MAE 0.110).
   - Haiku is not repeatable: 5 of 20 labels flipped on a re-run, only 10% of fractions were identical, and the Spearman correlation between runs is 0.58.
3. **The pre-registered cascade did NOT improve on Hệ 1 alone for tile loss detection.**
   - MCC 0.58 vs 0.59, MAE 0.065 vs 0.048, ρ 0.79 vs 0.90.
   - Averaging in the LLM's biased fraction made MAE worse.
   - It beats every LLM-only option on MAE at about half the Opus cost, and it routes only 6 disagreement tiles to humans.
   - The LLM's value in the cascade is therefore **not better pixel vision**. It is what this benchmark does not measure: reading permits, the forest-status map and owner records, legal reasoning, and writing the case dossier (`research/system1.md`).
   - **Do not claim that the LLM second opinion improves detection accuracy.**

## 3. LLM failure modes (quantified in `benchmark.json` → `llm_failure_modes`)

- **Green ≠ forest.** On the 55 mosaic tiles where ≥ 50% of the area was already cleared before 2019, the LLMs over-estimate by +0.05 (Sonnet) to +0.12 (Opus).
  - They read crops, coffee and young regrowth as canopy and then count its change as loss.
  - Their error correlates with the share cleared before 2019 (ρ 0.46–0.52). Example: T91.
- **Clear-and-replant between the dates is invisible.** On the 7 heaviest-loss tiles (reference ≥ 0.30), every LLM under-estimates by 0.18–0.24.
  - Land cleared in 2019–21 is green again by 2026, so two RGB snapshots look unchanged.
  - Hệ 1 sees it via NIR/SWIR and the label history. Examples: T40, T67.
- **Anchoring on round numbers:** Sonnet answered 0.12 on 29 of 100 tiles, just above the threshold.
- **Possible reference error, not LLM error:** on T37, Opus describes contour-terraced replanting of the upper half (it estimates 0.45) where Hansen says 0.14. Hansen is not field truth.
- **Clouds and shadows were not tested.** Both scenes are essentially cloud-free over the AOI (SCL 0.003–0.004%).

Examples (before | after | Hansen loss | Hệ 1 loss, with each model's estimate and reason): `data/out/web/benchmark_examples.png`. Chips: `data/out/web/bench_examples/{T37,T67,T91,T40}_{before,after}.png`. All disagreements are listed in `data/bench/llm_disagreements.json`.

## 4. Capabilities that the table does not capture

- **Hệ 1:** a per-pixel 10 m probability that becomes polygons with hectares, centroid and confidence (564 alerts > 0.5 ha in this AOI, `alerts.geojson`). It uses NIR/SWIR and is deterministic and re-auditable.
- **LLM:** one number and a 15-word sentence per km². No polygon, no hectare measurement, RGB only, sampled output, and a model version that changes server-side.

## 5. Honest limitations

1. **The reference is Hansen GFC, not field truth.**
   - Hệ 1 was trained on Hansen labels and uses a Hansen-derived forest-2019 mask, which advantages it in any "agreement with Hansen" metric.
   - Some LLM "errors" may be Hansen errors (e.g. T37).
2. **100 tiles from one hotspot AOI**, with 76% prevalence and spatially autocorrelated tiles, so the CIs are optimistic.
   - Results may differ in intact forest, in the north, or in the rainy season.
   - With n = 100, the MCC differences between Hệ 1 and Sonnet/Opus are not statistically significant.
3. **The LLM prompt is one reasonable design** (v1, not tuned).
   - Not tested: other prompts, few-shot examples, larger context chips, SWIR false-colour chips, and the forest mask given as an image.
   - Any of these could help the LLMs. The F1 and MCC of the LLMs should be read as "zero-shot RGB vision on 1 km chips".
4. **CLI overhead.** Measured CLI cost is about 1.7–2.5× the API-only figure here (about 545 overhead tokens plus the cache-write premium), and it would be far higher with the default Claude Code configuration (about 27k tokens per call).
   - Latency was measured through the CLI under concurrency 4.
   - Opus and Haiku spent 194 and 355 thinking tokens per call, which are billed.
5. **Hệ 1 timing was measured on a shared host under heavy load** (load average 44–55 on 36 threads).
   - Training time (12–76 s per fold) and imagery I/O are excluded.
   - The single-tile, single-thread median is 0.64 s.
6. **Repeatability was measured for Haiku only** (20 tiles × 2 runs). Opus and Sonnet repeatability is unknown.
7. **National extrapolation** is a linear scale-up from one AOI; the cascade's national rate uses Hansen annual change shares as a proxy for escalation. The figures are order-of-magnitude.