feat(filter): reduce observed agent confusion in git diff/status/log - #320

Adds a new `tokf doctor` post-hoc analysis subcommand that scans the
local `tracking.db` and reports per-filter health signals:

  - retry-burst detection (same exact command run >=N times within
    a window — the issue's main signal, manually applied in #320)
  - workaround-flag frequency cross-referenced against each filter's
    declared `passthrough_args`, surfacing flags the agent reaches
    for that the filter doesn't already handle
  - empty-output-then-retry pattern (the `git log` ambiguity case)
  - filters with negative token savings (filtered output > raw)
  - composite per-filter health score 0-100 with capped penalties
    so no single signal can dominate

Closes #321 (Phase 1 only — runtime LRU surfacing and interactive
fixes are explicitly deferred to follow-up issues).

Schema migration:
  - new `events.project` column populated from `tokf::history::current_project()`
    so `tokf doctor` defaults to the current repo (matches the
    convention `tokf history` already uses)
  - `idx_events_command_timestamp` and `idx_events_filter_timestamp`
    indexes for the burst-detection query path

Architecture:
  - `crates/tokf-cli/src/doctor/mod.rs` — orchestration + health-score
  - `crates/tokf-cli/src/doctor/queries/` — pure analysis functions
    over slim `EventRow` slices fetched once from the DB
  - `crates/tokf-cli/src/doctor/render/` — human (TTY-aware) + JSON
  - `crates/tokf-cli/src/doctor/noise.rs` — temp-dir / test-fixture
    exclusion + `command_shape()` redaction helper
  - `crates/tokf-cli/src/doctor_cmd.rs` — clap entry point
  - `crates/tokf-cli/src/commands.rs` — `DoctorArgs` (extracted to
    keep `main.rs` under the 700-line hard limit) + `SortByCli`

Tests:
  - 65 unit tests in `tokf::doctor::*` covering burst session split,
    arg-varying exploration negative case, empty-retry windowing,
    legacy-zero-raw exclusion, score monotonicity, sort orderings,
    JSON round-trip, render snapshots
  - 7 end-to-end tests in `tests/cli_doctor.rs` seeding `tracking.db`
    via `TOKF_DB_PATH`, asserting on stdout/exit codes
  - 4 new migration tests in `tracking/tests_project.rs`

Docs:
  - `docs/diagnostics.md` gains a `## tokf doctor` section with an
    example output and per-metric explanation; README regenerated.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

## Summary

Adds a new `tokf doctor` post-hoc analysis subcommand that scans the
local `tracking.db` and reports per-filter health signals. Phase 1 of
#321 — runtime surfacing (Phase 2) and interactive fixes (Phase 3) are
explicitly deferred.

Closes #321

### What it detects

- **Retry bursts** — same exact command run ≥N times within W seconds
(default: ≥5 within 60s). The main signal from the #320 investigation.
- **Workaround-flag frequency** — flags like `--no-stat`, `-p`,
`--name-only` that appear often but aren't in the filter's
`passthrough_args`. Cross-referenced against the filter discovery
system.
- **Empty-output retries** — filtered output looks empty, followed by
the same command again within the window. Signal that the filter should
use `on_empty`.
- **Negative token savings** — filters where the average filtered output
is _larger_ than the raw command output.
- **Per-filter health score** (0–100) — composite of all four signals
with capped penalties so no single dimension dominates.
- **Median arg uniqueness** — per-filter ratio distinguishing confusion
(low) from exploration (high).

### Schema changes

- `events` table gains a `project TEXT NOT NULL DEFAULT ''` column,
populated from `tokf::history::current_project()` (same function `tokf
history` uses). `tokf doctor` defaults to the current project scope;
`--all` to see globally.
- New indexes: `idx_events_command_timestamp` and
`idx_events_filter_timestamp` for burst-detection query performance.

### Architecture

```
doctor/
├── mod.rs           — DoctorReport, run(), score_filter(), median_uniqueness()
├── queries/mod.rs   — pure analysis functions over &[EventRow]
├── render/mod.rs    — human (TTY-aware) + JSON output
└── noise.rs         — temp-dir/test-fixture exclusion + command_shape()
doctor_cmd.rs        — clap entry point
commands.rs          — DoctorArgs + SortByCli (extracted to keep main.rs under 700 lines)
```

### CLI flags

| Flag | Default | Description |
|---|---|---|
| `--json` | off | Machine-readable JSON output |
| `--burst-threshold N` | 5 | Min identical commands to count as a burst
|
| `--window SECS` | 60 | Time window for burst detection |
| `--filter NAME` | all | Scope report to one filter (`git/diff` or `git
diff` both work) |
| `--project PATH` | cwd project | Scope to specific project |
| `--all` | off | Show events from all projects |
| `--include-noise` | off | Don't filter out temp-dir/test-fixture
events |
| `--sort {health,bursts,tokens}` | health | Sort order for the
per-filter table |
| `--no-color` | off | Disable ANSI colour codes |

## Test plan

- [x] `cargo fmt -- --check` clean
- [x] `cargo clippy --workspace --all-targets -- -D warnings` clean
- [x] `cargo test -p tokf` — **1500+ passed**, 0 failed
- [x] 68 unit tests in `tokf::doctor::*` covering burst detection,
workaround flags, empty retries, negative savings, score monotonicity,
sort orderings, rendering, median uniqueness, noise filtering
- [x] 9 end-to-end tests in `tests/cli_doctor.rs`: empty DB, burst
detection, JSON mode, `--all`, `--include-noise`, `--filter` scoping,
`--filter` slash-form normalization, default project scoping, `--help`
sanity
- [x] 4 migration tests in `tracking/tests_project.rs`
- [x] Multi-agent review (4 parallel): Acceptance Criteria (16/16 PASS
after fix), Code Quality (CLEAN/MINOR), Architecture (CLEAN), Test
Coverage (CLEAN)
- [x] Smoke-tested against real `tracking.db` — correctly identifies
git/diff with 67 burst sessions, 4056 events, health score 20

## Commits

1. `feat(cli): tokf doctor — detect filters causing agent confusion` —
main implementation
2. `fix(cli): add median arg-uniqueness metric to tokf doctor` —
addresses MAJOR review finding
3. `test(cli): add integration tests for --filter normalization and
--project default` — addresses MINOR review findings

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

🤖 I have created a release *beep* *boop*
---


<details><summary>e2e-tests: 0.1.28</summary>

### Dependencies


</details>

<details><summary>tokf-common: 0.2.40</summary>

##
[0.2.40](tokf-common-v0.2.39...tokf-common-v0.2.40)
(2026-04-13)


### Features

* **cli:** tokf doctor — detect filters causing agent confusion
([#329](#329))
([694c66c](694c66c))
* **filter:** generic [tree] transform for path-list outputs
([#325](#325))
([322e133](322e133))
</details>

<details><summary>tokf-filter: 0.2.40</summary>

##
[0.2.40](tokf-filter-v0.2.39...tokf-filter-v0.2.40)
(2026-04-13)


### Features

* **filter:** generic [tree] transform for path-list outputs
([#325](#325))
([322e133](322e133))


### Dependencies

* The following workspace dependencies were updated
  * dependencies
    * tokf-common bumped from 0.2.39 to 0.2.40
</details>

<details><summary>tokf-hook-types: 0.2.40</summary>

##
[0.2.40](tokf-hook-types-v0.2.39...tokf-hook-types-v0.2.40)
(2026-04-13)


### Features

* **hook:** forward permission decision reasons to AI tools
([#317](#317))
([cf6b5d7](cf6b5d7))
</details>

<details><summary>tokf-server: 0.2.40</summary>

##
[0.2.40](tokf-server-v0.2.39...tokf-server-v0.2.40)
(2026-04-13)


### Miscellaneous

* **tokf-server:** Synchronize workspace versions


### Dependencies

* The following workspace dependencies were updated
  * dependencies
    * tokf-common bumped from 0.2.39 to 0.2.40
    * tokf-filter bumped from 0.2.39 to 0.2.40
</details>

<details><summary>catalog-types: 0.2.40</summary>

##
[0.2.40](catalog-types-v0.2.39...catalog-types-v0.2.40)
(2026-04-13)


### Miscellaneous

* **catalog-types:** Synchronize workspace versions
</details>

<details><summary>tokf: 0.2.40</summary>

##
[0.2.40](tokf-v0.2.39...tokf-v0.2.40)
(2026-04-13)


### Features

* **cli:** tokf doctor — detect filters causing agent confusion
([#329](#329))
([694c66c](694c66c))
* **filter:** generic [tree] transform for path-list outputs
([#325](#325))
([322e133](322e133))
* **filter:** reduce observed agent confusion in git diff/status/log
([#320](#320))
([6992712](6992712))
* **hook:** forward permission decision reasons to AI tools
([#317](#317))
([cf6b5d7](cf6b5d7))


### Bug Fixes

* **rewrite:** skip filtering when output is redirected to a file
([#323](#323))
([dd66681](dd66681))


### Dependencies

* The following workspace dependencies were updated
  * dependencies
    * tokf-common bumped from 0.2.39 to 0.2.40
    * tokf-filter bumped from 0.2.39 to 0.2.40
    * tokf-hook-types bumped from 0.2.39 to 0.2.40
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: repository-butler[bot] <166800726+repository-butler[bot]@users.noreply.github.com>