Context Retrieval

Retrieval Evaluation

Retrieval Dataset v1.0.0 · Agent Dataset v1.1.0。在相同 Ground Truth 與相同 K 下比較 Keyword 與 Hybrid Semantic Retrieval。

目前結論

品質持平，Hybrid 增加延遲

目前小型 fixture 中 Keyword 已達到 100% Recall，沒有證據支持全面改用 embedding。

待審核

Recall@K100%

Precision@K40%

MRR0.90

平均延遲<1 ms

Context333 chars

Recall@K100%

Precision@K40%

MRR0.90

平均延遲1358 ms

Context328 chars

Ground Truth 是每題解題或判斷風險所需的檔案；Patch 成功率不混入檢索分數。

K = 3

案例	Ground Truth	文件數	Keyword 排名	Hybrid 排名	Recall
修正 cancelled 狀態顏色patch-cancelled-badge	`status-badge.js`	20	distractors/status-theme.js status-badge.js distractors/cancellation-service.js	distractors/status-theme.js status-badge.js distractors/cancellation-service.js	100% / 100%
修正分頁少一筆patch-pagination-boundary	`pagination.js`	20	pagination.js distractors/pagination-config.js distractors/calculator-ui.js	pagination.js distractors/pagination-config.js distractors/cursor-pagination.js	100% / 100%
修正 Email 正規化patch-normalize-email	`normalize-email.js`	20	normalize-email.js distractors/email-validator.js distractors/user-profile.js	normalize-email.js distractors/email-validator.js distractors/user-profile.js	100% / 100%
跨檔案傳遞發票幣別patch-invoice-currency	`currency.jsinvoice.js`	21	invoice.js currency.js distractors/invoice-preview.js	invoice.js currency.js distractors/invoice-preview.js	100% / 100%
庫存不可降為負數patch-inventory-floor	`inventory.js`	20	inventory.js distractors/calculator-ui.js distractors/cancellation-service.js	inventory.js distractors/warehouse-alert.js distractors/cart-quantity.js	100% / 100%

Each measured case ranks 19 to 21 files, including a shared corpus with overlapping names, keywords, and responsibilities.
Patch success is reported separately because model behavior is a confounding variable.
政策拒絕與資訊不足案例沒有可由 Query 推導的目標檔案，因此保留在 Agent Benchmark，但不製造假的 Retrieval Ground Truth。