Thanks for this. Gnuinely appreciate the depth here, this is exactly the kind of scrutiny the eval needed.
You've got it right: I'll re-run scoring with ambiguous matches marked as a hard failure (apply_failed) instead of the first-occurrence guess, and check whether dart_pass for steps lands near 0.56 or those rows just shift into failures instead. That's the direct test of your hypothesis, and the cheapest one to run first.
Separately, I want to go after the root cause too. Plan: refinetune all models on a line-numbered diff format. That is: inject line numbers into the code shown to the model, so <SEARCH> can in principle resolve to a specific line instead of relying on the search text being unique in the file. I'll run this two ways, one with line-numbered pretraining, one keeping default pretraining on ordinary code, both then fine-tuned on the numbered git-diff format — to see whether the model can actually pick up and use line numbers reliably (I doubt it, but worth testing), or whether it needs to have seen numbered code from the start. If it works, it should structurally eliminate the duplicate-<SEARCH>-match problem you found, rather than just flagging it after the fact.
Will compare all of it against the current results once it's done. Thanks again for digging this deep. This is a real find.