Brian "bex" Exelbierd has published
a blog
post exploring follow-up questions raised by
the recent debate about the use of the LLM-based review
tool Sashiko
in the memory-management subsystem. His main finding is that Sashiko reviews are
bi-modal with regards to whether they contain reports about code not directly
changed by the patch set — most do not, but the ones that do often have several
such comments.
Hypothesis 1: Reviewers are getting told about bugs they didn't create.
Sashiko's review protocol explicitly instructs the LLM to read surrounding code,
not just the diff. That's good review practice — but it means the tool might
flag pre-existing bugs in code the patch author merely touched, putting those
problems in their inbox.
Hypothesis 2: The same pre-existing bugs surface repeatedly. If a known
issue in a subsystem doesn't get fixed between review runs, every patch touching
nearby code could trigger the same finding. That would create a steady drip of
duplicate noise across the mailing list.
I pulled data from Sashiko's public API and tested both.