Checks that report green

The worst thing a check can do is pass because it was never able to fail. Three shapes of that, all of which report clean.

A check that cannot see anything reports that nothing is wrong. One shape of document produces no parts at all, so a coverage check reports zero parts uncovered. That is indistinguishable from a clean result and it means the opposite of one. It is the worst version available, because a clean result there gets read as complete enough to build from.

A catch written for one thing swallows a different fault. A handler written for "not a member" will swallow a render error caused by a missing import. The page renders as a logged out visitor would see it, so the harness reports the true and useless fact that a card did not appear. An hour goes into it.

A pattern with a trailing word boundary silently disables a ban, because that boundary does not match ordinary plurals. Eleven patterns here were affected. The rule reported green while real errors sat in the data.

The self graded version of the same fault

There is a worse variant, where the check works and the thing doing the checking is the thing being checked.

A read back of the finished document agrees with the mechanical count exactly while the work is sparse. Improve the work and it starts calling everything uncovered. Here it went from an exact six to seven out of seven. One of the seven it named was sitting in the object twice, word for word.

So the signal is reliable while the work is bad and junk once the work is good, which is the exact inverse of what anyone needs. The mechanism is drift. The question quietly becomes "the content of this part is not in the object", which is true of every part by design.

The same drift elsewhere. Asked which claims have nothing behind them, an uncapped run returns forty five candidates, one per line of the document, which is the same as returning none. The guard is a comparison rather than a judgement. If every part is named, the answer is worthless.

Stated generally: any headline number that depends on the tool agreeing with itself has to be replaced by something counting from outside.

What to do about it

Go looking for the check that cannot fail, on purpose, as a task. A pattern that never matches anything. A rule with no site to fire at. A validator handed an empty list. All three report green and all three are lying by omission.

The cheap test is to break something deliberately and confirm the check goes red. If it cannot be made to go red, it is not a check.