How verification works
Every playbook on this site carries a confidence band. This page says exactly how that band is arrived at, so you can disagree with it.
The short version
There is no “verified” flag anywhere in this system. Nobody can set one, including us. A confidence band is derived from the evidence recorded against one exact, immutable revision, by a rule that is written down below and applied the same way every time.
Evidence belongs to a revision, not to a playbook
When somebody publishes an edit, it does not overwrite what was there. It creates a new revision, and the new revision starts with no evidence at all.
This is the single most important rule here, and it is the one most sites get wrong. If evidence followed the page instead of the version, anybody could accumulate reproductions on a correct procedure and then edit it into something else while keeping the badge. Here that is not a policy that could be relaxed — the database physically cannot express it. An evidence record points at a revision id and has no column for a playbook.
Published revisions never change afterwards, for the same reason. You can read any previous revision, and the evidence you see on it is the evidence that revision actually earned.
The kinds of evidence, and why they are not ranked
These are different claims about different things. A CI run and three people reporting success in three real environments answer different questions, and averaging them into one score would lose the distinction that makes either useful.
- Contributor documentation
- The author's own account of what worked for them. It is evidence that a procedure was written down, never evidence that it works — on its own it can only produce Unverified.
- Reproduction
- Somebody independent followed the author's written steps and reported the outcome. This is the evidence the confidence bands are mostly built from.
- Independent confirmation
- Somebody independent reached the same outcome, by their own route rather than by following the steps. Recorded separately from reproduction because merging the two would overstate what is known.
- Failure observation
- A report that the procedure did not resolve the problem, from somebody who did not work through the whole playbook. Counted, and never filtered out.
- Official reference
- A link to upstream documentation, an issue, or a changelog that supports the cause the playbook names. Supports the explanation; is not a report that the steps run.
- Maintainer attestation
- A statement from somebody who has proved they maintain the project in question. One signal among several — it never outranks reproduction, because being the maintainer is not the same as having tried this on your machine.
- CI execution / matrix execution
- An automated run, on one environment or across a matrix of them. Precise about what it tested and silent about everything else.
How the band is derived
It is a decision procedure, not a weighted score — the conditions are checked in order and the first one that matches wins. A weighted score would let a large amount of weak evidence add up to a strong claim, which is precisely the failure this design exists to avoid.
- Deprecated
Set by a maintainer because the procedure no longer applies. Lifecycle overrides evidence entirely: a deprecated playbook with fifty successful reproductions is still deprecated, because those reproductions happened before whatever changed.
- Superseded
A newer revision has replaced this text. It is not a judgement about whether the procedure worked — the evidence below it is still true about the wording it was recorded against, which is why superseded revisions stay readable at their own URLs rather than redirecting.
- Needs reverification
At least 4 reports, of which at least a third failed. Recent failures pull a revision back regardless of how many successes it has — a fix that fails for one reader in three is not strong evidence of anything except that its environment constraints are wrong.
- Strong evidence
At least 6 independent successes across at least 3 distinct environments.
- Moderate evidence
At least 3 independent successes across at least 2 distinct environments.
- Limited evidence
At least 1 independent success.
- Unverified
Written down, and nobody independent has reported trying it. Every new revision starts here, including revisions of playbooks that were previously well evidenced.
“Independent” excludes the author. An author reporting that their own playbook worked is not evidence that it works for anybody else, so it is recorded, shown, and not counted toward the band. The same applies to a report filed without signing in: it is shown to readers and weighted at zero, because there is nothing to distinguish one anonymous report from ten.
Failures are kept
A report that says the fix did not work is stored, displayed alongside the successes, and counted. It cannot be edited or deleted afterwards — not by the reporter, not by the author, not by an administrator. The tables are append-only at the database level.
This is what makes the successes worth anything. A system where an inconvenient result can quietly disappear produces numbers that mean nothing, and there is no way to tell such a system apart from an honest one by looking at it — which is why this one removes the ability rather than promising not to use it.
Why you rarely see a percentage
Below 5 reports, this site shows the counts and refuses to render a percentage. “100% success” from two reports is a true statement that communicates something false, and the reader who acts on it has been misled by arithmetic rather than by a claim anybody made.
Environment is part of the claim
Evidence gathered on one setup says nothing certain about another, so reports are recorded against an immutable snapshot of the environment they ran in and shown segmented by it rather than averaged. When two environments disagree, that disagreement is the finding — it is stated in words on the evidence page, not smoothed into “mostly works”. A combination nobody has tried is shown as untested, which is an invitation rather than a blank.
What AI does here, and what it does not
AI is used to help structure a submission into steps, to suggest which technologies a contribution is about, to spot likely duplicates, and to assist moderation. All of that is assistance to a person who then decides.
No model can create, weight, or alter a piece of evidence, and none can set or influence a confidence band. That is not a guideline in a document — the tasks a model is permitted to perform are an allow-list in code, and the forbidden ones are named individually with the reason attached.
If you think a band is wrong
Every playbook has an evidence page listing every record the band was derived from, including the failures. If the band looks wrong, the fastest way to change it is to add the evidence it is missing: try the procedure and report what actually happened, whichever way it went.