Which queries have no winner
Run the same query six times, then compare the set of sources each run named. Across our five queries that overlap ranged from 0.472 to 0.700 — and that single number says whether a query has an incumbent, and whether it is worth competing for at all.
The measure
For each query we ran six times and recorded the set of sources the written answer named. Then we compared every pair of runs: how much do those two citation sets overlap? We average those pairwise overlaps and call it run-over-run overlap of the cited set. A high number means the same sources keep getting named. A low number means the answer keeps landing on a different shortlist.
This is a measurement of instability, not of quality. It says nothing about whether a page is good. It says something about whether the answer has settled.
A query whose cited set barely changes between runs is a query somebody already owns. A query where two runs share less than half their sources is a query whose answer is still being decided.
The range across our five queries
- Most stable: 0.700 on “robots.txt allow ai crawler gptbot”. 5 sources in the pool, two or three of them named per run, and one source named in every single run.
- Least stable: 0.472 on “best ai ppt maker”. 14 sources in the pool, and only one of them named in all 5 usable runs.
- Same model, same temperature, same day. The only variable is the query.
The unstable one, in detail
On “best ai ppt maker” the pool held 14 distinct URLs across 5 usable runs, and only one of them — ppt.ai — was named in all five. 8 of the 14 were never named at all. Two runs of that query typically share less than half of the sources they cite between them.
Read as a decision rather than a score, that number says: nobody has this query locked. It is expensive to measure — six runs per query is not free — and it is the only honest basis for deciding where to spend the next month.
The stable one, and what it warns against
The most stable query in our set is a technical one about robots.txt directives, at 0.700. 5 sources in the pool, one of them named in every run. If you write a page for that query, you are competing against an answer that has already been decided — and the first thing you would need to beat is not the other pages but the fact that the engine already knows what it is going to say.
What this does not tell you
A high overlap can mean the answer is settled, or it can mean the pool is tiny and there is nothing else to pick. We print the pool size next to the overlap on every query page for exactly that reason — 5 sources at 0.700 is a different situation from 14 sources at 0.472, and the number alone cannot distinguish them.
Questions this piece raises
What exactly is run-over-run overlap?
Take the set of sources each run named, compare every pair of runs, and average the Jaccard overlap of those pairs. 0.70 means two runs typically share seven of the ten sources they cite between them; 0.47 means they share fewer than half. It is computed on the cited sets, not on the retrieval pools.
Can I use this to pick keywords?
As one input, yes — a low overlap means the cited set is still moving, so there is room. It is not a demand signal: a query can be unstable and also be something nobody searches. We do not have search volume data and do not pretend to.
Why only five queries?
Each one costs six retrieval runs, and the whole point of publishing the method is that the cost is visible. Five is what we could run honestly in one sitting; more is a matter of budget, not of method.
Check your own page
Enter one URL and get the eight checks, including the ones that fail. Free, no signup, no score.