AI search change versus noise: a bounded classification
SYNTHETIC EXAMPLE — Asterfall Systems, OrbitDesk, every .example URL, capture, selector, change, prompt, context, and answer mark below are fictional; this is not RankEcho data, customer data, provider evidence, or a benchmark. This Prove method compares three post-change binary feature marks with three same-cell baseline marks. A cell is outside observed variation only when both sets are stable and every post-change mark is the opposite value; all other eligible cells remain within observed variation. These are descriptive classes, not rates, significance tests, causal effects, provider estimates, or product metrics. Site traffic is a separate outcome and is not required.
What question does this classification answer?
It asks whether a fixed cell's observed post-change feature marks satisfy one predeclared descriptive rule relative to that same cell's recorded baseline repeats. The rule can preserve a direction without calling the movement beneficial or caused by the deployment.
It does not estimate a provider population or natural-noise distribution. Six repeats cannot establish statistical significance, and outside observed variation is not a confidence interval or causal effect.
What must be frozen before observation?
Freeze five prompts, two fictional contexts, the exact feature label for each prompt, three repeats per window, exclusion states, and the rule before the 2026-08-28T12:00:00Z deployment.
The baseline window ends before deployment and the post-change window starts afterward. Changed prompts, contexts, features, or collection procedures make a cell contaminated or incomparable rather than favorable.
Worked synthetic fixed-cell ledger
Ten coordinates are planned. Six retain complete baseline and post-change repeats; P07 is contaminated, P08 unavailable, P09 failed, and P10 incomparable. Those four cells carry no feature marks or class.
P01, P04, and P06 are outside observed variation. P06 preserves negative direction, so the table cannot silently turn every outside result into an improvement. P03 and P05 show recorded baseline variation.
| Cell | Prompt / context | Find ID | Baseline repeats | Post-change repeats | Class | Direction |
|---|---|---|---|---|---|---|
| P01 | Q01 / CTX-A | E03 | absent · absent · absent | present · present · present | outside-observed-variation | positive |
| P02 | Q01 / CTX-B | E03 | absent · absent · absent | absent · present · absent | within-observed-variation | mixed-or-none |
| P03 | Q02 / CTX-A | E05 | absent · present · absent | present · absent · present | within-observed-variation | mixed-or-none |
| P04 | Q02 / CTX-B | E05 | absent · absent · absent | present · present · present | outside-observed-variation | positive |
| P05 | Q03 / CTX-A | E03 | present · absent · present | present · present · present | within-observed-variation | mixed-or-none |
| P06 | Q03 / CTX-B | E03 | present · present · present | absent · absent · absent | outside-observed-variation | negative |
| P07 | Q04 / CTX-A | E04 | contaminated | contaminated | not classified | not classified |
| P08 | Q04 / CTX-B | E04 | unavailable | unavailable | not classified | not classified |
| P09 | Q05 / CTX-A | E05 | failed | failed | not classified | not classified |
| P10 | Q05 / CTX-B | E05 | incomparable | incomparable | not classified | not classified |
What do the counts show?
Six of ten fixed cells are comparable: three are outside observed variation and three are within it. The outside group contains two positive directions and one negative direction.
Coverage is reported as 6 comparable cells of 10 fixed. If no cell is comparable, the display is N/A; unavailable, failed, contaminated, and incomparable cells never become no-change observations.
| Measure | Count | Boundary |
|---|---|---|
| Fixed cells | 10 | planned coordinates |
| Comparable | 6 | complete baseline and post-change repeats |
| Outside observed variation | 3 | descriptive rule only |
| Within observed variation | 3 | not proof of no change |
| Contaminated / unavailable / failed / incomparable | 1 / 1 / 1 / 1 | not classified |
| Positive / negative outside direction | 2 / 1 | not an effect estimate |
Which deterministic validation protects the result?
The arithmetic, window order, repeat depth, state exclusions, derived class, negative direction, and zero-comparable display are checked directly from the saved rows.
| Invariant | Required result | Synthetic result |
|---|---|---|
| Fixed coordinates | Five prompts × two contexts | 10 fixed cells |
| Repeat depth | Three baseline and three post-change marks per comparable cell | 36 observed marks |
| Coverage | Non-comparable states stay outside the classes | 6 comparable cells of 10 fixed |
| Class partition | Comparable = outside + within | 3 outside + 3 within = 6 |
| Direction | Outside movement retains positive and negative direction | 2 positive + 1 negative |
| Zero comparable | Do not manufacture a result | Display N/A |
How is this different from adjacent measurements?
Fixed-prompt retest owns the collection of common pre/post cells. Answer volatility owns repeat-to-repeat disagreement within a frozen group. This page consumes those kinds of retained observations and adds only the predeclared descriptive class.
It does not design target and control prompts, calculate a citation rate, estimate observation lag, or establish causal attribution. Those remain separate jobs.
How does this connect to RankEcho?
RankEcho's Proof Loop can retain fixed prompts and dated post-change observations. It does not currently compute this companion table's outside-versus-within observed-variation class as a supported product metric.
Return the full result—including negative, unavailable, failed, contaminated, and incomparable cells—to Find. Another change needs its own directly evidenced defect and bounded repair.
Sources reviewed
Material technical claims below were checked against primary provider documentation. The sources support the documented control or signal, not a guarantee of indexing, ranking, an AI impression, or a citation.
4 claim-level source records
| Claim reviewed | Official source | Review record |
|---|---|---|
| Google says ordinary Search eligibility applies to AI Overviews and AI Mode, important content should be available in textual form, and displayed responses and links can vary. | Google Search: AI features and your website | Checked 2026-09-07 · Current Google Search AI-features documentation · This is Google-specific eligibility guidance. It does not make initial HTML a universal provider requirement or guarantee crawling, indexing, selection, citation, recommendation, ranking, or traffic. · Confidence: High |
| OpenAI states that generative AI output is variable and recommends scoped, task-specific evaluations with explicit objectives, datasets, metrics, comparisons, and retained logs. | OpenAI API: evaluation best practices | Checked 2026-09-07 · Current OpenAI evaluation guidance · This supports freezing a classification rule and retaining repeated observations. It does not define an AI-search change-versus-noise metric, causal design, significance test, or provider-population claim. · Confidence: High |
| NIST's Generative AI Profile describes uncertainty and the immature state of generative-AI measurement as constraints on risk estimation and evaluation. | NIST AI 600-1: Generative AI Profile | Checked 2026-09-07 · NIST AI 600-1 · The profile supports explicit uncertainty and limited interpretation. It does not define prompt-cell outcome labels, a search-provider sampling frame, causal attribution, or this page's descriptive rule. · Confidence: Medium |
| NIST's Engineering Statistics Handbook describes comparing repeated check-standard measurements with historical variability when monitoring a measurement process for change. | NIST handbook: control of a measurement process | Checked 2026-09-07 · NIST/SEMATECH Engineering Statistics Handbook · This is general measurement guidance with assumptions that this synthetic answer panel does not establish. It does not validate a significance threshold, causal effect, or universal AI-search noise model. · Confidence: Medium |
Frequently asked questions
No. It satisfies only this predeclared descriptive rule for the saved repeats and is not a statistical-significance result or confidence interval.
No. The timing and fixed coordinates constrain interpretation. This does not identify a causal effect. Other explanations remain possible.
No. They retain null feature values and stay outside the comparable-cell classes.
An outside result can move from present to absent. Preserving direction prevents the class from being interpreted automatically as improvement.
No. Answer volatility measures repeat disagreement within a frozen group. This companion example applies a separate descriptive pre/post classification.