How Many Prompts Should You Track for AI Visibility?

By Abiot Y. Derbie - Founder, RankEcho - Updated 2026-09-13

{
  "schemaVersion": 1,
  "purpose": "descriptive-monitoring",
  "population": "declared-panel-only",
  "inference": "not-estimated",
  "selectionPriority": "coverage-floor-then-breadth-then-repeats",
  "intentBuckets": [
    { "id": "alternatives", "minPrompts": 2, "maxPrompts": 8 },
    { "id": "integration", "minPrompts": 2, "maxPrompts": 8 },
    { "id": "pricing", "minPrompts": 2, "maxPrompts": 8 },
    { "id": "security", "minPrompts": 2, "maxPrompts": 8 }
  ],
  "surfaces": ["configured-surface-A", "configured-surface-B"],
  "windows": ["baseline", "follow-up"],
  "maxRepeatsPerWindow": 4,
  "promptSlotLimit": 24,
  "attemptBudget": 240,
  "reserveAttempts": 24,
  "reviewSecondsPerAttempt": 30,
  "reviewBudgetSeconds": 7200
}

Which count are you deciding?

A monitoring panel needs useful coverage before it needs a larger total. Define the product, buyer, market and decisions you intend to inspect. Choose traceable questions using the SaaS portfolio, then set a minimum number of distinct questions for each required intent. A paraphrase that adds no relevant decision should not fill an empty coverage slot.

Keep four quantities in the handoff: unique prompt wording, planned answer attempts, usable observed answers and independent units for a particular analysis. The last two are unknown before collection and design assessment. A repeated request can produce another answer record without adding another buyer question or independently assigned source cluster.

What can the same collection budget buy?

Suppose the team has 24 prompt slots, capacity for 240 attempts across two windows, and 120 minutes for answer review. It holds 24 attempts in reserve and budgets 30 seconds per attempt, including that reserve. Each of the three plans below schedules 192 attempts and retains capacity for 24 reserve attempts: 216 total, or 108 review minutes. Another 24 attempts and 12 minutes remain unallocated.

The wider panel covers more approved questions. The deeper panel revisits fewer questions more often within each window. Choose breadth when important buyer decisions remain uncovered; consider repeats when the selected questions are relevant but their answer behavior needs closer inspection. These are operating priorities. The table does not rank statistical precision, and two repeats are not enough to characterize every kind of variability.

How do you adapt and run the planner?

Copy the JSON and name up to four distinct intent buckets. Give each a minimum and maximum of one to eight prompts, then name the actual surfaces and windows. Record the precise product, locale, account/search settings, model where exposed, dates and prompt IDs in your collection protocol; the capacity file's surface and window labels are only identifiers. Keep each context comparable across its own windows.

Enter your prompt-slot limit, total attempt capacity, reserve allowance, review seconds per attempted answer and total review seconds. Capacity is measured in attempts, not subscription credits or API tokens. Check how your provider charges repeats, models, locales and retries separately. If review time varies substantially or you need unequal surface frequencies, split the work into separate plans and reconcile their combined capacity yourself.

From a repository checkout, run python3 test/fixtures/sample_budget/planner.py --plan path/to/plan.json. With no --plan argument it uses the included example. Add --exercise to reproduce the bundled checks. Python's standard library is sufficient; the helper makes no provider requests. It returns a proposed slot schedule with null outcomes, not a queue submitted to an account.

The declared selection rule maximizes the smallest bucket count, then total prompts, then repeats. Ties use sorted bucket identifiers. It selects six prompts in every bucket and two repeats for this example. Equal bucket priority is a teaching choice; inspect the equal-budget options and change the planning approach if your decisions have different priorities. An empty feasible set means the requested minimum coverage does not fit: change the budget or explicitly narrow the scope, then save a new plan.

What was actually tested?

The September 13, 2026 execution checked all 9,604 count/repeat combinations permitted by the example: seven choices for each of four buckets, across four repeat counts. Of these, 5,137 fit all constraints. The selected plan contains 192 distinct planned cells, and reordering buckets, surfaces and windows produces the same result.

Boundary checks admit the exact minimum plan, hold it when either attempt capacity or review time is one unit short, and preserve coverage floors. The exercise also confirms that increasing both capacities retains at least as many feasible plans. Ten malformed inputs are rejected, including duplicate surfaces, fractional slots, an excessive reserve, an all-buyers population claim and a confirmatory-experiment purpose.

These results verify arithmetic and the stated selection policy on fictional inputs. No AI answers, observed failure rate, provider billing, measured review speed, market coverage or effect estimate were produced. The retained source and execution receipt allow the calculation to be reproduced.

Why is this not a confidence or power calculator?

A large answer count does not establish that a handpicked panel represents all buyer questions. AAPOR's survey guidance distinguishes probability-sample error from uncertainty that depends on a model for a nonprobability sample. Applying that principle here, describe the declared panel and justify any wider inference separately. The planner leaves margin of error, power and independent-unit count unset.

Gumshoe's sample-size explainer treats repeated answers and pooled prompt/model observations as a way to estimate visibility precision. Before using such a calculation, define exactly which probability or weighted average you mean and justify the dependence and stability assumptions. Google documents differences between its own AI search surfaces; a multiplication of surfaces is useful workload arithmetic, but cannot itself establish a shared answer probability.

For a page-change experiment, start with the unit that receives treatment. More repeats do not create more independently assigned pages or source clusters. NIST's proportion-test example illustrates that a statistical sample requirement depends on a baseline, the change of interest and error/power choices. Its formula is not automatically applicable to clustered prompt observations. A suitable design and analysis review comes before a confirmatory claim.

What should you save before collection?

Retain the chosen counts, complete wording, source of each question, named contexts, scheduled windows, review owner and reserve policy. Use reserve capacity for predeclared recovery steps; preserve each failed attempt and its replacement linkage. Never rerun only unfavorable answers until they improve. The reserve is capacity, not an expected number of successes or an instruction to hide failures.

Report scheduled, attempted, usable and unavailable observations separately for each surface and window. Use the fixed-prompt retest to collect and match those records. If wording or scope changes later, version the panel and use the drift guide before comparing totals. More collected answers cannot repair a changed denominator by themselves.

Preview RankEcho's Proof Loop for ongoing same-prompt observations around a recorded shipment. Keep this external capacity plan alongside the workflow and verify your configured engines and plan limits. Budget optimization, prompt-slot scheduling, statistical power and causal estimation are not presented as native RankEcho features.

Frequently asked questions

Sources reviewed

Material technical claims below were checked against primary provider documentation. The sources support the documented control or signal, not a guarantee of indexing, ranking, an AI impression, or a citation.