# Prompt sample budget kit

Start with the buyer decisions you need to cover, then budget unique prompts, named surfaces, repeat slots and collection windows separately. Choose a panel you can collect and review consistently. There is no universal prompt count: 24 prompts run twice and 12 run four times can cost the same while answering different monitoring questions. Neither count establishes a statistical sample size for all buyers or a page-change experiment.

Pack version 1.0 · 17 September 2026. Original example and execution: 13 September 2026.

Guide: https://rankecho.io/resources/prove/prompt-sample-size

## Run the downloaded kit

Download [the ZIP kit](https://rankecho.io/downloads/prompt-sample-budget-kit-v1.zip), extract all files to a local folder and open a terminal in that folder. Keep `planner.py` beside the included `plan.json`. Python 3 is required; only its standard library is used. No installation, account, API key or network access is needed by the helper. On Windows, use `py -3` instead of `python3` if that is your Python launcher.

```sh
python3 planner.py
python3 planner.py --exercise
```

The first command prints the proposed capacity allocation and empty observation slots. The second reproduces the included teaching exercise. These commands print JSON. Redirect the output to a new file to retain your own run; the historical execution in the guide remains dated September 13.

Copy `plan.json` to `my-plan.json` before editing it. Alternatively, save the complete JSON block below as `my-plan.json`. Validate and use your copy with:

```sh
python3 planner.py --plan my-plan.json
```

Keep the included `plan.json` unchanged for `--exercise`; its checks and expected figures are specific to the teaching inputs. `GUIDE.md` is this complete editable guide. `SHA256SUMS.txt` records the original kit contents so you can distinguish the downloaded example from your later edits.

The example evaluates 9,604 candidates and finds 5,137 feasible plans. Its selected 24 prompt slots, two repeats, two surfaces and two windows schedule 192 attempts; 24 reserve attempts raise required capacity to 216 and review allowance to 6,480 seconds. Your inputs can produce a different allocation or no feasible plan.

This is an operating budget, not a sample-size recommendation. It does not collect answers, submit work to an account, price provider credits, establish independent units or calculate a margin of error, statistical power or a treatment effect. Every scheduled outcome remains null.


## Your editable plan

Fictional operating assumptions for four intent buckets, two surfaces and two windows. Replace the scope and limits with your actual collection and review capacity. The planner allocates slots; you supply the approved prompt wording and collection context.

```json
{
  "schemaVersion": 1,
  "purpose": "descriptive-monitoring",
  "population": "declared-panel-only",
  "inference": "not-estimated",
  "selectionPriority": "coverage-floor-then-breadth-then-repeats",
  "intentBuckets": [
    { "id": "alternatives", "minPrompts": 2, "maxPrompts": 8 },
    { "id": "integration", "minPrompts": 2, "maxPrompts": 8 },
    { "id": "pricing", "minPrompts": 2, "maxPrompts": 8 },
    { "id": "security", "minPrompts": 2, "maxPrompts": 8 }
  ],
  "surfaces": ["configured-surface-A", "configured-surface-B"],
  "windows": ["baseline", "follow-up"],
  "maxRepeatsPerWindow": 4,
  "promptSlotLimit": 24,
  "attemptBudget": 240,
  "reserveAttempts": 24,
  "reviewSecondsPerAttempt": 30,
  "reviewBudgetSeconds": 7200
}

```

A tested capacity exercise. The chosen count, 24-attempt reserve and 30-second review allowance are teaching inputs, not measured failure rates, recommended sample sizes or vendor plan limits.

## Which count are you deciding?

A monitoring panel needs useful coverage before it needs a larger total. Define the product, buyer, market and decisions you intend to inspect. Choose traceable questions using the SaaS portfolio, then set a minimum number of distinct questions for each required intent. A paraphrase that adds no relevant decision should not fill an empty coverage slot.

Keep four quantities in the handoff: unique prompt wording, planned answer attempts, usable observed answers and independent units for a particular analysis. The last two are unknown before collection and design assessment. A repeated request can produce another answer record without adding another buyer question or independently assigned source cluster.

| Quantity | Teaching plan | Meaning |
| --- | --- | --- |
| Unique prompt slots | 24 | Six approved questions in each of four intent buckets |
| Planned answer attempts | 192 | 24 prompts × 2 surfaces × 2 repeats × 2 windows |
| Usable answers | Unknown; none collected here | Retain failures and unavailable results separately |
| Independent units | Not established | Requires the sampling or assignment design and dependence assessment |

- [Choose relevant buyer questions](https://rankecho.io/templates/prompt-portfolio-b2b-saas)
- [Review the experiment-design boundary](https://www.microsoft.com/en-us/research/articles/patterns-of-trustworthy-experimentation-pre-experiment-stage/)

## What can the same collection budget buy?

Suppose the team has 24 prompt slots, capacity for 240 attempts across two windows, and 120 minutes for answer review. It holds 24 attempts in reserve and budgets 30 seconds per attempt, including that reserve. Each of the three plans below schedules 192 attempts and retains capacity for 24 reserve attempts: 216 total, or 108 review minutes. Another 24 attempts and 12 minutes remain unallocated.

The wider panel covers more approved questions. The deeper panel revisits fewer questions more often within each window. Choose breadth when important buyer decisions remain uncovered; consider repeats when the selected questions are relevant but their answer behavior needs closer inspection. These are operating priorities. The table does not rank statistical precision, and two repeats are not enough to characterize every kind of variability.

| Plan | Prompts per intent / total | Repeats per window | Scheduled attempts |
| --- | --- | --- | --- |
| More question coverage | 6 / 24 | 2 | 192 |
| Middle allocation | 4 / 16 | 3 | 192 |
| More repeats | 3 / 12 | 4 | 192 |

- [Inspect answer variability](https://rankecho.io/resources/prove/answer-volatility)
- [Inspect the executed equal-budget comparisons](https://github.com/abioty/rankecho/tree/main/test/fixtures/sample_budget)

## How do you adapt and run the planner?

Copy the JSON and name up to four distinct intent buckets. Give each a minimum and maximum of one to eight prompts, then name the actual surfaces and windows. Record the precise product, locale, account/search settings, model where exposed, dates and prompt IDs in your collection protocol; the capacity file's surface and window labels are only identifiers. Keep each context comparable across its own windows.

Enter your prompt-slot limit, total attempt capacity, reserve allowance, review seconds per attempted answer and total review seconds. Capacity is measured in attempts, not subscription credits or API tokens. Check how your provider charges repeats, models, locales and retries separately. If review time varies substantially or you need unequal surface frequencies, split the work into separate plans and reconcile their combined capacity yourself.

From a repository checkout, run python3 test/fixtures/sample_budget/planner.py --plan path/to/plan.json. With no --plan argument it uses the included example. Add --exercise to reproduce the bundled checks. Python's standard library is sufficient; the helper makes no provider requests. It returns a proposed slot schedule with null outcomes, not a queue submitted to an account.

The declared selection rule maximizes the smallest bucket count, then total prompts, then repeats. Ties use sorted bucket identifiers. It selects six prompts in every bucket and two repeats for this example. Equal bucket priority is a teaching choice; inspect the equal-budget options and change the planning approach if your decisions have different priorities. An empty feasible set means the requested minimum coverage does not fit: change the budget or explicitly narrow the scope, then save a new plan.

- [Get the planner and complete input](https://github.com/abioty/rankecho/tree/main/test/fixtures/sample_budget)
- [Freeze the full collection protocol](https://rankecho.io/resources/prove/fixed-prompt-retest)

## What was actually tested?

The September 13, 2026 execution checked all 9,604 count/repeat combinations permitted by the example: seven choices for each of four buckets, across four repeat counts. Of these, 5,137 fit all constraints. The selected plan contains 192 distinct planned cells, and reordering buckets, surfaces and windows produces the same result.

Boundary checks admit the exact minimum plan, hold it when either attempt capacity or review time is one unit short, and preserve coverage floors. The exercise also confirms that increasing both capacities retains at least as many feasible plans. Ten malformed inputs are rejected, including duplicate surfaces, fractional slots, an excessive reserve, an all-buyers population claim and a confirmatory-experiment purpose.

These results verify arithmetic and the stated selection policy on fictional inputs. No AI answers, observed failure rate, provider billing, measured review speed, market coverage or effect estimate were produced. The retained source and execution receipt allow the calculation to be reproduced.

- [Reproduce the planner checks](https://github.com/abioty/rankecho/tree/main/test/fixtures/sample_budget)

## Why is this not a confidence or power calculator?

A large answer count does not establish that a handpicked panel represents all buyer questions. AAPOR's survey guidance distinguishes probability-sample error from uncertainty that depends on a model for a nonprobability sample. Applying that principle here, describe the declared panel and justify any wider inference separately. The planner leaves margin of error, power and independent-unit count unset.

Gumshoe's sample-size explainer treats repeated answers and pooled prompt/model observations as a way to estimate visibility precision. Before using such a calculation, define exactly which probability or weighted average you mean and justify the dependence and stability assumptions. Google documents differences between its own AI search surfaces; a multiplication of surfaces is useful workload arithmetic, but cannot itself establish a shared answer probability.

For a page-change experiment, start with the unit that receives treatment. More repeats do not create more independently assigned pages or source clusters. NIST's proportion-test example illustrates that a statistical sample requirement depends on a baseline, the change of interest and error/power choices. Its formula is not automatically applicable to clustered prompt observations. A suitable design and analysis review comes before a confirmatory claim.

- [Understand the role of sampling assumptions](https://aapor.org/statements/understanding-a-credibility-interval-and-how-it-differs-from-the-margin-of-sampling-error-in-a-public-opinion-poll/)
- [Read the first-party sample-size explainer](https://gumshoe.ai/blog/how-much-data-do-you-need-to-measure-ai-visibility-with-confidence/)
- [Keep named AI surfaces distinct](https://developers.google.com/search/docs/appearance/ai-features)
- [Inspect the inputs to a statistical sample calculation](https://www.itl.nist.gov/div898/handbook/prc/section2/prc242.htm)
- [Plan treatment units and controls](https://rankecho.io/resources/prove/test-and-control-prompts)

## What should you save before collection?

Retain the chosen counts, complete wording, source of each question, named contexts, scheduled windows, review owner and reserve policy. Use reserve capacity for predeclared recovery steps; preserve each failed attempt and its replacement linkage. Never rerun only unfavorable answers until they improve. The reserve is capacity, not an expected number of successes or an instruction to hide failures.

Report scheduled, attempted, usable and unavailable observations separately for each surface and window. Use the fixed-prompt retest to collect and match those records. If wording or scope changes later, version the panel and use the drift guide before comparing totals. More collected answers cannot repair a changed denominator by themselves.

Preview RankEcho's Proof Loop for ongoing same-prompt observations around a recorded shipment. Keep this external capacity plan alongside the workflow and verify your configured engines and plan limits. Budget optimization, prompt-slot scheduling, statistical power and causal estimation are not presented as native RankEcho features.

- [Collect and match the agreed panel](https://rankecho.io/resources/prove/fixed-prompt-retest)
- [Handle a later panel change](https://rankecho.io/resources/prove/prompt-panel-drift)
- [Preview the Proof Loop](https://rankecho.io/demo/proof)
- [Explore ongoing observations](https://rankecho.io/product/proof-loop)

## Are 24 prompts enough for AI visibility tracking?

Twenty-four is the output of this fictional capacity plan and its coverage-first preference. Adequacy depends on the decisions, scope and analysis. Use the planner to find a feasible panel, then assess relevance and any statistical claim separately.

## Do 192 answer attempts mean a sample size of 192?

They mean 192 scheduled observation cells. Usable answers are only known after collection; independence and population inference require a suitable design. The planner estimates neither.

## Should I add more prompts or repeat the existing ones?

Add relevant questions when needed decisions are missing. Consider more repeats when the chosen panel covers the task but answer behavior needs closer inspection. Compare both options against collection and review capacity.

## Does the helper collect answers or import a schedule into RankEcho?

No. It produces a local capacity plan and empty observation slots. Collection, account configuration and the full frozen protocol remain separate operator work.

## Ongoing observations

Preview the Proof Loop — https://rankecho.io/demo/proof. Explore ongoing same-prompt observations — https://rankecho.io/product/proof-loop. Panel reconciliation, assignment design and capacity planning remain external work.