There is no Search Console for ChatGPT. No impression count, no query report, no notification when an assistant tells a buyer your product does something it does not do.
That is the observability gap, and it is the reason most AI visibility reporting is either absent or wrong. Teams either do not measure at all, or they ask an assistant a question once, screenshot the answer, and treat it as a reading.
A single answer is not a reading. The research on this is now unusually clear, and it says that one observation of AI visibility is close to statistically worthless, for reasons that have nothing to do with how well the check was performed.
This guide covers why one check tells you almost nothing, how many runs and how many days the evidence says you actually need, what to record on each one, which platforms to monitor and why they disagree, the technical monitoring most teams skip entirely, and how to turn any of it into action.
Why One Check Tells You Almost Nothing
The most useful study in this area comes from researchers at the University of St. Gallen, published in April 2026, which tracked four AI search engines across four verticals over a 45 to 46 day window and then re-ran identical prompts within single days to separate model randomness from genuine change.
The headline finding is that the sources cited in AI answers turn over constantly. Day-to-day source overlap, measured as Jaccard similarity, averaged between approximately 0.34 and 0.42 across campaigns, meaning roughly 65 percent of cited sources changed from one day to the next.
The obvious objection is that the web changed underneath the measurement. The authors tested that directly by issuing the same prompt up to ten times within a 24 hour window, and found pairwise source overlap of approximately 0.32 to 0.43.
That is the same range as the day-to-day figures. Most of the instability is not the web moving. It is the model producing different answers to an identical question minutes apart.
Brand mentions were somewhat steadier than source citations, with day-to-day overlap of roughly 0.45 to 0.59, but still far from stable.
The practical consequence is the one the paper’s title states. Visibility in AI search is a probability of being mentioned across repeated runs, not a position you occupy. Treating it like a ranking produces reports that swing wildly for no reason anyone can explain, which is a fast route to losing credibility internally.
That instability is also why the underlying sourcing matters more than any single answer, a mechanism covered in this breakdown of how AI chatbots source information about brands.
What The Research Says About How Much Measurement Is Enough
The same paper ran convergence analysis to answer the question every team asks next, which is how many checks constitute a real measurement. The answers are higher than almost anyone is currently doing.
1. A single run is statistically uninformative
The authors calculated the standard error of a per-brand detection rate estimated from a given number of runs. At one run, the standard error was approximately 0.370.
Their own interpretation is blunt. A true detection rate of 50 percent could appear anywhere across the full range in a nominal 95 percent interval from a single observation.
If your current process is one prompt asked once, the honest description of your data is that it contains no usable information about your visibility level. It contains an anecdote, which is occasionally useful as an example and never useful as a metric.
2. Seven runs per prompt is the practical floor
Standard error fell below 0.10 at approximately seven runs per prompt, giving a 95 percent confidence interval of roughly plus or minus 0.158, and below 0.08 at eight runs.
The curve is steep between one and five runs and flattens afterwards, which means the returns on the first handful of repetitions are large and the returns beyond nine are small.
Seven runs is therefore adequate for detecting large differences, such as a brand appearing in 80 percent of runs against 20 percent. It is not adequate for finely ranking brands whose visibility is similar, and reports should say so rather than implying precision that is not there.
3. Source-level tracking needs more runs than brand tracking
For source coverage, meaning which specific URLs get cited, standard error remained above 0.10 until approximately eight runs, reflecting the higher randomness in URL selection documented above.
That asymmetry has a budget implication. If you want to know whether your brand gets named, seven runs will do. If you want to know which of your pages is doing the work, you need more.
Most programmes want both, so plan for eight and accept that the URL-level numbers will always be noisier than the brand-level ones.
4. Two to four weeks is the minimum observation window
Repetition within a day is not sufficient on its own, because AI search engines update and reindex. The paper ran a rolling-window analysis and found per-brand standard error falling below 0.10 at approximately ten days and below 0.05 at approximately 24 days.
A 14 day window left standard error at roughly 0.080, which the authors describe as sufficient for directional monitoring but not for fine-grained brand comparison.
The recommendation that follows is a rolling two to four week aggregation. Anything shorter is measuring noise and short-lived model updates rather than sustained visibility, which is why monthly reporting cycles work better here than weekly ones.
5. One or two prompts cannot represent a category
Prompt-level variation was substantial. Some prompts produced consistently high overlap across runs, above 0.8, while others stayed persistently below 0.2, within the same campaign.
The authors note that query specificity appears to drive this, with specific product questions producing more consistent source and brand sets than broad generic ones.
Monitoring on one or two prompts therefore measures the idiosyncrasies of those prompts. A portfolio of diverse questions is the only way to get a category-level reading, and the composition of that portfolio matters as much as its size.
6. Brand presence is a more reliable KPI than cited URLs
Because brand-level stability exceeded source-level stability across every campaign, the paper suggests brand presence aggregated across a prompt portfolio is the more dependable headline metric.
Source-level monitoring keeps its value for diagnosis rather than reporting. It tells you which content is earning inclusion, which is what you act on.
Splitting the two this way also protects the report. A headline number built on the more stable measure moves less arbitrarily, and the volatile detail sits underneath it where it belongs.
What To Record On Every Run
The columns matter more than the tooling. Most reporting failures here come from collapsing several different problems into one number.
- Whether an AI answer appeared at all, since some queries do not trigger a generative response and an absent answer is not an absent brand.
- Whether your brand was named in the answer text, which is the mention metric and the more stable of the two headline measures.
- Whether your domain was cited as a source, which is a separate outcome. Semrush’s index of 126 million prompts found the overlap between mentioned brands and cited domains falling as low as 30 percent on Gemini.
- Whether the description was factually accurate, recorded independently of whether you appeared, because zero visibility and bad visibility look identical if you only count appearances.
- Which competitors were named instead of or alongside you, which is usually the column that gets budget approved.
- Which third-party domains were cited, since a recurring set of external pages is your off-site roadmap rather than a detail.
- Where in the answer you appeared, given that citations cluster early in generated responses and a late mention carries less weight than a first one.
Being named without being cited is an authority gap on your own domain. Being cited without being named usually means your brand name is missing from the page the model chose to quote. Those need different work, which is the distinction developed in this guide to getting mentioned in ChatGPT.
Which Platforms To Monitor And Why They Disagree
A single AI visibility score across all platforms is not a simplification. It averages away the only actionable information in the data.
1. Only about 11 percent of cited domains overlap
An analysis of 680 million citations found roughly 11 percent of domains cited by ChatGPT were also cited by Perplexity, with the overlap dropping below 1 percent on individual queries.
Attribution for that dataset is inconsistent across the coverage, with different write-ups crediting Profound and Averi for the same figure, so I am reporting the finding without asserting the source.
Independent studies of smaller samples have reached similar overlap figures, which is why the number is worth taking seriously despite the confusion.
2. Citation behaviour differs sharply between platforms
Muck Rack’s May 2026 analysis of more than 25 million cited links found ChatGPT citing sources in approximately 96 percent of responses, Gemini in 82 percent, and Claude in 55 percent.
The top cited domain differed by platform as well, with Wikipedia leading for ChatGPT, PubMed Central for Claude and Reddit for Gemini.
That last finding matters if you operate in a regulated category, because it implies the answer pool is narrower and more institutional than in consumer verticals.
3. Stability also differs by platform
The St. Gallen data broke source overlap down by engine, finding ChatGPT at approximately 0.233 and Perplexity at 0.282, against Gemini at roughly 0.505.
Gemini was therefore roughly twice as consistent as ChatGPT on which sources it cited for the same prompt. A report that averages those together conceals which platform is genuinely moving.
Different stability also means different run counts are appropriate per platform, which is a refinement worth making once the basic programme is running.
4. Citation concentration differs too
The same study measured inequality in citation distribution using Gini coefficients, finding a mean of approximately 0.715 across engines and campaigns, with Google AI Mode most concentrated at roughly 0.782 and Perplexity least at 0.671.
High concentration means a handful of domains capture most of the available visibility. That changes what a realistic target looks like, and it differs by engine.
The authors’ recommendation is engine-specific baselines rather than one threshold applied everywhere, which is the correct conclusion for reporting as well as for strategy.
5. ChatGPT often does not search at all
A detail from the same paper deserves attention. Approximately 57.8 percent of ChatGPT runs in the dataset returned zero citations, which the authors attribute to the platform suppressing web search on definitional queries.
An answer with no citations is usually training memory rather than retrieval. That distinction determines whether your fix is a same-quarter content problem or a multi-quarter web-wide consistency problem.
Log citation presence as its own field for exactly this reason, since it is the cheapest available signal about which mechanism produced the answer you are looking at.
The Technical Monitoring Most Teams Skip
Content monitoring assumes the crawlers can reach you. That assumption is wrong often enough to check first, because no amount of content work fixes a blocked retrieval agent.
Most vendors run separate crawlers for training and for retrieval, and blocking the wrong one is the most common self-inflicted wound here. OpenAI operates GPTBot for training, OAI-SearchBot for search indexing and ChatGPT-User for user-initiated fetches.
Anthropic runs a comparable three-way split with ClaudeBot, Claude-SearchBot and Claude-User, each requiring its own directive, so disallowing one does nothing to the others. Perplexity documents PerplexityBot and Perplexity-User. Google-Extended is a training control token rather than a crawler, and blocking it does not affect Googlebot or your Search rankings.
The scale of accidental blocking is worth knowing. Analysis of billions of bot requests across millions of sites reportedly found OAI-SearchBot blocked by approximately 49 percent of sites and ChatGPT-User by around 40 percent, against 62 percent for GPTBot and 69 percent for ClaudeBot.
I have not verified that dataset at source, so treat the percentages as indicative. The direction is consistent with what most audits find, which is that a lot of sites blocked more than they meant to.
Filter 30 days of server logs for those user agents. Look for outright absence and for error responses, since a bot receiving a 403 or a challenge page is functionally blocked even where robots.txt permits it.
Then check the raw HTML of your key pages rather than the rendered view. Facts that appear only after JavaScript executes are facts several retrieval systems will not see.
For Google specifically there is a first-party option worth using. Google’s AI optimization guide points to a Generative AI performance report in Search Console for measuring how content performs in generative features on Search and Discover.
The same guidance adds a caution worth repeating to any vendor pitching you. Google states plainly that no third-party tool has access to its internal ranking or AI systems, and advises scepticism toward tools claiming otherwise.
What Not To Do When Monitoring
Several habits actively degrade the quality of the data, and most of them feel like diligence at the time.
- Do not react to a single answer. Given a single-run standard error near 0.370, one bad response is not evidence of a trend, and escalating it burns credibility you will need later.
- Do not change the prompt wording between checks. Changing the prompt changes the answer and destroys any ability to say whether something improved. Freeze the phrasing even when a better version occurs to you.
- Do not test using follow-up turns. Start a fresh conversation each time, because a follow-up measures something a buyer never experiences.
- Do not report one cross-platform score. With roughly 11 percent domain overlap and citation rates ranging from 55 to 96 percent, an average across engines describes nothing real.
- Do not read percentage shifts as your own doing. Semrush tracked more than 230,000 prompts over 13 weeks and found ChatGPT’s Reddit citations collapsing from close to 60 percent of responses to around 10 percent inside about six weeks. Platform reweighting of that size dwarfs anything a content programme does.
How to Turn Monitoring Into Action
Monitoring only earns its cost if it changes decisions, and there are three decisions it should be feeding.
The first is attribution of change. Tracking which other domains moved alongside you is what separates a platform-level reweighting from something your team caused, and without that column every movement gets misattributed.
The second is source-level correction. Where a specific outdated article, review or thread supplies a wrong claim repeatedly, that source is the project rather than the answer, because there is no meaningful correction form for an AI response.
The third is expectation setting on timing. Retrieval-based answers can change within days of the underlying sources changing, while claims drawn from training memory persist until a future model absorbs a corrected web. Profound reportedly found a median of around 6.81 days to first citation for newly published pages, though I have not confirmed that figure at source.
Expect the results to be lumpy rather than gradual. AirOps found citation distribution to be bimodal, with approximately 58 percent of pages never cited for any query they appeared in and around 24.7 percent cited every time, leaving only about 17 percent in between.
Finally, be honest in the report about what the numbers can support. Approximately 45 percent of marketing leaders surveyed by Semrush said they cannot accurately measure brand visibility inside AI answers, and only around 9 percent reported tools tracking all relevant metrics across platforms. A programme that states its own confidence intervals is already ahead of most, a point the wider GEO statistics and AEO statistics reinforce.
How Professional Reputation Management Runs This Continuously
Running eight repetitions of a twenty-prompt portfolio across four platforms on a rolling monthly window, while also auditing crawler access and correcting entity records, is more process than most teams can absorb alongside their actual jobs.
The work itself is not exotic. It is a frozen prompt set, enough repetitions to mean something, separate columns for named and cited and accurate, and a documented route to correcting a source when a description drifts.
Underneath it sits identity, because a system has to know which company it is describing before accuracy is even possible, which is the work covered in this explanation of entity SEO and reputation.
Categories with long research cycles need it running permanently rather than in campaigns, which is the case made in this piece on SaaS reputation management, with wider context in these ORM statistics and this explanation of what AEO means for reputation work.
Nadernejad Media Inc. treats visibility and reputation as one connected programme, pairing search work with the monitoring that catches an inaccurate AI description early, an approach set out further in this case for a professional partner.
Handled that way, what an assistant says about your brand stops being a rumour you overhear and becomes a number you can defend in a meeting.
Frequently Asked Questions
1. How many times should I run each prompt?
At least seven for brand-level monitoring and at least eight if you care which of your URLs gets cited, based on the St. Gallen convergence analysis. A single run carries a standard error around 0.370, which makes it uninformative rather than merely imprecise.
2. How often should I report?
Monthly, on a rolling two to four week window. Per-brand standard error in the same study fell below 0.10 at around ten days and below 0.05 at around 24 days, so weekly reporting is largely reporting noise.
3. Can I just use a tool instead?
Tools help with collection and scale, and several track citations across engines. What they cannot do is give you access to internal platform data, and Google states explicitly that no third-party tool has access to its internal ranking or AI systems. Ask any vendor how many runs per prompt they use and over what window, because that determines whether their numbers mean anything.
4. Why does my brand appear in one platform and vanish in another?
Because the platforms read substantially different internets. Roughly 11 percent of cited domains overlap between ChatGPT and Perplexity, citation rates range from about 55 to 96 percent depending on the engine, and citation concentration differs measurably between them. One score cannot represent four platforms.
5. What should I do when an answer is factually wrong about my business?
Work upstream rather than at the answer, since there is no correction form. State the correct fact plainly in server-rendered text on your own site, fix the structured and third-party records that feed the identity layer, then address the specific source supplying the wrong claim.
