AI search monitoring is the repeatable process of running a controlled prompt set across selected AI experiences, recording each answer, and comparing how often and how accurately a brand appears. The goal is not to collect a few screenshots. It is to create observations that can be reproduced, audited, and compared over time.
This guide explains the operating workflow. If you need the category definition, metric taxonomy, or recommended data model first, read What Is AI Search Tracking?.
What Should You Track in an AI Answer?
A brand can appear in several ways. It may be named in the answer body, included in a shortlist, recommended for a specific use case, compared with a competitor, or cited as a source. Those outcomes should not be collapsed into a single yes-or-no field.
For each answer, record at least:
- whether the brand is mentioned;
- where the mention appears;
- whether the brand is recommended, compared, or merely listed;
- which competitors appear;
- how the brand is framed;
- whether the answer cites a brand-controlled page;
- which other domains are cited;
- whether material statements about the brand match approved facts; and
- the platform, mode, model when visible, locale, account state, and test time.
A citation and a mention are different observations. A cited brand page shows that the page was presented as a source in that answer. A brand mention without a citation may come from another source, prior model knowledge, or an inference that is not exposed to the tester.
Step 1: Define the Monitoring Question
Start with a decision, not a tool. A useful monitoring question is narrow enough to determine which prompts, platforms, and metrics matter.
Examples include:
- Are we included when buyers ask for vendors in our category?
- Are our official pages cited for questions we are qualified to answer?
- Does AI describe our target customer and service scope accurately?
- Which competitors appear when our brand does not?
- Did visibility change after a specific set of content and evidence updates?
Write the question, the audience, the market, and the review period before building the prompt library. This prevents a dashboard from filling with observations that do not support a decision.
Step 2: Build a Prompt Matrix
A prompt matrix is a controlled inventory of questions grouped by buyer intent. It should include unbranded prompts because many discovery journeys begin with a problem or category, not a company name.
| Dimension | Example values |
|---|---|
| Journey stage | Learn, compare, shortlist, validate, decide |
| Intent | Category discovery, alternatives, use case, trust, implementation |
| Audience | Marketing lead, founder, SEO lead, procurement reviewer |
| Market | Country, language, or service region being tested |
| Brand condition | Unbranded, branded, competitor-branded |
| Expected answer type | Definition, list, comparison, recommendation, procedure |
Give every prompt a stable prompt_id. Preserve the exact wording during a measurement period. If wording changes, create a new version rather than silently overwriting the old prompt.
A practical first library may contain 20 to 40 high-priority prompts. Depth matters more than volume: each prompt should represent a real question and map to an intended decision.
Step 3: Define the Test Protocol
AI answers vary with time, product mode, model, location, personalization, and conversation context. The protocol should control or record those variables.
For every test cell:
- Start a new conversation unless the test explicitly studies follow-up behavior.
- Use the exact saved prompt.
- Record whether web search or another source mode is active.
- Record the platform, visible model or mode, locale, account state, and timestamp.
- Save the answer text or an approved durable capture.
- Save visible citation URLs and normalize their domains.
- Have a reviewer validate ambiguous mentions, recommendations, and framing labels.
Do not treat one answer as a stable ranking. Use repeated observations and report the sample size. For a baseline, run each priority prompt across the selected platforms and repeat the same matrix on a fixed cadence.
Platform interfaces differ. ChatGPT search responses may show inline citations and a Sources panel. Google says AI Overviews and AI Mode can surface supporting links and may use related searches across subtopics. Perplexity describes its search answers as source-linked. These behaviors can change or vary by experience, so capture what the tester actually sees instead of assuming that every answer supports citations.
Step 4: Use a Consistent Observation Schema
One row should represent one prompt-platform-run observation. Recommended fields are:
| Field | Purpose |
|---|---|
| run_id | Groups observations from one scheduled test |
| prompt_id and prompt_version | Preserves the question and its history |
| platform, mode, model | Identifies the tested experience |
| locale and account_state | Records important context |
| observed_at | Supports time-series comparison |
| valid_response | Excludes errors, refusals, or failed tests from denominators |
| brand_mentioned | Records a qualified brand-name appearance |
| mention_position | Records first, middle, last, or not applicable |
| recommendation_status | Recommended, considered, compared, cautioned, or absent |
| brand_citation_urls | Stores citations to approved brand-controlled domains |
| other_citation_urls | Stores external sources used in the answer |
| competitors | Stores normalized competitor entities |
| framing_labels | Stores controlled attributes such as enterprise, regional, or low-cost |
| claim_checks | Compares material statements with the approved fact set |
| response_capture | Points to the saved answer evidence |
Keep raw evidence separate from derived scores. If a label changes during quality review, the original answer should remain available.
Step 5: Calculate Metrics From Valid Observations
At minimum, calculate mention rate, brand citation rate, prompt visibility coverage, share of answer, and framing accuracy. Use the same denominator rules in every reporting period.
Do not compare a citation rate from a citation-capable search mode with a closed-model answer that exposes no sources. Mark an observation as not applicable when the interface cannot produce the measured outcome.
The formulas and worked calculations are defined in How to Measure GEO Success. The conceptual relationship between mentions, citations, framing, and recommendations is explained in The AI Answer Authority Model.
Step 6: Review Quality Before Comparing Periods
Monitoring data needs editorial quality control. Use a written labeling guide and review at least the ambiguous cases.
Check for:
- aliases that create false negatives;
- incidental words that create false brand mentions;
- redirects and tracking parameters that split one citation domain;
- duplicated answers or failed retries;
- mixed languages or markets;
- unsupported sentiment labels; and
- prompt, platform, or mode changes between periods.
If the protocol changes, annotate the time series. A larger prompt library or a newly added platform can change the aggregate rate even when the underlying visibility has not changed.
Step 7: Set a Monitoring Cadence
Use a cadence that matches the decision cycle. Weekly monitoring may suit an active launch or incident. Monthly monitoring is often enough for ongoing visibility work. A quarterly review can focus on prompt coverage, source gaps, and larger content priorities.
Each report should show:
- the prompt set and platform scope;
- the number of planned, completed, valid, and excluded observations;
- current metrics and the comparable prior period;
- examples of important gains, losses, and framing errors;
- cited-domain patterns;
- competitor changes; and
- the actions that the evidence supports.
Avoid attributing a change to one page edit without supporting evidence. AI answers and retrieval systems change independently of your work.
Manual Tracking, Platforms, and Custom Systems
A spreadsheet is sufficient for a controlled pilot. It makes the prompt set, raw answers, labels, and formulas inspectable. Dedicated AI visibility products can add scheduling, normalized captures, team review, and dashboards. A custom system can support specialized prompts or data joins, but it also requires platform-term review, rate-limit handling, storage controls, and ongoing maintenance.
Choose a tool category after defining the protocol. Automation cannot repair a weak prompt set or inconsistent labels.
Hypothetical Worked Example
The following example is illustrative and does not describe an UnderAI customer.
Northstar Cloud, a fictional B2B software company, defines 24 priority prompts across category discovery, comparison, security validation, and implementation. It tests each prompt in three AI search experiences, producing 72 planned observations per run.
During the baseline, 68 observations return valid answers. Northstar Cloud appears in 17, and its approved domain is cited in 6 of the 52 observations where citations are available. Reviewers also identify four answers that describe the product as suitable for a customer segment outside its approved positioning.
The team does not conclude that one content edit caused those results. It records the baseline, improves the official pages supporting the affected prompts, strengthens factual evidence, and repeats the same matrix one month later. The comparison uses identical prompt versions and reports platform changes separately.
Common Monitoring Mistakes
- Testing only the brand name instead of category and decision prompts.
- Treating a single answer as a rank.
- Combining mentions, citations, and recommendations into one score.
- Changing prompts without versioning them.
- Ignoring locale, search mode, model, or time.
- Counting invalid answers in the denominator.
- Reporting sentiment without a labeling rule.
- Automating before defining the observation schema.
- Claiming causation from a before-and-after comparison alone.
Final Checklist
Before the first reporting cycle, confirm that the team has a decision question, approved prompt matrix, stable test protocol, normalized observation schema, fact set for framing review, metric definitions, evidence-retention policy, and review cadence.
That system turns scattered AI answers into a usable monitoring record. It does not guarantee mentions or citations. It shows where the brand appears, how the answer is supported, and which gaps deserve investigation.