Prompt Regression Testing After a Site Redesign or Migration
Detect AI citation losses after redesigns that standard SEO tools miss.
A site redesign or migration can cause a brand to lose AI citations without setting off its usual SEO alarms. Prompt regression testing makes that loss visible by comparing the same buyer questions across AI platforms before and after the change.
Why a site redesign or migration is an AI citation event, not just an SEO event
Teams working on a migration usually watch for crawl errors, redirect chains, and changes in organic traffic. Yet those checks do not establish whether an AI platform can extract a page’s meaning or select it as a source. A page can remain part of the site while changes to its headings, content, or schema weaken its usefulness for citation.
The disruption starts with access but extends beyond it. GPTBot, ClaudeBot, and Google-Extended are allowed by default in robots.txt unless explicitly blocked with a Disallow rule. A migration that resets the file can introduce restrictions affecting all three. Meanwhile, multiple H1s or mismatched heading nesting can confuse the structural mapping models use before deciding what to cite, causing pages to be skipped or cited inconsistently.
Content changes can create a separate disturbance. Repositioning brand-owned pages changes which topics a model associates with the domain, independent of keyword placement. Incomplete JSON-LD covering Organization, Article, and FAQPage can also weaken the signals used to interpret and surface pages. These changes can happen together during a migration, so the resulting citation loss need not have a single cause. That makes an apparently clean technical launch an incomplete account of what changed.
Why no existing analytics surface shows when AI citation presence collapses
Google Search Console, rank trackers, and traffic dashboards do not show whether a brand has disappeared from answers to a tracked set of buyer questions. A collapse in AI citation presence can therefore produce no alert in the instruments a migration team already watches. Silence is the default. The missing evidence is the answer itself, including whether the brand appeared and what source supported it.
How would a team distinguish migration damage from ordinary volatility? AI citation presence already fluctuates, and a migration amplifies that churn. Without checking answers more than once before the change, the team has no sound basis for judging a weak result afterward. Only a small minority of marketing teams measure AI citations at all, leaving most redesign teams without that baseline. An isolated post-launch check cannot reconstruct it.
The gap exists between platforms as well as between analytics tools. A brand may recover on Perplexity while remaining absent on Claude or ChatGPT. A positive result from one platform cannot establish what happened elsewhere. The team must therefore record each platform’s answers on their own to check whether the whole migration has recovered.
What prompt regression testing is and how it differs from standard prompt evaluation
Prompt regression testing for AI citation means running a fixed, versioned set of brand-relevant buyer queries against multiple AI platforms on a cadence. The team compares results over time to detect losses, recoveries, and changes in how the brand is cited. The question stays stable so that the answer can be compared. A regression may involve citation frequency, position within the answer, or source attribution.
That is different from evaluating prompts inside an AI application. Standard prompt evaluation asks whether an application’s own prompts produce the intended outputs. Citation regression testing checks how systems outside the team’s control show a brand to people who might buy from it. The prompt is the buyer’s question. This makes the practice a marketing measurement discipline.
Its core metric is Share of Model, or SoM, calculated as brand mentions divided by total brand mentions across all AI answers in the tracked set. A brand visibility score instead expresses the share of responses in which the brand appears within its target query cluster. The denominators distinguish these two readings of visibility. Neither replaces the underlying citation record, and each must be read separately for ChatGPT, Perplexity, Gemini, and Claude because their citation rates and source-selection logic differ.
How to build a prompt set that survives a migration and still catches regressions
Start with a library of actual buyer questions across the purchase journey. Internal product descriptions and brand-name lookups do not provide that coverage. The set needs enough prompts on each platform to distinguish that platform’s citation behavior from a genuine gap in visibility.
Each query needs a stable prompt ID and version, with its exact text and grouping preserved. Even small changes to the words can change what gets cited. A run on day 90 after migration must remain comparable to the pre-migration baseline. A handover checklist offers a useful model in its export acceptance logic, which asks what needs to be kept so the team can still use the measurements. That principle applies to continuity during both a site migration and a change of measurement vendor.
ChatGPT Browse, Perplexity, Gemini, Claude, Google AI Mode, and Google AI Overview must remain distinct where they were measured separately before the migration. Mentions, source citations, grounding links, and recommendations need portable definitions, while timeouts, blocked answers, and partial answers must remain visible in the denominator. Otherwise, failed observations can disappear and inflate apparent visibility. Prompt-to-page mappings also need review because product changes and page restructuring can make an old mapping misleading even when the query itself remains useful.
When to run the regression suite and how to pace it around a migration
The sequence begins before the site changes. Establish the baseline through repeated observations at least two full cadence cycles before the migration date. A major migration then becomes a named trigger for immediate re-testing, as do large content refreshes, digital PR campaigns, and product launches. Each can change several layers of citation at once. If the measurement system also changes, keep the pre-migration system running until a parallel run produces explainable parity.
Within that sequence, the recommended rhythm has nested loops. Run the core suite and regression detection each night or through automation, and name who is responsible for each failure alert. Review the panel weekly in standup, focusing on regression count, coverage trend, and unacknowledged alerts. A monthly full catalog walk then retires stale prompts, updates migration documentation, and checks that prompt-to-page mappings still reflect the site.
After launch, the re-test must record more than presence. Document whether the answer includes the brand, which source it cites, where the brand ranks within the answer, and which platform surfaces it. These details separate a returning mention from a restored citation. If mention frequency rises while source attribution remains weak, the result is partial recovery, and the next run needs to retain that classification.
What the content and structure of cited pages must look like post-migration for recovery to hold
The records guide what content the team works on. Recovery requires pages that crawlers can reach and models can extract structured meaning from, as well as brand content in the off-site locations platforms draw from. The on-site checks begin with robots.txt permitting GPTBot, ClaudeBot, and Google-Extended, followed by resolving conflicting headings and mismatched nesting.
Schema needs to be complete. Organization markup should include sameAs and knowsAbout, while Article markup should include author and dates, with FAQPage also covered. These details support how models interpret and surface the pages after migration. A returning citation is more useful evidence when the intended page is accessible and its meaning is clearly organized.
Off-site recovery is a separate task with platform-specific demands. ChatGPT draws a substantial share of citations from third-party directories such as Yelp, TripAdvisor, and MapQuest, while Perplexity tilts toward niche and vertical directories. Brand mention density in third-party sources correlates with citation probability, and broken or removed canonical brand pages weaken the signal those sources point toward. Focusing only on the site can leave citation counts improving on platforms that weight owned pages more heavily while staying flat on those that lean on third-party sources.
How a multi-stage content pipeline with human review integrates prompt regression testing as a standing operation
A suite first run after a migration loses citations starts too late to build the baseline the team needs. The durable approach embeds testing in the content workflow, with migration serving as one trigger for an established system. A multi-stage AI content pipeline moves through research, brief, draft, human editorial review, publication, and post-publish measurement. Post-publish measurement checks whether the page changes citations for its target query cluster.
A human must still review the work before it is published. AI can provide speed and structure, but drafts still need verified factual claims, named authorship signals, and accuracy. Humans also bring their own insights and the brand’s voice. A good citation result still leaves those tasks for humans to do. The editorial gate addresses the quality of what is published, while regression testing addresses how external systems represent it.
The prompt library needs the same discipline as the content pipeline. Prompts are versioned assets with owners, and adding, retiring, or rewording them requires review. An undocumented query change alters the instrument and can break the trend line. Together, these practices let the team ask whether publication moved citation counts or whether a structural migration error suppressed them, without confusing content changes with changes to the test.
How to read regression test results and decide when recovery is real
Mention frequency, citation sources, position, and type must return to baseline across all tracked platforms together. The cited URL needs to be the intended page, with cached or redirect-chain versions excluded. A bare mention is not equivalent to a source citation, grounding link, or recommendation, even if the brand appears just as often.
Platform-specific baselines keep the comparison honest. Repositioning brand-owned pages affects Gemini differently from disrupting directory listings, which hits ChatGPT harder, so recovery timelines can differ with the damage. Perplexity and Claude also have distinct baseline citation rates. An aggregate score can conceal those differences. Elevated timeouts or partial answers mean recovery is not yet complete, even when some citations have returned.
Before declaring success, the team must also rule out instrumentation drift. A changed vendor, query wording, or engine surface definition can create apparent recovery through a change in method. Preserved measurement identity and explainable parallel-run parity are therefore essential. The incident can close when regression counts stay at zero for two full cadence cycles, citation sources match intended URLs, all platforms meet or exceed their pre-migration baselines, citation position and type have recovered, and failure-state rates are back to normal.
