Preserve three different records
AI-answer exports and citation cards can represent the same page with analytics parameters, fragments, parameter-order changes, or a trailing slash. Counting every string as a unique source can distort citation totals and publisher concentration. The remedy is not to discard the original URL.
Keep all three fields. Never replace the observed URL with the comparison key. A verified destination is a later finding that can change when a publisher updates or redirects a page.
Conservative normalization sequence
- Parse only absolute HTTP(S) URLs. Keep invalid rows with an error state instead of silently removing them from the denominator.
- Lowercase the hostname. Preserve path and query casing because a server may treat them as meaningful.
- Remove the fragment. Fragments are not sent in the HTTP request, but the raw observation still preserves the section a user saw.
- Remove a documented analytics allowlist. A practical start is
utm_*,gclid,dclid,fbclid,msclkid,mc_cid,mc_eid, and common_hs*fields. - Retain unknown and content-bearing parameters. Fields such as
id,page,edition,lang,version, orquerycan change the content. - Sort retained parameters. Sort by key and value while preserving repeated entries.
- Handle trailing slashes as a configurable rule. Some servers distinguish
/reportfrom/report/. - Do not infer schemes, redirects, or canonicals. Do not silently force HTTPS, strip
www, merge subdomains, decode paths, or remove extensions.
Synthetic example
The examples below use reserved placeholder domains and are not observed AI citations.
| Observed URL | Comparison key | Logged transformation |
|---|---|---|
https://docs.example.org/guide?utm_source=email#evidence | https://docs.example.org/guide | Analytics parameter and fragment removed |
https://docs.example.org/guide?utm_source=social | https://docs.example.org/guide | Analytics parameter removed |
https://research.example.net/report/?gclid=example | https://research.example.net/report | Click ID and trailing slash removed |
https://updates.example.com/story?edition=morning&utm_campaign=test | https://updates.example.com/story?edition=morning | Content parameter retained; analytics parameter removed |
The first two rows share one comparison key. They remain two observations in the ledger.
Minimum audit ledger
| Field | Purpose |
|---|---|
observation_id | Stable ID for the dated answer observation |
prompt_id and prompt_version | Connect the citation to a frozen prompt set |
answer_system and run_timestamp | Record the observable product surface and collection time |
observed_url | Preserve the exact evidence string |
normalization_policy_version | Make transformations reproducible across reruns |
normalized_key and publisher_hostname | Support duplicate and concentration checks |
transformations | List every rule applied to the row |
verification_status, verified_final_url, publisher_canonical | Keep local normalization separate from later network verification |
Count without collapsing meaning
Report observations, unique URL keys, and unique publisher hostnames as separate measures. Do not call all three “citations.” A prompt set can cite one page across repeated runs, several pages from one publisher, or several publishers.
top_publisher_share = top_publisher_observations / all_valid_observationsAlways state the observation count, prompt-set version, answer systems, markets, and collection window. Publisher share describes only the analyzed sample. It is not market share, source quality, ranking strength, or proof of influence.
Verification after normalization
- Open the observed URL in an authorized, isolated context.
- Record the HTTP result, redirect chain, and final URL with a timestamp.
- Inspect the publisher-declared canonical without overwriting the observed value.
- Compare page content before merging keys created by a risky transformation.
- Classify the page's source role: definition, evidence, comparison, implementation, or validation.
- Review whether the page supports the associated answer claim.
A timeout, challenge page, authentication wall, or regional restriction is not proof that a URL is invalid. Record it as an unresolved verification state.
Implementation test set
Test hostname casing while preserving path casing; fragments; one and multiple analytics fields; retained content parameters; repeated query keys; parameter order; root and non-root trailing slashes; Unicode and percent-encoded paths; invalid and relative inputs; referral-rule behavior; and unchanged URLs. Label every fixture as synthetic.