Site crawler

Audience: Merchant·Plan: free·Read: 10 min·Updated: 2026-08-03

In 30 seconds

• The crawler is the data source under Technical health, Indexability, Internal links, the Risk register and Broken links. If it has not run — or ran against the wrong hostname — those surfaces have nothing correct to show.
• Crawl profiles let you keep several named crawls with their own targets, page budgets and cadences ("weekly full site", "daily key pages").
• The crawler paces itself against your host — it speeds up while responses are healthy and backs off hard on 429s, instead of hammering through a rate limit and recording the challenge pages as your site.
• When the page budget is smaller than your catalogue, a weighted frontier decides what to crawl first, so you audit your most valuable pages rather than whatever the link graph reached.
• Every issue becomes a work item — assignable, snoozable, resolvable, exportable to Jira, Linear or Asana.

How to use it

1. Point it at your live domain. Not the .myshopify.com mirror. This is the single most consequential setting on the page.
2. Set a page budget you are comfortable with, and a crawl strategy.
3. Save it as a profile with a schedule so comparisons have matched pairs to work from.
4. Work the issues from the remediation workbench, not from a static report.
5. Suppress what you have decided not to fix with an ignore rule, so it stops crowding out real findings.

⏱ ~5 min setup · 💳 Free (scheduling Starter+) · 🎯 One crawl feeds six other surfaces

Crawl profiles

A profile is a saved, named crawl configuration. Most stores end up with two or three.

Setting	What it controls
Name	How the profile appears in history and alerts
Target URL	Where the crawl starts. Validated as a public http(s) address
Page budget	The maximum number of pages this run will fetch
Strategy	balanced · sitemap_first · value · breadth — how the frontier prioritises
Schedule	daily · weekly · monthly, or none
Notify email	Where regression alerts for this profile go
Regression threshold	The score above which this profile raises an alert
SLA	Maximum run minutes before the run is considered stalled

Profiles exist because one schedule per store was not enough: "weekly full site" and "daily critical pages" are different jobs with different budgets, and a single shared setting forced you to choose. Running a profile enqueues an ordinary crawl — the machinery underneath is the same.

Crawl the domain you actually sell on

If your store has a custom domain, the .myshopify.com address is a mirror that canonicalises almost every page to the real domain. A crawl of the mirror produces a report that is internally consistent and completely useless — and it poisons the comparison against your real crawls, the internal-link suggestions, and the indexability split.

Target URLs are validated and SSRF-guarded, but nothing can tell us which of your two working hostnames you meant. Set it once, in a profile, and stop typing it by hand.

What decides the crawl order

When your page budget is smaller than your catalogue — which it usually is — the order matters more than the number. The frontier scores every discovered URL and crawls the lowest score first, blending:

• Whether the URL appears in your sitemap, and its declared priority there
• How recently lastmod says it changed
• URL depth — how many clicks from the home page
• Path-type heuristics — a product page is not a tag archive

The result is that a 500-page budget on a 5,000-page store audits the 500 pages worth auditing, rather than 500 paginated collection URLs.

Politeness: how hard the crawler pushes

The crawler adapts its rate to your host rather than using a fixed delay.

This matters more than it sounds. A storefront behind a CDN or WAF answers a burst of requests with 429 challenge pages. A fixed-rate crawler keeps going, records those challenge pages as your site's HTML, and hands you an audit full of fabricated errors — while losing the coverage it was meant to gather. The adaptive loop backs off instead, and a sustained failure trips a breaker that pauses rather than grinding.

What a crawl produces

Signal	What you get
Page inventory	Status code, title, meta description, canonical, robots directives, depth, word count, headings
Indexability classification	The eight-bucket split used by Technical health
Issues	Typed, severity-graded findings per page
Internal link graph	Adjacency built from real links found on real pages
Internal PageRank	An iterative simulation over that graph, showing where link equity actually pools
Core Web Vitals	Persisted per URL and strategy, so vitals trend instead of being fetched and thrown away
Hreflang consistency	Missing x-default, non-reciprocal pairs, and other international-SEO failures
JS rendering gaps	Pages whose content is injected by JavaScript, and therefore invisible to an HTML-only crawler
Accessibility	The WCAG subset detectable from raw HTML — missing lang, generic link text, unlabelled inputs, no main landmark
AI report	A structured, persisted health summary with ranked actions, each linked to its source issue

Two of these are worth calling out.

Internal PageRank answers a question a page-by-page audit cannot: your highest-margin collection may be technically perfect and still starved of internal links. The simulation shows equity flow, and the Internal links tab is where you act on it.

Core Web Vitals are persisted, not sampled live. The earlier behaviour — fetch field data, render it, discard it — meant "our LCP got worse this week" was unanswerable and no regression could ever be alerted on. Each capture is now stored per URL and strategy, and a regression against the previous capture raises an alert.

The AI report is generated once and stored, with each action carrying its source issue's identity. It is re-readable and exportable without spending tokens again on every click.

The remediation workbench

Crawl issues are not a static list. Every issue becomes a work item on the shared SEO work-item spine, which means it has a lifecycle:

• Assign it to a person
• Snooze it until a date
• Resolve it, or dismiss it with a reason
• Export it to Jira, Linear, Asana or a generic webhook — the link is recorded, so the workbench shows the live ticket and never double-exports

Because issues share the work-item spine with the rest of the SEO suite, an assignment made here shows up in the Risk register as owned, not orphaned.

Suppressing issues you have decided about

An ignore rule suppresses matching issues from the issues view, the counts, the summaries, the exports and the AI prompt. Rules match on issue type, URL, or both, and can be scoped to one crawl or to the whole store.

Issues are never deleted. Removing the rule brings them back. That is deliberate: a suppression you cannot reverse is indistinguishable from data loss, and the counts elsewhere in the suite depend on the underlying rows continuing to exist.

Scheduling and comparison

Scheduled crawls are swept hourly. Both the per-store schedule and the newer named profiles are checked on the same sweep, and a run that is overdue fires on the next tick.

If your schedule has never fired

A schedule that is active with an empty "last run" and a due date in the past means the sweep was not reaching it. This was the case on stores running builds from before August 2026 — every crawl on those stores was manual, and because manual crawls were usually typed against whichever hostname was in the box, the comparison pairs were mismatched too. Once a schedule fires on its configured target, host consistency fixes itself.

Comparison between two crawls is what produces regression incidents; see Technical health for how they are scored, levelled and correlated against what changed in your store.

Plan tiers

Capability	Free	Starter	Pro	Agency	Enterprise
Manual crawl, page inventory, issues	✓	✓	✓	✓	✓
Indexability classification	✓	✓	✓	✓	✓
Issue suppression rules	✓	✓	✓	✓	✓
Internal link graph + PageRank	✓	✓	✓	✓	✓
Named crawl profiles	—	✓	✓	✓	✓
Scheduled crawls + regression alerts	—	✓	✓	✓	✓
Core Web Vitals trend + regression alert	—	✓	✓	✓	✓
Hreflang consistency report	—	✓	✓	✓	✓
Remediation workbench + issue-tracker export	—	✓	✓	✓	✓
AI crawl report	—	—	✓	✓	✓

The AI report makes model calls per run and sits with the other AI-backed SEO tools at Pro.

Best practices

• One profile per intent. "Full site, weekly" and "top 200 pages, daily" are different questions. Do not compromise into one crawl that answers neither.
• Keep the target stable. A comparison between two crawls of different hosts is not a comparison. Set it in a profile and leave it.
• Raise the budget before raising the frequency. A partial crawl repeated daily gives you noisy diffs; a complete crawl weekly gives you real ones.
• Suppress deliberately, not defensively. An ignore rule is a decision you are recording. Bulk-suppressing to make a number go down hides the finding from the risk register too.
• Watch the politeness signals. If a crawl consistently backs off, your host is rate-limiting us — lower the concurrency rather than assuming the site is broken.

See also

• Technical health — the indexability report and regression incidents this feeds
• Site audit — page-level scoring and the broken-link worklist
• Internal links — acting on the link graph the crawler builds
• Risk register — crawl issues rolled into one prioritised backlog
• Sitemap — what you publish for Google to crawl