Site crawler
The crawler that feeds every technical SEO surface — crawl profiles and schedules, adaptive politeness, the frontier that decides what gets crawled first, internal PageRank, Core Web Vitals trends, hreflang checks, and the remediation workbench.