Crawler documentation
Adviora Bot
Adviora Bot is the web crawler Adviora uses to discover, analyse and collect public web data for SEO, GEO and AEO intelligence. This page documents what it does, how it behaves, and how to control it.
Overview
What is AdvioraBot?
A web robot — also called a spider or crawler — that automatically discovers, analyses and collects web data for AI-powered search optimisation and competitive intelligence.
AdvioraBot is Adviora's own crawler. It reads publicly available pages to build the analyses behind our SEO, generative-engine optimisation and answer-engine optimisation tooling. It identifies itself on every request with the user-agent below, which is the string to match in your server logs or your robots.txt.
User-agent
AdvioraBot/1.0 (+https://adviora.ai/bot)
In robots.txt, match the product token on its own — User-agent: AdvioraBot — not the full string above.
What it feeds
Data collected by Adviora Bot
Crawled data is used exclusively to power the analyses below.
Technical SEO
Technical SEO audit
Analyses on-page SEO, technical issues and site structure.
Trust
EEAT analysis
Evaluates experience, expertise, authoritativeness and trustworthiness signals.
GEO
GEO intelligence
Tracks content optimisation for generative search engines.
AEO
AEO optimisation
Analyses answer-engine optimisation and structured-data readiness.
Competitive
Competitor analysis
Gathers competitive intelligence on strategy, content and positioning.
Off-page
Backlink intelligence
Discovers and analyses link profiles and domain-authority metrics.
Content
Content analysis
Evaluates content quality, topical authority and coverage gaps.
Market
Customer discovery
Maps market segments and prospect-readiness scores.
Behaviour
How Adviora Bot crawls your site
Three stages, from a seed list to a full map of your site.
- 01
Crawl initiation
The crawl starts from a seed list of URLs — typically your sitemap or your homepage. AdvioraBot visits each one and extracts the hyperlinks it finds.
- 02
Progressive discovery
That link list, the crawl frontier, is worked through according to Adviora's crawl policies to map the site and surface what has changed.
- 03
Analysis at scale
Tiered crawling with request throttling keeps the crawl off your server's critical path. robots.txt, crawl delays and response codes are all respected.
What progressive discovery surfaces
- Content changes and updates
- New pages and landing pages
- Broken or dead links
- URL structure and hierarchy
- Internal linking patterns
- Site architecture and organisation
Control
How to block Adviora Bot
Adviora respects website owners' crawl preferences. If you do not want AdvioraBot to access part or all of your website, configure your robots.txt file using the rules below.
The file must sit at the root of the origin you want to control — for example https://yourdomain.com/robots.txt. Anywhere else and it has no effect on AdvioraBot at all.
User-agent: AdvioraBot
Disallow: /User-agent: AdvioraBot
Disallow: /admin/
Disallow: /internal/Supported robots.txt extensions
Crawl-delay. Honoured up to 10 seconds between requests; higher values are capped at 10. With no crawl-delay set, AdvioraBot adjusts its own request rate based on how your server is responding.
Wildcards. Standard * patterns are supported in rules.
Three things that catch people out
Subdomains need their own file. robots.txt applies to one origin. A file on the parent domain does not cover shop.yourdomain.com — without one of its own, that subdomain is treated as fully crawlable.
The status code matters. robots.txt must return HTTP 200. A 4xx is read as "no robots.txt exists", so no restrictions apply — this is the single most common cause of a block that does not work. A 5xx has the opposite effect and stops AdvioraBot crawling the site at all.
Changes are not instant. Allow up to one hour, or roughly 100 requests, for an edit to be picked up.
Do not block AdvioraBot by IP address. The crawler does not run from a fixed or contiguous IP range, so filtering by address will not reliably stop it — and will quietly stop working. Use robots.txt.
FAQ
Frequently asked questions
AdvioraBot follows links it finds across the web, not just on your site. If another domain links to a URL on yours — including one you removed years ago — the crawler may still try to reach it. After a URL returns 404 consistently across several attempts, AdvioraBot stops requesting it.
AdvioraBot reads forms to understand your site's structure and functionality. Where a login or a submission uses the HTTP GET method, those URLs appear in your link structure like any other and may be requested.
Use POST for login forms and any sensitive submission. GET puts the submitted values in the URL, where they reach your access logs, your analytics and every crawler that follows the link — this is worth fixing regardless of AdvioraBot.
Migrate the robots.txt file along with everything else — a move from HTTP to HTTPS, or to a new domain, is a new origin as far as the crawler is concerned. AdvioraBot caches crawl state per domain, so old rules can persist. Put robots.txt in place on the new origin before the migration, or contact us to reset the crawl state for your domain.
Give AdvioraBot its own group with an empty Disallow, which grants full access, and disallow everything for the wildcard group:
User-agent: *
Disallow: /
User-agent: AdvioraBot
Disallow:Yes — AdvioraBot honours the standard robots meta directives, including noindex, nofollow and noarchive. Note that a page still has to be fetched for those tags to be read, so a request in your logs is expected even for a page marked noindex. Use robots.txt if you need the request itself to stop.
If you use the Adviora platform for your own domain, the audit reports, competitor analysis and recommendations are all in your dashboard. Otherwise, contact us and we can talk through what is available.
Reference
More about web crawlers
For the standard itself, rather than our implementation of it:
- robotstxt.org — the robots.txt specification and its conventions.
- Google Search Central — how Google's crawlers interpret robots.txt, which most others follow.
Support
Getting support
Questions about AdvioraBot, concerns about crawl frequency, or a robots.txt rule that isn't being honoured — we'd rather hear about it.
Email bot@adviora.ai. We reply to all enquiries within 24 business hours.
If AdvioraBot is causing server strain or ignoring your rules, send:
- The domain and the exact URLs affected
- Server log lines showing the requests, with timestamps and timezone
- Your current robots.txt, and when it last changed
That is enough to reproduce the behaviour and stop it, rather than starting with a round of questions.
Curious what AdvioraBot sees on your site?
Run the same analysis on your own domain — no sign-up needed to start.