crawler transparency
About our crawler
When someone runs a Surfaced audit, our server fetches the entered website once — the homepage and its robots.txt — to generate an AI-visibility report. It identifies itself honestly with this User-Agent:
SurfacedBot/0.1 (+https://surfaced.co.nz/about/bot; AI-visibility checker)
What it does
- Fetches the page a person asked us to audit, plus that site's robots.txt.
- Reads the raw HTML only — it does not execute JavaScript, set cookies, or log in.
- Caps downloads (3 MB page, 200 KB robots.txt) and times out after a few seconds.
- Results for a domain are cached for 24 hours, so repeat checks don't re-fetch.
What it doesn't do
- No site-wide crawling — it fetches the specific page requested, not your whole site.
- No scraping for resale. Reports are generated for the person who requested them.
- No requests to private networks — the fetcher refuses internal addresses by design.
Opting out
Email hello@surfaced.co.nzwith your domain and we'll exclude it from audits. Blocking "SurfacedBot" in your robots.txt is respected for our batch/aggregate work as well.