FossickBot

FossickBot is an automated web crawler operated by DATAFREAK LTD, the company behind StackScope (stackscope.dev), a site that records the technologies new websites are built with.

If you found FossickBot in your logs: FossickBot visits newly launched websites to check whether a real site has gone live, so StackScope can add it to its catalogue of new launches.

What it does

• Requests /robots.txt first, and obeys it (including Crawl-delay).
• Loads the home page once and renders it the way a browser does, running the page's own JavaScript.
• Does not log in, submit forms, or crawl any other page on the site.
• Identifies as FossickBot and comes from crawler.fossick.bot.
• From that single rendered page it records the title, description, a short text excerpt, and the technologies the site is built with, enough to tell a real launch from a parked or placeholder page and to add it to the StackScope catalogue.

That is all FossickBot does. It visits once and does not come back to crawl the rest of the site.

Opt out

Add this to your robots.txt (honoured on the next visit):

User-agent: FossickBot
Disallow: /

Or email abuse@fossick.bot with the domain and it will be excluded by hand.

Operated by DATAFREAK LTD, registered in England and Wales (company no. 17328826), the company behind StackScope. Contact: abuse@fossick.bot