BuildWithSiddeshBot

What the bot is

BuildWithSiddeshBot belongs to the free Labs tools on buildwithsiddesh.com, run by Siddesh Bobade. Every request it makes carries the user agent shown above. It never presents itself as a browser, as Googlebot or as any other bot.

When it fetches

Only when a signed-in person runs one of the Labs tools, and only the pages that tool needs. It does not follow links from page to page, and it does not crawl on a schedule.

• SEO Audit Agent: the page the person enters, plus the top 5 pages that rank in Google for their keyword.
• Content Refresh Agent and Metadata Copilot: the pages the person enters.
• Schema Generator and Internal Linking Agent: the pages or sitemap the person enters, and a limited sample of the pages that sitemap lists.
• Growth Dashboard OS: the sitemap of a site the person can access in Google Search Console.

It also reads a few single files directly, without checking robots.txt, because they are named documents rather than crawling: a Google Sheet that a signed-in person names as input, and, once a day, a small fixed set of public news feeds about search for the Pulse page. It never fetches Google search result pages or Reddit; the Google results the tools show come from a search data provider.

robots.txt

Before it fetches a page, the bot reads that site’s robots.txt and obeys the group that names BuildWithSiddeshBot, or the * group when no group names it. It reads up to 500 KiB of the file and follows up to 5 redirects to it.

• No robots.txt (a 404 or another client error) means everything is allowed.
• When robots.txt cannot be reached (a server error, a 429 or a timeout), the whole site is treated as disallowed for that run.
• The most specific rule wins, and Allow wins a tie.

One exception: when someone runs the SEO Audit Agent, Content Refresh Agent or Metadata Copilot on a single URL they typed, and only the * group disallows it or robots.txt cannot be reached, the bot fetches that one page once and the report says so. It never does this when a group naming BuildWithSiddeshBot disallows the page, or when the site has opted out.

What it reads and keeps

The bot reads each page in memory to build the report for the person who ran the tool. It does not store copies of pages. A report holds facts measured from the page and short excerpts, such as its title, headings and word count. Some tools pass short extracts of the page to an AI model (Google Gemini, or Groq as a fallback) to write parts of the report.

A report is kept only in that person’s Labs run history, or behind a share link they choose to create. The privacy policy explains how long each is kept.

How to opt out

There are three ways:

• robots.txt: add a group that names the bot, for example

  User-agent: BuildWithSiddeshBot
  Disallow: /

• The form below: enter the site’s host name.
• The contact page: send a message naming the site.

An opt-out through the form or the contact page covers the host and every subdomain (a leading www. is ignored), and it applies even when someone types one of your addresses into a tool. The one exception is a person who has proved in Google Search Console that they manage the site: their own checks of it still run, so nobody else can switch off a site owner’s audits. Once a host is on the list, tool runs stop fetching it within 10 minutes. Most form requests are added at once. A request that would cover a whole shared hosting platform, a service this site relies on, or a country’s domain suffix waits for a manual review, and so do later requests when the form receives an unusual number in a day.

IP addresses

Requests come from Vercel’s shared infrastructure, so we cannot publish a list of IP addresses. Identify the bot by its user agent, and contact us if something looks wrong.

Contact

Questions about the bot, or a problem with how it fetched your site? Use the contact page.