Monitoring competitor websites ethically

The ethical method is also the useful one

Some competitor monitoring is done badly: aggressive scrapers that slow sites down, fake accounts, click fraud on ads. None of it is necessary, and all of it carries risk, from breaching a site's terms to costing a rival money in a way that could come back to you.

Public pages are what customers see, so they are what you should be comparing. A slow, careful, dated record beats a huge, messy dump of data you cannot interpret.

Respect robots.txt

Most sites publish a file at /robots.txt. Google describes it as telling search engine crawlers which URLs the crawler can access on a site, and notes it is used mainly to avoid overloading a site; it is not a mechanism for keeping a page out of Google (Google Search Central).

If you use any automated tool to check rivals' pages, treat robots.txt as the site owner's stated wishes and follow it. If a path is disallowed, do not fetch it with a tool. Reading a page in your own browser as an ordinary visitor is different from running a crawler against it, but the spirit is the same: take only what the site openly offers to the public.

Polite rates and public pages only

A useful monitoring routine looks at a small, fixed set of pages: the homepage, a few key category pages, a handful of best-selling product pages, delivery and returns, and the about or pricing page for service firms. That is a few dozen requests per rival per check, spread out, not thousands in a burst.

• Name your tool in its user agent if you use one. Do not disguise it as a browser.
• Space requests out and check on a sensible schedule, such as weekly.
• Stop if you see errors or rate-limit responses.
• Stay on public pages. Do not create accounts, log in, use trade logins you were given for another purpose, or try to reach customer-only areas.
• Do not place real orders or make bookings just to see the next step. See building a seed inbox for the narrow case where adding to a basket is reasonable.

Never click competitors' ads

If you want to know what a rival is advertising, do not search for their brand and click the paid result. Each click costs them money, and deliberately generating clicks you have no intention of acting on is exactly the behaviour ad platforms try to filter out.

Use the transparency tools the platforms provide instead. The Meta Ad Library is a public, searchable library of ads running on Meta's apps; you can search by advertiser and filter by active or inactive status, and entries show when an ad started running. The Google Ads Transparency Center is described by Google as a searchable repository of advertisers and the ads they have served; you can search an advertiser, see ads run during a period and filter by date and targeted location (Google support). The competitor ad watch loop, which will make this a routine, is in build and not on sale.

Sitemaps and structured data

Two public files do much of the work for you.

Sitemaps. Most shops publish an XML sitemap, often listed in robots.txt. It lists the URLs the site wants search engines to find. Comparing a rival's sitemap between checks shows new product lines, new landing pages, new location pages for a service firm and retired ranges, without crawling the whole site.

Structured data. Product pages often carry JSON-LD with name, price, currency, availability and review ratings. Reading it gives you clean, comparable fields rather than scraping prices out of page designs. The JSON-LD tool shows you what a page publishes.

Keep a dated change log

The value of monitoring comes from change over time. A single snapshot tells you what a site looks like; a log tells you what the rival is trying.

For each check, record the date, the page, what changed and a screenshot or saved copy. Useful things to log include:

• Homepage headline, hero offer and main call to action
• Navigation changes and new categories
• Delivery charges, free delivery threshold and returns terms
• Prices on your benchmark products (see benchmarking your prices)
• New trust signals, guarantees, awards or payment options
• Promotions running and how they are presented

A worked example: a landscaping firm logs three local rivals each month. Over a quarter the log shows Rival A added a page for each nearby town, and Rival B started showing a fixed design fee on its services page. Neither fact proves anything about their enquiries, but both are clear prompts to look at the firm's own location pages and fee transparency.

Write what you saw, not what you think it means. “Rival A moved free delivery from the footer to the top banner” is an observation. “Rival A's conversion went up” is a guess you cannot see from outside. Keep hypotheses in a separate column and treat them as things to test, not facts.

The competitor website monitor, which will keep this log on a schedule, is in build and not on sale. Until it ships, use a shared spreadsheet and a recurring calendar slot.