New rival pages → content gaps

Why it matters

New pages are one of the clearest public signals of where a rival is investing. A cluster of new guides on one topic, a new category, or a fresh set of location pages tells you what they expect to sell next. Seeing it early lets you decide whether to answer, ignore it, or go somewhere they are not.

The competitor website monitor loop is in build and not on sale. When it ships it will record new and removed pages on the sites you choose. Until then, a rival's sitemap and a careful look at their navigation get you most of the way, provided you read only what they publish openly.

What you need

• - The home page address of up to three rivals
• - Each rival's public sitemap (often linked from their robots.txt file) and their main navigation
• - A list of your own pages, from your sitemap or your platform's page list
• - A previous snapshot of the rival's sitemap, if you have one; the first run sets the baseline

Setup

1. Open each rival's robots.txt file and note the sitemap address it lists. Respect any areas it asks crawlers to stay out of.
2. Copy the page addresses from the sitemap into a sheet with the date. If you kept an earlier copy, mark which addresses are new.
3. Open the new pages and note the topic, the type (guide, category, product, location, comparison) and who it seems written for.
4. List your own pages in the same sheet so the model can see where you already have coverage.
5. Paste both lists into the prompt below and review the gaps it returns against what you can realistically write well.

The prompt

Paste this into any capable model, followed by the evidence you gathered in the setup.

You are helping a UK business find content gaps from its rivals' new pages.
Below are: a list of NEW rival pages (address, page title, type, a short note on the topic, date found), and a list of OUR existing pages (address and title).
Do this:
1. Group the new rival pages by topic and by the intent behind them (learning, comparing, buying, finding a local provider). Quote the page title for each.
2. For each group, say which rivals published in it and how many pages, using only the list given.
3. Compare with our pages. For each group, say whether we have no coverage, thin coverage or good coverage, and quote the titles of our pages that relate.
4. Rank the gaps. For each, give a one-line brief for a page we could write, and rate effort (low/medium/high) and likely impact (low/medium/high), with a reason.
5. Flag any group where you think the rival is preparing a new product line or service area. Mark this as a hypothesis.
Do not recommend copying a rival's page; recommend what a better, different page would cover.
Rules:
- Quote the evidence for every point, word for word, and say where it came from (which rival, which page or ad, which date I captured it).
- Label each point OBSERVATION (you can see it in the evidence) or HYPOTHESIS (your reading of why). Never present a hypothesis as a proven cause.
- Never invent a figure, a date, a price or a result. If the evidence does not show it, write "not in the evidence".
- Use British English and plain words.

Lists:
[paste the lists here]

Doing it by hand today

Save a copy of each rival's sitemap once a month and compare it with the previous copy. A spreadsheet comparison of two columns of addresses shows the new ones quickly.

Read only what is public. robots.txt tells crawlers which addresses they may access, and it exists mainly to avoid overloading a site; checking a sitemap by hand is light, but if you script it, keep requests slow and follow the file.

For service firms, new location or service pages are the signal to watch. A solicitor adding a new practice area, or a landscaper adding a new town, is telling you where they plan to compete. The free website audit for solicitors shows how your own pages read against that standard.