Adwright

[ Crawler ]

AdwrightBot, our crawler

When someone pastes a link into Adwright to make ads for their product, AdwrightBot reads that page. While it researches the market, it also reads the public pages of the competitors the research names. Here’s how to recognise it, what it keeps, and how to keep it away.

How to recognise it

Every request it makes carries this user agent:

Mozilla/5.0 (compatible; AdwrightBot/1.0; +https://adwright.co/crawler)

It never sends cookies, passwords or any other credential, and never fills in or submits a form.

What it reads

  • The page someone pastes: a website, a product page or an app’s page. On the same site, up to 3 more pages that say more about the product, like pricing, features or about.
  • That page’s stylesheets, for the brand’s colours and type.
  • The pictures on those pages that could go in an ad: product shots, screenshots and the logo.
  • For the research, the public pages of up to 6 competitors that the person names or the research suggests.
  • Never anything behind a sign-in. With no cookies or passwords, it only sees what any visitor sees.

App Store listings are read through Apple’s public lookup service, not from your site.

What it keeps

What it finds belongs to the project of the person who pasted the link: only they see it, unless they share it. Nothing from your site is published by us.

Kept

  • Facts from the page, for the person who pasted the link: the product’s name, headings and copy, prices and offers, its rating and the reviews quoted on the page.
  • The brand’s colour values, the names of its type faces, and its logo.
  • Pictures that can go in an ad (product shots, screenshots, the logo), copied into that person’s project so their ads can use them.
  • For a competitor: what its site says about itself (its claims, offers and prices), summed up in the project’s research.

Never kept

  • Anything behind a sign-in.
  • A page’s HTML, CSS or scripts: they’re read in memory, then dropped.
  • Pictures from a competitor’s site.
  • Anything from a site that has opted out.

The person who made the project can delete it, and the pictures in it, at any time.

Opting out

  • Once a person here has checked that a request comes from the site’s owner, AdwrightBot stops reading the domain and every subdomain: for links anyone pastes, and for competitor research.
  • A request sent from an address at the site’s own domain stops competitor research reading the site as soon as it arrives.
  • Someone who pastes a link to a site that has opted out is told we can’t read it.
  • A domain many unrelated sites share (like github.io or co.uk) can’t be opted out as a whole. Opt out your own site’s domain, or email us.

How it goes about it

  • It reads a site only when someone asks for a project, never on a schedule of its own.
  • Each request is capped in size and time, and it follows at most 5 redirects.
  • It only asks for public addresses, never one on a private network.

Questions

Email support@adwright.co. The form on this page is the quickest way to opt out.