GigListerBot
GigListerBot reads public gig listings for GigLister, a small service that helps local editors keep a what's-on list for their area. It is in a pilot in Redhill, Reigate and Horley.
What it fetches
- Gig listing pages on venue, promoter and band websites (their schema.org Event data where a page has it), plus public calendar and RSS feeds.
- Official APIs, used within their terms.
- A page is fetched about once a night, with a conditional request, so an unchanged page costs you very little. It never logs in or fills in forms.
How it behaves
- It checks
robots.txtbefore every fetch (kept for a day) and does what it says, includingCrawl-delay(up to 30 seconds between requests). - It identifies itself as
GigListerBot/0.1 (+https://giglister.uk/bot). - It never reads Facebook, Instagram or Linktree, and nothing robots.txt forbids.
- Every gig it finds links back to where it came from.
To stop it
Add this to your site's robots.txt, and it will stop within a day:
User-agent: GigListerBot Disallow: /
Or write to hello@giglister.uk and we'll stop at once, and remove anything it collected from your site if you'd like.