Home» Forum» Site Feedback» Why is the site specifically allowing AI scraping?

Why is the site specifically allowing AI scraping?

Back to Site Feedback

From https://friendrewind.com/robots.txt:

# ── AI Crawlers: Welcome ──
# Allow AI answer engines to index and cite FriendRewind content
User-agent: GPTBot
Allow: /<br>...

Yet archive.org is blocked? 

# ── Internet Archive / Wayback Machine only (not AI crawlers above) ──
User-agent: ia_archiver
Disallow: /

<br>

I can understand blocking both for privacy reasons or neither for visibility, but this seems backwards, this website is a place where people put themselves into public view and now are basically welcoming themselves as training data? I really like this site but this makes me uneasy...

You must be logged in to reply.