CHECK YOUR "robots.txt" FILE ON NEOCITIES!!

I was looking through and noticed a file on my site titled "robots.txt". It comes with your site when you make a neocities site. I hadn't checked it out yet but I'm so glad I did now.

The robots.txt file tells bots what they are allowed to see on your site. It's defaulted to allow, so I disallowed everything. If you do NOT want AI bots crawling on your site I HIGHLY recommend you do this. I looked into it more and I saw that it does not always stop all bots but many will listen when they see this. If you know any extra info feel free to drop it in the comments. 

Here what mine looks like now (I added some extra stuff). You can even copy/paste and replace the section with this (careful not to delete something important)

User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-Agent: aiHitBot
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-Agent: AwarioBot
User-Agent: AwarioSmartBot
User-Agent: AwarioRssBot
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-Agent: Claude-User
User-Agent: Claude-SearchBot
User-agent: cohere-ai
User-Agent: cohere-training-data-crawler
User-Agent: Cotoyogi
User-Agent: DataForSeoBot
User-agent: Diffbot
User-agent: DuckAssistBot
User-agent: FacebookBot
User-Agent: Factset_spyderbot
User-Agent: FirecrawlAgent
User-agent: FriendlyCrawler
User-Agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GPTBot
User-agent: iaskspider/2.0
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: img2dataset
User-agent: ISSCyberRiskCrawler
User-agent: Kangaroo Bot
User-Agent: Meltwater
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: OAI-SearchBot
User-agent: omgili
User-agent: omgilibot
User-agent: PanguBot
User-Agent: peer39_crawler
User-agent: PerplexityBot
User-Agent: Perplexity-User
User-agent: PetalBot
User-agent: Scrapy
User-Agent: Seekr
User-Agent: SemrushBot-OCOB
User-Agent: Sentibot
User-agent: Sidetrade indexer bot
User-Agent: TikTokSpider
User-agent: Timpibot
User-Agent: TurnitinBot
User-agent: VelenPublicWebCrawler
User-agent: Webzio-Extended
User-agent: YouBot
Disallow: /<br style="max-width: 100%;">DisallowAITraining: /<pre style="max-width: 100%; white-space: pre-wrap; overflow-wrap: break-word;"># Block any non-specified AI crawlers (e.g., new
# or unknown bots) from using content for training
# AI models, while allowing the website to be
# indexed and accessed by bots.  These directives
# are still experimental and may not be supported
# by all AI crawlers.
User-Agent: *
DisallowAITraining: /
Content-Usage: ai=n
Allow: /