# ============================================================ # Knowasiak — robots.txt # https://www.knowasiak.com # ============================================================ # --- Content Signals (https://contentsignals.org) -------------- # Machine-readable declaration of content-use preferences. # ai-train=no : do not use pages for AI training datasets # search=yes : search engine indexing is welcome # ai-input=yes : AI assistants may use content to answer # user queries and cite Knowasiak as a source Content-Signal: ai-train=no, search=yes, ai-input=yes # --- Default rules for all well-behaved crawlers --------------- User-agent: * # Public content (explicitly allowed) Allow: / Allow: /themes/default/statics/ Allow: /themes/default/css/ Allow: /themes/default/js/ Allow: /upload/images/ Allow: /upload/photos/ Allow: /upload/videos/ Allow: /upload/avatars/ Allow: /upload/covers/ Allow: /upload/default/ Allow: /sitemap/ # Backend / internal (block crawling) Disallow: /apps/ Disallow: /core/ Disallow: /install/ Disallow: /site_backups/ Disallow: /themes/default/apps/ Disallow: /api.php Disallow: /api_docs/ Disallow: /core/configs/ Disallow: /core/libs/ Disallow: /core/logs/ Disallow: /core/loc_db/ Disallow: /upload/documents/ Disallow: /upload/receipts/ # Pure JSON endpoints — no HTML/meta-robots mechanism applies to these, # so robots.txt is the only way to keep crawlers off them. Disallow: /native_api/ Disallow: /mobile_api/ Disallow: /sitemap/temps/ # Private user actions (prevent indexing of transient/auth pages) Disallow: /*?ref= Disallow: /*&ref= Disallow: /*?token= Disallow: /*&token= Disallow: /*?access_token= # --- AI / LLM crawlers ----------------------------------------- # Asymmetric policy: training crawlers are blocked from HTML pages # but ALLOWED on the public Markdown mirrors (every UGC URL has a # /index.md sibling) plus /llms.txt and /markdown-mirrors.txt. # This gives AI systems the structured, citation-friendly version # without serving them the full HTML payload. # # Markdown mirror URL patterns (all 200 OK, text/plain UTF-8): # //index.md # ///index.md (posts|media|likes|about|shop|replies) # /thread//index.md # /startups//index.md # /marketplace//index.md # /ads//index.md # /entity/.json (normalized JSON, same data as the profile mirror) User-agent: GPTBot Disallow: / Allow: /*/index.md Allow: /thread/*/index.md Allow: /startups/*/index.md Allow: /marketplace/*/index.md Allow: /ads/*/index.md Allow: /llms.txt Allow: /markdown-mirrors.txt Allow: /entity/*.json Allow: /*.md$ Allow: /recent.md Allow: /*/feed.md Allow: /sitemap-mirrors.xml # Full-access crawlers below get their own copy of the backend/internal # exclusions from the User-agent: * block above — a bot with its own # dedicated User-agent section does NOT inherit User-agent: * rules per the # robots.txt spec, so "Allow: /" alone would leave /native_api/, /apps/, # /core/, /api.php etc. wide open to exactly these crawlers. (This is what # let Meta-ExternalAgent hit /native_api/* unrestricted before this fix.) # User-initiated browsing: allow when a user asks ChatGPT to read a page User-agent: ChatGPT-User Disallow: /apps/ Disallow: /core/ Disallow: /install/ Disallow: /site_backups/ Disallow: /themes/default/apps/ Disallow: /api.php Disallow: /api_docs/ Disallow: /native_api/ Disallow: /mobile_api/ Disallow: /upload/documents/ Disallow: /upload/receipts/ Allow: / User-agent: OAI-SearchBot Disallow: /apps/ Disallow: /core/ Disallow: /install/ Disallow: /site_backups/ Disallow: /themes/default/apps/ Disallow: /api.php Disallow: /api_docs/ Disallow: /native_api/ Disallow: /mobile_api/ Disallow: /upload/documents/ Disallow: /upload/receipts/ Allow: / User-agent: Google-Extended Disallow: / Allow: /*/index.md Allow: /thread/*/index.md Allow: /startups/*/index.md Allow: /marketplace/*/index.md Allow: /ads/*/index.md Allow: /llms.txt Allow: /markdown-mirrors.txt Allow: /entity/*.json Allow: /*.md$ Allow: /recent.md Allow: /*/feed.md Allow: /sitemap-mirrors.xml User-agent: CCBot Disallow: / Allow: /*/index.md Allow: /thread/*/index.md Allow: /startups/*/index.md Allow: /marketplace/*/index.md Allow: /ads/*/index.md Allow: /llms.txt Allow: /markdown-mirrors.txt Allow: /entity/*.json Allow: /*.md$ Allow: /recent.md Allow: /*/feed.md Allow: /sitemap-mirrors.xml User-agent: anthropic-ai Disallow: / Allow: /*/index.md Allow: /thread/*/index.md Allow: /startups/*/index.md Allow: /marketplace/*/index.md Allow: /ads/*/index.md Allow: /llms.txt Allow: /markdown-mirrors.txt Allow: /entity/*.json Allow: /*.md$ Allow: /recent.md Allow: /*/feed.md Allow: /sitemap-mirrors.xml User-agent: ClaudeBot Disallow: / Allow: /*/index.md Allow: /thread/*/index.md Allow: /startups/*/index.md Allow: /marketplace/*/index.md Allow: /ads/*/index.md Allow: /llms.txt Allow: /markdown-mirrors.txt Allow: /entity/*.json Allow: /*.md$ Allow: /recent.md Allow: /*/feed.md Allow: /sitemap-mirrors.xml User-agent: Claude-Web Disallow: /apps/ Disallow: /core/ Disallow: /install/ Disallow: /site_backups/ Disallow: /themes/default/apps/ Disallow: /api.php Disallow: /api_docs/ Disallow: /native_api/ Disallow: /mobile_api/ Disallow: /upload/documents/ Disallow: /upload/receipts/ Allow: / User-agent: cohere-ai Disallow: / Allow: /*/index.md Allow: /thread/*/index.md Allow: /startups/*/index.md Allow: /marketplace/*/index.md Allow: /ads/*/index.md Allow: /llms.txt Allow: /markdown-mirrors.txt Allow: /entity/*.json Allow: /*.md$ Allow: /recent.md Allow: /*/feed.md Allow: /sitemap-mirrors.xml User-agent: Bytespider Disallow: / # AI search engines (allow full crawl for citation/discovery) User-agent: PerplexityBot Disallow: /apps/ Disallow: /core/ Disallow: /install/ Disallow: /site_backups/ Disallow: /themes/default/apps/ Disallow: /api.php Disallow: /api_docs/ Disallow: /native_api/ Disallow: /mobile_api/ Disallow: /upload/documents/ Disallow: /upload/receipts/ Allow: / User-agent: Perplexity-User Disallow: /apps/ Disallow: /core/ Disallow: /install/ Disallow: /site_backups/ Disallow: /themes/default/apps/ Disallow: /api.php Disallow: /api_docs/ Disallow: /native_api/ Disallow: /mobile_api/ Disallow: /upload/documents/ Disallow: /upload/receipts/ Allow: / User-agent: FacebookBot Disallow: /apps/ Disallow: /core/ Disallow: /install/ Disallow: /site_backups/ Disallow: /themes/default/apps/ Disallow: /api.php Disallow: /api_docs/ Disallow: /native_api/ Disallow: /mobile_api/ Disallow: /upload/documents/ Disallow: /upload/receipts/ Allow: / User-agent: Meta-ExternalAgent Disallow: /apps/ Disallow: /core/ Disallow: /install/ Disallow: /site_backups/ Disallow: /themes/default/apps/ Disallow: /api.php Disallow: /api_docs/ Disallow: /native_api/ Disallow: /mobile_api/ Disallow: /upload/documents/ Disallow: /upload/receipts/ Allow: / User-agent: Applebot Disallow: /apps/ Disallow: /core/ Disallow: /install/ Disallow: /site_backups/ Disallow: /themes/default/apps/ Disallow: /api.php Disallow: /api_docs/ Disallow: /native_api/ Disallow: /mobile_api/ Disallow: /upload/documents/ Disallow: /upload/receipts/ Allow: / User-agent: Amazonbot Disallow: /apps/ Disallow: /core/ Disallow: /install/ Disallow: /site_backups/ Disallow: /themes/default/apps/ Disallow: /api.php Disallow: /api_docs/ Disallow: /native_api/ Disallow: /mobile_api/ Disallow: /upload/documents/ Disallow: /upload/receipts/ Allow: / # --- Aggressive / unwanted bots ------------------------------- User-agent: AhrefsBot Crawl-delay: 10 User-agent: SemrushBot Crawl-delay: 10 User-agent: DotBot Disallow: / User-agent: MJ12bot Disallow: / # --- Sitemaps -------------------------------------------------- Sitemap: https://www.knowasiak.com/sitemap/sitemap-index.xml Sitemap: https://www.knowasiak.com/sitemap-mirrors.xml