# India Tax Planner - robots.txt
# https://smarttaxcalc.in/robots.txt

# Default policy: allow legitimate search engines
User-agent: *
Allow: /
Disallow: /_framework/
Disallow: /service-worker.js
Disallow: /service-worker-assets.js

# Google AdSense crawler. Mediapartners-Google is the bot that crawls pages
# to decide ad-targeting; explicitly allowing it ensures AdSense can verify
# the site's content matches the policy review. Without this directive,
# AdSense approval is often delayed by a few days even when the rest of the
# site is fine. The pattern below ALSO unblocks AdSense's mobile-targeted
# verifier (Mediapartners-Google-Mobile is the same crawler family).
User-agent: Mediapartners-Google
Allow: /

# AdsBot-Google is the Google Ads landing-page quality checker. We're not
# running Google Ads (we're AdSense, not Ads buyer), but having this allowed
# helps Google's quality-rater path when they cross-reference ad pages.
User-agent: AdsBot-Google
Allow: /

User-agent: AdsBot-Google-Mobile
Allow: /

# ===========================================================================
# CLOUDFLARE EDGE POLICY — read before changing any AI bot line below.
#
# Cloudflare decides before this file is consulted. On 2026-09-15 the single
# "Block AI bots" switch was replaced by three category controls at
# Dashboard -> Security -> Settings -> "Configure AI bot policies".
#
# Zone config verified 2026-09-20 (do not change without re-reading this):
#   Search              : Allow (recommended)
#   Agent               : Allow (recommended)
#   Training            : Disallow
#   Bot Preference Sync : OFF
#
# Cloudflare's category for each bot named in this file:
#   AI Search    -> Search   : OAI-SearchBot, PerplexityBot
#   AI Assistant -> Agent    : ChatGPT-User, Perplexity-User
#   AI Crawler   -> Training : GPTBot, ClaudeBot, CCBot, Bytespider
#
# The consequence is the important part: every bot that actually delivers
# REFERRAL traffic sits in Search or Agent, and both are Allow. Everything the
# edge blocks is in Training, and Training holds only crawlers that feed model
# training — none of which send traffic back. So Training: Disallow costs
# nothing while still blocking CCBot and Bytespider, which ignore robots.txt
# anyway. Do NOT "fix" Training to Allow; that trades real protection for
# nothing. (An earlier version of this comment advised the opposite, on the
# mistaken premise that GPTBot was what brought chatgpt.com referrals.)
#
# Search: Allow also leaves Googlebot + Bingbot untouched, so there is no SEO
# risk here. AdSense is outside these controls entirely — Mediapartners-Google
# and AdsBot-Google are ad crawlers, not AI crawlers.
#
# Bot Preference Sync, if enabled, PREPENDS Cloudflare's directives to this
# file at the edge, so the served /robots.txt would stop matching this source
# file. Leave it OFF to keep this file authoritative. Verified 2026-09-20: the
# served file is byte-identical to this one.
# ===========================================================================

# OpenAI bots. OAI-SearchBot (ChatGPT's search index) and ChatGPT-User (live
# fetch when a user asks a question) are what make ChatGPT cite this site and
# refer users; per analytics chatgpt.com is a meaningful referral source. Those
# two lines below are what protect that traffic, and both are Allow at the edge.
#
# GPTBot is OpenAI's TRAINING crawler and contributes nothing to citations or
# referrals. It was Allow here until 2026-09-20 on the mistaken premise that it
# was what brought chatgpt.com traffic. Now aligned with every other training
# crawler in this file, and with the zone's Training: Disallow, which already
# blocks it at the edge. This changes no behaviour today — it makes the stated
# policy match the enforced one, and it is what would apply if Training were
# ever switched to Allow.
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

# Perplexity bots: ALLOWED based on GA4 data (2026-06-21) — Copilot is already
# bringing AI-assistant traffic (4 sessions in 28 days). Perplexity is in the
# same category and is becoming a meaningful traffic channel for tax/finance
# queries. Allowing both the crawler and the user-agent variant.
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Other AI training/scraping bots remain blocked.
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Omgili
Disallow: /

User-agent: ImagesiftBot
Disallow: /

# Sitemap
Sitemap: https://smarttaxcalc.in/sitemap.xml