[Fixed] Cloudflare Blocking Google Ad-Bot From Crawling Ads.txt

Cloudflare users sometimes face a problem where Cloudflare unintentionally blocks Google’s Ad-Bot from crawling the ads.txt file. This issue can have significant implications, as the ads.txt file plays a crucial role in verifying and authorizing digital ad inventory. Mostly this issue arises after turning on Botfight mode from the Cloudflare security section.

In this article, we will provide you with a step-by-step guide to resolving this issue using a custom Web Application Firewall (WAF) rule to allow incoming requests that contain the “AdsBot-Google” value in the User Agent.

Steps to fix Cloudflare Blocking Google Bot Issue

Step 1: Temporarily Disable Bot Fight Mode

To begin addressing this issue, you need to temporarily disable the Bot Fight Mode feature in Cloudflare. While this feature is designed to protect your website from malicious bots, it can occasionally interfere with legitimate bots like Google’s Ad-Bot. Follow these steps to disable it:

1. Log in to your Cloudflare account.
2. Select the website where you’re experiencing the issue.
3. Navigate to the “Firewall” tab.
4. Click on “Bots” in the submenu.
5. Find the “Bot Fight Mode” section and toggle the switch to “Off.”

Remember, disabling Bot Fight Mode is a temporary measure. Once you’ve created a custom WAF rule (as described in the next step), you can re-enable it after your error in the Adsense account is gone.

Step 2: Create a Custom WAF Rule

After disabling Bot Fight Mode, the next step is to create a custom WAF rule that allows incoming requests with the “AdsBot-Google” value in the User Agent. This will ensure that Google’s Ad-Bot can crawl your ads.txt file without being blocked by Cloudflare. Here’s how to do it:

1. Go to the “Security -> WAF” tab in your Cloudflare dashboard.
2. Click on the “Create rule” button available on the page.
3. Give your rule a name, such as “Allow Google Ad-Bot.”
4. In the “Field” dropdown menu, choose “User Agent.”
5. In the “Operator” dropdown menu, select “Contains.”
6. Enter “AdsBot-Google” in the “Value” field.
7. In the “Choose an action” dropdown menu, select “Skip”
8. Now check options to skip such as “All super bot fight mode rules” (see the following screenshot)
9. Click on ” Save”

By following these steps, you should be able to resolve the issue of Cloudflare blocking Google’s Ad-Bot from crawling your ads.txt file.

Remember to re-enable Bot Fight Mode once you’ve created the custom WAF rule to ensure ongoing protection against malicious bots.

As always, it’s essential to monitor your website’s performance and traffic to ensure that all bots, whether good or bad, interact with your site as expected.