BotObservatory – Live Bot Tracker & Anti-Scraper

Maelezo

BotObservatory is a WordPress request observatory for site owners who want to see more than a single “unknown traffic” bucket.

It records requests that actually reach WordPress, separates ordinary human traffic from recognized crawler profiles, detects supported search and AI referrals when a browser supplies a usable referrer, and keeps suspicious or unidentified automation in a dedicated Weird Creatures registry.

BotObservatory can also apply temporary HTTP 429 quarantine rules to aggressive IP addresses. Automatic quarantine is disabled by default so administrators can first verify how their hosting or reverse proxy exposes the real client IP address.

Official documentation, a live demonstration, and product information are available on the IAQ Products page.

Main features

• Live dashboard for human and machine requests that reach WordPress.
• Separate Search Referrals and AI Referrals dashboard counters.
• Editable Bot Profiles for AI, search, social, SEO, and custom crawlers.
• Dashboard/Public Card selection for individual bot profiles.
• Weird Creatures registry for suspicious and unidentified automation.
• Optional anti-scraper quarantine using configurable request thresholds.
• Default quarantine duration of 24 hours when protection is enabled.
• Public live machine observatory through the [botobservatory_live] shortcode.
• Configurable colors for the public observatory.
• Paginated Recent Logs view for larger traffic histories.
• Log filtering by type, source, IP address, and URL.
• Filter-aware CSV log export.
• Per-card log cleanup and complete log cleanup.
• Automatic log retention with a 7-day default; retention can be changed or disabled in Display Settings.
• Optional read-only Cloudflare Analytics integration for edge-vs-origin crawler visibility.
• Cloudflare crawler destination reporting with time-window and provider filters.
• Obvious credential, configuration, traversal, and exploit-probe paths are excluded from Cloudflare crawler analytics.
• WordPress infrastructure requests are excluded from visitor statistics.
• Translation-ready strings with an included POT template.
• No external BotObservatory analytics service is required.

Bot Profiles

Bot Profiles classify requests by configured User-Agent fragments. Profiles can be enabled, disabled, edited, prioritized, and selected for dashboard/public bot cards.

User-Agent matching is classification, not identity verification. User-Agent strings can be spoofed. A profile match should therefore not be treated as cryptographic or organizational proof of origin.

Search and AI referrals

When a browser provides a usable HTTP Referer header, BotObservatory can classify supported search and AI referral sources.

Search referral detection includes Google, Bing, Yahoo, Yandex, DuckDuckGo, Baidu, Ecosia, Brave Search, Startpage, Qwant, Naver, Seznam, and AOL.

AI referral detection includes ChatGPT, Perplexity, Claude, Microsoft Copilot, Google Gemini, Grok, Poe, You.com, Phind, and Meta AI.

Some browsers, applications, privacy tools, redirects, and referrer policies remove or suppress referral information. BotObservatory does not guess a source when reliable referral information is absent; those visits remain Direct or ordinary Referral traffic.

Weird Creatures

Requests that look automated but do not match an enabled Bot Profile can be recorded as Weird Creatures. Security probes can also be surfaced there for inspection.

Being listed as a Weird Creature does not by itself mean an IP address is blocked. Quarantine is a separate behavior controlled by the anti-scraper rules.

Administrators can also inspect and release active temporary blocks.

Anti-scraper quarantine

Automatic quarantine is disabled by default.

When enabled, the default thresholds are:

• 30 requests in 5 minutes, or
• 12 requests in 60 seconds, or
• 5 security probes inside the configured burst window.

The default block duration is 24 hours. Blocked requests receive HTTP status 429 and a Retry-After header.

BotObservatory uses the server-provided REMOTE_ADDR by default. It does not blindly trust X-Forwarded-For, X-Real-IP, or CDN-specific client-IP headers. Sites behind a trusted reverse proxy or CDN can integrate verified client-IP handling with the botobservatory_client_ip filter.

Before enabling automatic quarantine, confirm that WordPress sees the correct client IP address for your hosting architecture.

WordPress infrastructure filtering

BotObservatory is intended to observe visitors and machines, not routine WordPress plumbing.

Requests such as wp-cron.php, xmlrpc.php, wp-login.php, admin-ajax.php, admin-post.php, comment-posting, and trackback endpoints are excluded from visitor statistics. Ordinary non-bot requests to WordPress admin, REST, includes, and content infrastructure paths are also excluded where appropriate.

Recognized bot profiles may still be observed on appropriate crawlable paths when useful for crawler visibility.

Public live observatory

Add this shortcode to a post, page, or supported widget area:

[botobservatory_live]

To use a custom heading:

[botobservatory_live title="Machine Activity"]

The public observatory displays aggregate activity. When Cloudflare Edge is configured, it can also show named crawler families seen only at the edge by reusing the latest successful cached snapshot; public page loads never call the Cloudflare API. It does not expose visitor IP addresses or raw request logs.

Cloudflare Edge integration

BotObservatory can optionally connect to the Cloudflare GraphQL Analytics API with a read-only API token and Zone ID. When configured, the Cloudflare Edge screen compares named crawler requests observed at the Cloudflare edge with matching crawler traffic recorded by WordPress.

The integration includes:

• 1, 6, 24, 72, and 168-hour windows,
• provider/family filtering,
• edge request counts,
• WordPress-origin request counts,
• origin reach percentage and difference,
• crawler destination paths,
• Edge only / Reached origin / Origin only states, and
• manual refresh with a 24-hour successful snapshot cache.

Cloudflare access is read-only. BotObservatory does not modify Cloudflare WAF, DNS, caching, or bot-management settings. The API token is stored as a WordPress option unless credentials are supplied with BOTOBSERVATORY_CLOUDFLARE_ZONE_ID and BOTOBSERVATORY_CLOUDFLARE_API_TOKEN constants. The saved token is never rendered back into the settings screen.

External service: Cloudflare

The Cloudflare Edge integration is optional and disabled until a site administrator configures it. When an administrator requests or refreshes Cloudflare analytics, BotObservatory sends the configured Zone ID and a GraphQL analytics query directly to Cloudflare’s API. The configured Cloudflare API token is sent directly to Cloudflare for authentication. BotObservatory does not send the token, Zone ID, analytics query, or returned analytics data to the plugin author and does not operate an intermediary service.

Cloudflare API documentation: https://developers.cloudflare.com/api/
Cloudflare Terms: https://www.cloudflare.com/website-terms/
Cloudflare Privacy Policy: https://www.cloudflare.com/privacypolicy/

To keep the Cloudflare view focused on meaningful crawler activity, BotObservatory excludes obvious credential/configuration discovery, traversal, and exploit-probing paths such as .env, .git, .ssh, cloud credential files, database dumps, wp-config.php backups, profiler/debug endpoints, and /proc/self/environ. User-Agent matching still identifies the crawler profile, so Cloudflare provider/ASN ownership alone is not treated as proof of crawler identity.

Log retention and cleanup

BotObservatory automatically removes old log rows after the configured retention period. The default is 7 days. In Display Settings, administrators can change the day count or disable automatic log deletion entirely.

Administrators can also clear:

• all request logs,
• human logs,
• bot logs,
• strange/security logs,
• search referral logs,
• AI referral logs, or
• logs associated with an individual selected bot card.

Clearing request logs does not automatically delete the Weird Creatures registry or active-ban records.

What BotObservatory can see

BotObservatory runs at the WordPress application layer. It can observe requests that execute WordPress.

Requests blocked by a firewall, web server, security appliance, or CDN before WordPress executes cannot be recorded by the plugin. Full-page edge caching can also prevent PHP from executing for some page views. The included same-site browser beacon helps restore visibility for ordinary browser page views in cached environments, but it cannot reveal machine traffic that never reaches WordPress.

Privacy

BotObservatory may store request URLs, referrers, User-Agent strings, IP addresses, classifications, evidence/messages, and timestamps in the site’s WordPress database.

The plugin does not require a remote BotObservatory analytics service for its core functionality.

If the optional Cloudflare Edge integration is enabled, administrator-initiated analytics requests are sent directly to Cloudflare as described in the External service: Cloudflare section above.

IP addresses and request metadata may be personal data under laws applicable to a site. Site owners are responsible for choosing an appropriate retention period, limiting administrator access, updating their privacy disclosures where required, and handling applicable data-subject requests.

The default log retention period is 7 days. Automatic deletion can be disabled or the retention period can be changed in Display Settings.