Our crawler

EagleEnforceBot

If you found this page from your server logs: hello. This explains what our crawler is, what it reads, and how to limit it.

What it does

EagleEnforce helps brands check the prices at which their own products are publicly advertised. When a brand adds a store it sells through, our crawler reads that store's public product pages and product feeds to find the brand's products and record the advertised price. It reads only what any visitor can see without signing in. It does not add items to carts, create accounts, submit forms, or read anything behind a login.

How it behaves

• It obeys robots.txt. Before any page is fetched we read your /robots.txt. Rules for EagleEnforceBot and for * are both honoured, and Crawl-delay is respected. A path robots.txt disallows is never fetched.
• It prefers feeds to pages. Where a platform publishes a product feed (for example Shopify's products.json or the WooCommerce Store API) we read that instead of crawling page by page, which keeps our footprint to a handful of requests per store.
• It is slow on purpose. One domain is read by one worker at a time with a minimum delay between requests, and we back off when a server answers 429 or 503.
• It identifies itself. Every request carries the user agent above. We do not use proxies, rotating IP addresses, or browser impersonation.
• It comes from a fixed address. Requests originate from our servers in the United States. Email us if you want the current IP address for an allow-list.

How to limit or block it

Add rules to your robots.txt. They take effect on our next read of the file, which happens at least daily:

To slow it down instead of blocking it:

You can also email support@ecommeagle.com with your domain and we will exclude it by hand.

Why a brand might be reading your store

If your store carries a brand that uses EagleEnforce, that brand asked us to read your public prices to check them against its Minimum Advertised Price policy. Questions about the policy itself should go to the brand. Questions about our crawler, or about data we hold about your business, can come to us: see our Privacy Policy.