Crawlers and Bots
This page details network ranges, IPs, and points-of-presence (PoPs) for well known crawlers and bots. We have also provided a list of the top crawler/bot hosting providers.
The surge in AI-driven bot traffic necessitates granular visibility; our DPI engine provides the precision required to distinguish legitimate automated services from aggressive scraping and malicious activity.
Top Crawlers and Bots
The Netify application data feeds provides information on the following crawlers/bots:
Crawler and Bot Hosting
The following is a list of the top crawler/bot hosting providers based on the number of different crawlers/bots.
| Host | IPs | Crawlers |
|---|---|---|
Microsoft Azure
|
4,139 | 1 |
Meta CDN
|
1,610 | 1 |
Huawei Cloud
|
1,500 | 1 |
Scaleway
|
1,042 | 3 |
Hurricane Electric
|
914 | 1 |
Hetzner
|
597 | 3 |
OVHcloud
|
584 | 3 |
Amazon AWS
|
426 | 1 |
Vultr
|
375 | 1 |
Google Cloud Platform
|
364 | 2 |
THG Ingenuity
|
189 | 1 |
DigitalOcean
|
182 | 1 |
Veloxserv
|
14 | 1 |
Leaseweb
|
9 | 1 |
Cloudflare
|
2 | 1 |
Zenlayer
|
2 | 1 |
IOmart
|
1 | 1 |
Lumen
|
1 | 1 |