Bots and Crawlers

Bots and Crawlers Hierarchy

Crawlers and bots are widely used by search engines, cybersecurity scanners, and AI systems. We classify this activity under a distinct application category to improve visibility and control.

For instance, you might want to block ChatGPT's crawler traffic from scanning your network, while still allowing employees to access its web app. In other cases, it may be beneficial to combine both applications under a unified application definition.

Application Hierarchies

Ahrefs
App ID: 11895

Ahrefs is a search engine optimization toolset that provides data on backlinks, keyword rankings, and site performance.

AhrefsBot

Business

App ID: 11894

Babbar
App ID: 11996

Babbar is an SEO tool that analyzes website factors, including backlinks and semantics, to help improve search engine rankings.

Babbar Bot

Business

App ID: 11997

Bing
App ID: 10022

Bing is a web search engine owned and operated by Microsoft. The service provides a variety of search services, including web, video, image and map search products.

Bing Maps	Reference	App ID: 10226
Bingbot	Business	App ID: 11892

Censys
App ID: 11919

Censys is a search engine and cybersecurity platform that scans and indexes devices, servers, and networks across the internet to provide insights into exposed assets and potential vulnerabilities.

Censys Scanner

Cybersecurity

App ID: 11920

DataForSEO
App ID: 12003

DataForSEO is an API provider that offers comprehensive SEO data solutions, including keyword research, search engine analysis, backlinks data, and rank tracking.

DataForSEO Bot

Business

App ID: 12004

Driftnet
App ID: 12137

Driftnet is a platform that provides internet-wide intelligence, scanning, and attack surface management tools for security professionals.

Driftnet Census

Cybersecurity

App ID: 12138

Majestic
App ID: 11896

Majestic is a search engine optimization tool specializing in backlink analysis and site performance metrics.

MajesticBot

Business

App ID: 11897

Naver
App ID: 11599

Naver is a South Korean technology company that operates various online services and platforms including, webtoons, online gaming, news aggregation, and social networking.

LINE	Messaging	App ID: 10906
Naver Bot	Business	App ID: 12101
Youdao	Career and Education	App ID: 11426

Onyphe
App ID: 11923

Onyphe is a cybersecurity search engine that aggregates and analyzes data from internet-connected devices and networks.

Onyphe Bot

Cybersecurity

App ID: 11924

OpenAI
App ID: 11404

OpenAI is an artificial intelligence company that develops advanced AI models and tools, including ChatGPT.

ChatGPT	Technology	App ID: 11445
ChatGPT Bot	Technology	App ID: 11898

Petal Search
App ID: 11992

Petal Search is a mobile search engine developed by Huawei. It provides users with search for apps, news, images, videos, and more.

PetalBot

Business

App ID: 11993

Qwant
App ID: 12245

Qwant is a French privacy-focused search engine that delivers search results without tracking users or personalizing based on their data.

Qwantbot

Business

App ID: 12244

Rapid7
App ID: 10852

Rapid7 is a provider of security data and analytics solutions enabling organizations to implement an active approach to cybersecurity.

Rapid7 Scanner

Cybersecurity

App ID: 12141

Semrush
App ID: 11916

Semrush is a comprehensive digital marketing tool that offers SEO, content marketing, competitor analysis, and advertising insights to help businesses improve their online visibility and strategy.

Semrush Bot

Business

App ID: 11917

Shadowserver
App ID: 11921

Shadowserver is a non-profit organization that gathers, analyzes, and shares cybersecurity data globally to help detect, prevent, and mitigate cyber threats and vulnerabilities.