How It Works
Netify DPI - How It Works
The open-source Netify DPI Agent does one thing and one thing very well: network analysis using deep packet inspection (DPI). The DPI Agent passively captures network traffic and produces both network metadata streams and high-speed Linux datasets. This solution can be used to:
- Prioritize Zoom traffic
- Put BitTorrent traffic into a lower-priority QoS bucket
- Block devices from using weak security ciphers on the network
- Measure application bandwidth and health by IP/subscriber
- Provide zero-rating on Netflix traffic
- Send network metadata to an in-house message queue system
In the Netify DPI Agent developer documentation, we go through the basic setup for delivering the above solutions. But first, let's take a look at the big-picture problem.
Problem: we need to get Zoom and the all-important Fortnite network traffic through with low latency and then shuffle less time-sensitive traffic like BitTorrent and Netflix to the "I don't care about latency" part of the network.
In the modern network, these application/protocol features require tapping into Layer 7 to get the necessary information. At Layer 2/3 (MAC addresses, IPs, UDP/TCP port numbers), Fortnite and Zoom can look like plain old HTTPS traffic on a common cloud platform like Amazon AWS. Differentiating this traffic using IPs and ports is complex and often impossible. Tricky protocols like BitTorrent, Wireguard, and others are impossible to manage at Layer 3.
The Linux kernel, traffic control tools (tc), iptables, nftables, and other engines do not speak native Layer 7, and nor should they. But that leaves us with a crucial problem:
Hello Netify DPI!
Netify DPI enters the chat. The DPI engine provides several ways to interact with Layer 7 data using Layer 3 tools and formats. Before diving into the details below, stepping back and understanding the Netify DPI workflow is essential. There are three stages:
- Inputs - the source of the network data
- Processors - processor plugins convert Layer 7 traffic into Layer 3 formats
- Outputs - output plugins can export the data to a log, socket, or message queue
Let's look at each step of the workflow in more detail.
Netify DPI Workflow
The agent ingests network traffic from what we call an Input. The input is either:
- Network interfaces, or
- Packet capture files
Most Netify Agent deployments are configured to capture live traffic from network interfaces, but capture files can be handy for testing, development, and offline processing.
Netify DPI ingests the network traffic from the inputs, examines the first 1 to 32 packets of a connection, and performs deep packet analysis. Subsequent packets are tracked for bandwidth statistics and various key performance metrics.
With the inputs configured, the agent uses processor plugins to transform the analysis into various metadata and datasets. The metadata can be used for bandwidth statistics, cybersecurity analysis, and more. The datasets, on the other hand, can be used by standard Layer 3 tools to implement:
- QoS and QoE
There are three standard processor plugins available in Netify DPI - these are described below.Flow Actions Processor
The Flow Actions Plugin provides a high-speed interface into firewall, QoS, and routing systems in Linux. In particular, the plugin produces three types of datasets:
- Extracted hostnames
- TLS ciphers
- Bandwidth statistics
- Performance indicators
- and many more
Third-party tools typically ingest this flow metadata to provide enhanced network intelligence solutions.Aggregator Processor
Flow data from Netify DPI generates a very large amount of data. However, the Aggregator Plugin provides a way to summarize this data more compactly. For example, the plugin can be configured to track total bandwidth usage by a particular application - an efficient way to offer zero-rating.
The final part of the Netify DPI agent workflow is outputs. The metadata from one or more of the Netify processors can be connected to any of the following outputs:
- Message Queue
And yes, you can mix and match these plugins. For example, the flow metadata from the core processor can be sent to the socket for analysis while simultaneously sending aggregator processor bandwidth data to a message queue server.
If you want to start diving into a Netify DPI agent implementation, you can jump into the Installing Netifyd documentation. Alternatively, feel free to explore the various features implemented in Netify's plugin system.