Netify DPI Agent

Netify DPI - How It Works

The Problem
The Solution: Netify DPI
Workflow

The open-source Netify DPI Agent does one thing and one thing very well: network analysis using deep packet inspection (DPI). The DPI Agent passively captures network traffic and produces both network metadata streams and high-speed Linux datasets. This solution can be used to:

Prioritize Zoom traffic
Put BitTorrent traffic into a lower-priority QoS bucket
Block devices from using weak security ciphers on the network
Detect devices on the network
Measure application bandwidth and health by IP/subscriber
Provide zero-rating on Netflix traffic
Send network metadata to an in-house message queue system

In the Netify DPI Agent developer documentation, we go through the basic setup for delivering the above solutions. But first, let's take a look at the big-picture problem.

The Problem

Problem: we need to get Zoom and the all-important Fortnite network traffic through with low latency and then shuffle less time-sensitive traffic like BitTorrent and Netflix to the "I don't care about latency" part of the network.

In the modern network, these application/protocol features require tapping into Layer 7 to get the necessary information. At Layer 2/3 (MAC addresses, IPs, UDP/TCP port numbers), Fortnite and Zoom can look like plain old HTTPS traffic on a common cloud platform like Amazon AWS. Differentiating this traffic using IPs and ports is complex and often impossible. Tricky protocols like BitTorrent, Wireguard, and others are impossible to manage at Layer 3.

The Linux kernel, traffic control tools (tc), iptables, nftables, and other engines do not speak native Layer 7, and nor should they. But that leaves us with a crucial problem:

How do we get Layer 7 information into a format standard Layer 3 tools can understand?

Solution: Hello Netify DPI!

Netify DPI enters the chat. The DPI engine provides several ways to interact with Layer 7 data using Layer 3 tools and formats. Before diving into the details below, stepping back and understanding the Netify DPI workflow is essential. There are three stages:

Inputs - the source of the network data
Processors - processor plugins convert Layer 7 traffic into Layer 3 formats
Outputs - output plugins can export the data to a log, socket, or message queue

Let's look at each step of the workflow in more detail.

Netify DPI Workflow

Inputs

The agent ingests network traffic from what we call an Input. The input is either:

Network interfaces, or
Packet capture files

Most Netify Agent deployments are configured to capture live traffic from network interfaces, but capture files can be handy for testing, development, and offline processing.

Netify DPI ingests the network traffic from the inputs, examines the first 1 to 32 packets of a connection, and performs deep packet analysis. Subsequent packets are tracked for bandwidth statistics and various key performance metrics.

Inputs

Traffic is captured off live network interfaces via a number of different drivers: pcap, tpacket v3, and NFQ.

Network Interfaces

Processors

With the inputs configured, the agent uses processor plugins to transform the analysis into various metadata and datasets. The metadata can be used for bandwidth statistics, cybersecurity analysis, and more. The datasets, on the other hand, can be used by standard Layer 3 tools to implement:

QoS and QoE
Firewalls
Routing

There are three standard processor plugins available in Netify DPI - these are described below.

Flow Actions Processor

The Flow Actions Plugin provides a high-speed interface into firewall, QoS, and routing systems in Linux. In particular, the plugin produces three types of datasets:

Core Processor

The Core Plugin provides network metadata information in JSON format. Metadata includes:

Application
Protocol
Extracted hostnames
TLS ciphers
Bandwidth statistics
Performance indicators
and many more

Third-party tools typically ingest this flow metadata to provide enhanced network intelligence solutions.

Aggregator Processor

Flow data from Netify DPI generates a very large amount of data. However, the Aggregator Plugin provides a way to summarize this data more compactly. For example, the plugin can be configured to track total bandwidth usage by a particular application - an efficient way to offer zero-rating.

Processors

Processors plugins transforms the network traffic into metadata streams and actions.

Outputs

The final part of the Netify DPI agent workflow is outputs. The metadata from one or more of the Netify processors can be connected to any of the following outputs:

Socket
Message Queue
Log

And yes, you can mix and match these plugins. For example, the flow metadata from the core processor can be sent to the socket for analysis while simultaneously sending aggregator processor bandwidth data to a message queue server.

Outputs

The data from the processors can be output to a number of different outputs:

Next Steps

If you want to start diving into a Netify DPI agent implementation, you can jump into the Installing Netifyd documentation. Alternatively, feel free to explore the various features implemented in Netify's plugin system.