Data Streams
Netify DPI - Data Streams
This document provides details on the network metadata information available through the Netify DPI Data Streams. If you would like to interact with the data in a live environment, please start with the getting started guide.
Integrations
Cybersecurity engines, QoE managers, performance analytics solutions, and other third party applications can connect to the Netify DPI Agent to interact with the network flow metadata. The Netify Agent's socket interface (TCP/IP and UNIX sockets) provides near real-time agent status and flow/DPI detection data via JSON encoded payloads.
With the Netify DPI Data Stream, you can develop and create your own integrations: open source, proprietary, in-house... it's up to you. You can find links to a couple of example open source applications that consume Netify data streams at the end of this document.

Data Stream Types
There are several different types of data available via the Netify Agent JSON encoded data streams:
- Flow - Flow detection metadata
- Flow Purge - Flow purge and performance data
- Flow Status - Flow status data
- Agent Hello - Agent startup information
- Agent Status - Agent status information
Information on each type of data is described below.
Type: Flow - Detection Metadata
Description
Detection data is available as soon as the first 1-10 packets of a network conversation has been completed. This data provides a path to quickly react to network traffic patterns on the network. It also provides the network metadata extracted from the deep packet inspection engine.
Let's take a look at some sample data (see adjacent JSON data). This particular flow comes from a mobile phone that started up the WhatsApp app. You can see the usual network data in the JSON structure: IP version, MAC addresses, IPs, ports, network interfaces, and timestamps. In addition, you can see the additional DPI data. Read on.
Application: WhatsApp
The detected_application_name shows that it was a flow generated by WhatsApp. In this particular case, this detection was made using the TLS/SSL SNI hostname and certificate common name. Other heuristics are also used to detect applications.
Protocol: HTTPS
The DPI engine detected the HTTPS protocol (detected_protocol). This is more than just mapping port 443 to HTTPS, the engine did a full deep packet inspection. In fact, Netify will detect SSL/SNI over any port.
DPI: HTTPS Details
Detecting the protocol is just the start. The Netify Agent also breaks down the details of the SSL/TLS connection:
- SNI hostname: static.whatsapp.net
- Certificate CN: *.whatsapp.net
- TLS version: 0x0303 (TLS 1.2)
- Encryption cipher: 0xc02b info
- JA3/TLS fingerprint: details
Other protocols will have different metadata attached. For example, HTTP traffic also includes the URL and user agent.
{
"type": "flow",
"interface": "ens34",
"internal": true,
"established": false,
"flow": {
"digest": "09c9e2b73d68fef245c09141cb63dad8d9001a6c",
"ip_nat": false,
"ip_version": 4,
"ip_protocol": 6,
"vlan_id": 0,
"other_type": "remote",
"local_origin": true,
"other_mac": "00:90:fb:29:ca:ba",
"local_mac": "a0:c9:a0:e5:2c:eb",
"other_ip": "31.13.80.53",
"local_ip": "192.168.4.105",
"other_port": 443,
"local_port": 38972,
"detected_protocol": 196,
"detected_protocol_name": "HTTPS",
"detected_application": 544,
"detected_application_name": "142.netify.whatsapp",
"detection_guessed": 0,
"ssl": {
"version": "0x0303",
"cipher_suite": "0xc02b",
"client_sni": "static.whatsapp.net",
"server_cn": "*.whatsapp.net",
"client_ja3": "d8c87b9bfde38897979e41242626c2f3",
"server_ja3": "6e15a5bf660856fa03186247ca41d059"
},
"first_seen_at": 1574786068665,
"first_update_at": 1574786068665,
"last_seen_at": 1574786068715
}
}
Example Use Cases
- SD-WAN routing policies based on applications
- Firewall and QoS policies based on DPI information
- Wireless access point (WAP) optimization
Type: Flow Purge - Stats and Performance Data
Description
When a network flow is either closed or becomes idle, the Netify Agent publishes a flow_purge record to the Data Stream. Today, this stream provides network statistics: byte counters, traffic counters, timestamps, etc. Later in 2021, we will be adding performance information to the payload, for example:
- Statistics on DNS response times
- HTTP response codes and times
- Network performance statistics
- and more.
Note: requires netifyd version 3.06 or later.
"type": "flow_purge"
"reason": "terminate",
"flow": {
"digest": "178bf5650a79d5e8ddc6a988d0c02b3d799180d0",
"last_seen_at": 1606232131756,
"local_bytes": 2434,
"local_packets": 21,
"other_bytes": 6139,
"other_packets": 16,
"total_bytes": 8573,
"total_packets": 37
... snip ...
Example Use Cases
- Cybersecurity analysis
- Policy and Charging Rules Function (PCRF)
- Zero rating policy engines
Type: Flow Status - Status Information
Description
Some network flows are long-lived, for example, audio/video streams and VPN connections. The Netify Agent will periodically publish a flow_status record to the Data Stream. The network status on active flows provides real-time insights into the network.
Example Use Cases
- Live bandwidth statistics
- QoE based on live data usage
Note: open-source version available mid-2021.
"type": "flow_status",
"flow": {
"digest": "178bf5650a79d5e8ddc6a988d0c02b3d799180d0",
"last_seen_at": 1606232131756,
"local_bytes": 2434,
"local_packets": 21,
"other_bytes": 6139,
"other_packets": 16,
"total_bytes": 8573,
"total_packets": 37
},
Type: Agent Hello
The agent_hello data type provides versioning and feature information. This makes it possible for third-party applications to manage upgrades and changes to the Netify Agent data stream.
"type": "agent_hello",
"agent_version": 3.06,
"build_version": "Netify Agent/3.06 (debian; x86_64; netlink; dns-cache; plugins; tcmalloc; inotify; regex) nDPI/2.9.0 JSON/1.90",
"json_version": 1.9
Type: Agent Status
The agent_status data type provides agent and system status information. This makes it possible to monitor and manage the performance of the underlying DPI engine.
"type": "agent_status",
"cpu_cores": 1,
"cpu_system": 0.157217,
"cpu_system_prev": 0.103613,
"cpu_user": 0.117912,
"cpu_user_prev": 0.103613,
"dhc_size": 0,
"dhc_status": true,
"flows": 25,
"flows_prev": 22,
"maxrss_kb": 42352,
"maxrss_kb_prev": 42292,
"sink_status": false,
"sink_uploads": false,
"tcm_kb": 22825,
"tcm_kb_prev": 22838,
"timestamp": 1606274389,
"update_imf": 1,
"update_interval": 15,
"uptime": 30
Further Reading
Here are some related topics:
- Developer Getting Started - hands-on developer getting started guide.
- Network Policies Overview - Layer 3 hooks for firewall, routing, and QoS.
- Netify DPI Overview - a big picture summary.
We have also built two example open source applications using Netify Data Streams:
- Netify Firewall Agent - a Python-based engine that uses data streams to block unwanted network traffic.
- Netify Console - an ncurses-based application for ClearOS that shows live flow data.
Netify Agent
Getting Started
Integration
Netify Tools
Netify DPI Alternatives
Integration and Custom Solutions
Do you have any questions about integration, APIs or custom development?
Contact Us