Data Streams

Netify - Data Streams

The Netify Agent handles the deep packet inspection and analysis, but how do you interact with the data available from the agent? Answer: data streams.

Data via Socket Interface

"type": "flow_purge"
  "flow": {
    "digest": "178bf5650a79d5e8ddc6a988d0c02b3d799180d0",
    "last_seen_at": 1606232131756,
    "local_bytes": 2434,
    "local_packets": 21,
    "other_bytes": 6139,
    "other_packets": 16,
    "total_bytes": 8573,
    "total_packets": 37
  },
  "reason": "terminate",
  ... snip ...

Getting Started with Live Data Streams

If you haven't already done so, please start with the getting started introduction for instructions on how to get an environment with the Netify Agent up and running. Once ready, we can use some shell tools to connect to netifyd and display the JSON encoded payloads. First, we need to install the shell utilities:

[socket]
listen_path[0] = /var/run/netifyd/netifyd.sock

Open up a separate terminal on a system with netifyd installed. On the first terminal, run the netifyd command to start processing either a PCAP capture file (instructions) or live network data (instructions). On the second terminal, run the following to connect to the JSON data stream and view formatted output:

sudo nc -U /var/run/netifyd/netifyd.sock | jq . -C

You should see a stream of JSON data similar to the following:

{
  "agent_version": 3.06,
  "build_version": "Netify Agent/3.06 (... features ...) nDPI/2.9.0 JSON/1.90",
  "json_version": 1.9,
  "type": "agent_hello"
}

Third Party Tools

We'll get to the details of the Data Stream Format in the next step. With a JSON-encoded data stream, it's possible to develop Netify-aware tools in any programming language:

• Python
• C/C++
• Rust
• etc.

In addition, it's also important to remember that the netifyd data stream is available over a TCP/IP socket (see man netifyd.conf to enable this feature). This makes it possible to run the Netify Agent detection on a dedicated network probe, but then run your Netify-enabled application on a different server host.

To give you a quick example, we have provided a shell script that connects to the UNIX socket file. The script will output the application, protocol, and a few other details in CSV format. Open up the second terminal on the netifyd-enabled system. Run the following example shell script (netifyd version 3.06 or later):

/usr/share/netifyd/json-socket-example.sh

Press Enter to start the script. If you are using a PCAP capture file, you will see warnings about waiting for the Netify Agent to start. In this case, go back to the first terminal and run the netifyd command to start processing the capture file. You can either wait for the DPI processing to complete, or you can interrupt it at any time. On the second terminal, you should see CSV formatted output with application, protocol, and other packet information for every flow. Please be patient, it will take a few seconds for data to appear.

timestamp,digest,local_ip,local_port,other_ip,other_port,protocol,application
"2020-11-24 15:35:16.918",59be4,192.168.4.189:48476,172.217.164.238:443,HTTPS,126.netify.google
"2020-11-24 15:35:20.939",d6a1d,192.168.4.189:35596,172.217.164.202:443,HTTPS,126.netify.google
"2020-11-24 15:35:20.971",0bc51,192.168.4.189:40480,172.217.165.10:443,HTTPS,126.netify.google
"2020-11-24 15:35:22.974",cb810,192.168.4.189:52895,91.189.89.199:123,NTP,Unknown
"2020-11-24 15:35:24.333",d3531,192.168.4.189:40328,162.159.135.234:443,HTTPS,206.netify.cloudflare
"2020-11-24 15:35:24.598",ff012,192.168.4.189:48540,35.182.46.62:443,HTTPS,10033.netify.netify
"2020-11-24 15:35:25.632",178bf,192.168.4.189:52974,52.216.110.189:443,HTTPS,10091.netify.amazon-aws
"2020-11-24 15:35:28.179",331ba,192.168.4.189:57621,192.168.4.255:57621,Spotify,156.netify.spotify
"2020-11-24 15:35:29.839",c9435,192.168.4.189:34308,172.217.164.229:443,HTTPS,126.netify.google
"2020-11-24 15:35:29.950",5c566,192.168.4.189:36434,239.255.255.250:1900,SSDP,Unknown
"2020-11-24 15:35:30.559",bcc70,192.168.4.189:48474,172.217.164.238:443,HTTPS,126.netify.google

Netify Integrations

With the Netify JSON network stream, you can develop and create your own integrations: open source, proprietary, in-house... it's up to you. The netifyd engine is open source and we welcome changes, tweaks, and contributions. We have also built two integrations to help with getting started with firewalls/QoS and local reporting:

• The Netify Firewall Agent is a Python-based engine that hooks into firewalls and QoS systems.
• The Netify Console is an ncurses-based application for ClearOS that shows flow data via the command line.

Next Steps

Now that you have been able to interact with the Netify Agent data stream, it's time to review what kind of network metadata is available in the stream. You can find more information in the Data Stream Types documentation.