Aggregator Processor
Netify Aggregator Processor Plugin
- Introduction
- License
- Installation
- Configuration
- Aggregator Types
- Upload and Download Conventions
- Examples
Introduction
The Aggregator Processor plugin is responsible for creating structured data objects from the internals of the Netify Agent's memory by aggregating individual flow data. This data is typically used for creating high-level dashboards or returning usage statistics from within data centers and SD-WAN aggregation points.
License
Netify Flow Actions Plugin is a proprietary plugin requiring a license. Please contact us for details.
Installation
Netify plugins are packaged in the same workflow as the agent and can usually be installed using a similar syntax that was implemented during the installation of the Netify agent. Exceptions to this rule occur when the plugin requires a Software License Agreement or if an upgrade or downgrade to a different version of the plugin is required.
Alma Linux
Alma Linux 9
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
cd /tmp/
wget https://client:secret@download.netify.ai/5/almalinux/9/stable/x86_64/netify-proc-aggregator_1.0.48-1.os9.x86_64.rpm
apt install ./netify-proc-aggregator_1.0.48-1.os9.x86_64.rpm
Alma Linux 8
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
cd /tmp/
wget https://client:secret@download.netify.ai/5/almalinux/8/stable/x86_64/netify-proc-aggregator_1.0.48-1.os8.x86_64.rpm
yum install ./netify-proc-aggregator_1.0.48-1.os8.x86_64.rpm
Debian
Debian 13 (Bookworm)
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
cd /tmp/
wget https://client:secret@download.netify.ai/5/debian/10/amd64/netify-proc-aggregator_1.0.48-1_amd64.deb
apt install ./netify-proc-aggregator_1.0.48-1_amd64.deb
Debian 11 (Bullseye)
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
cd /tmp/
wget https://client:secret@download.netify.ai/5/debian/10/amd64/netify-proc-aggregator_1.0.48-1_amd64.deb
apt install ./netify-proc-aggregator_1.0.48-1_amd64.deb
Debian 10 (Buster)
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
cd /tmp/
wget https://client:secret@download.netify.ai/5/debian/10/amd64/netify-proc-aggregator_1.0.48-1_amd64.deb
apt install ./netify-proc-aggregator_1.0.48-1_amd64.deb
OpenWRT
OpenWRT 23.05
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
pkg add https://client:secret@download.netify.ai/5/openwrt/23.05/amd64/netify-proc-aggregator_1.0.48-1_x86_64.ipk
OpenWRT 22.03
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
pkg add https://client:secret@download.netify.ai/5/openwrt/22.03/amd64/netify-proc-aggregator_1.0.48-1_x86_64.ipk
OpenWRT 21.02
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
pkg add https://client:secret@download.netify.ai/5/openwrt/21.02/amd64/netify-proc-aggregator_1.0.48-1_x86_64.ipk
OpenWRT 19.07
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
pkg add https://client:secret@download.netify.ai/5/openwrt/19.07/amd64/netify-proc-aggregator_1.0.48-1_x86_64.ipk
OPNsense
OPNsense 24.1 - Savvy Shark
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
pkg add https://client:secret@download.netify.ai/5/freebsd/13.2/amd64/netify-proc-aggregator_1.0.48,1.pkg
OPNsense 23.7 - Restless Roadrunner
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
pkg add https://client:secret@download.netify.ai/5/freebsd/13.2/amd64/netify-proc-aggregator_1.0.48,1.pkg
pfSense
pfSense Plus 23.x
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
pkg add https://client:secret@download.netify.ai/5/freebsd/14.0/amd64/netify-proc-aggregator_1.0.48,1.pkg
pfSense CE 2.7.x
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
pkg add https://client:secret@download.netify.ai/5/freebsd/13.2/amd64/netify-proc-aggregator_1.0.48,1.pkg
Rocky Linux
Rocky Linux 9
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
cd /tmp/
wget https://client:secret@download.netify.ai/5/rockylinux/9/stable/x86_64/netify-proc-aggregator_1.0.48-1.os9.x86_64.rpm
apt install ./netify-proc-aggregator_1.0.48-1.os9.x86_64.rpm
Rocky Linux 8
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
cd /tmp/
wget https://client:secret@download.netify.ai/5/rockylinux/8/stable/x86_64/netify-proc-aggregator_1.0.48-1.os8.x86_64.rpm
yum install ./netify-proc-aggregator_1.0.48-1.os8.x86_64.rpm
openSUSE Linux
openSUSE Leap 15.5
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
cd /tmp/
wget https://client:secret@download.netify.ai/5/opensuse/15.5/stable/x86_64/netify-proc-aggregator_1.0.48-1.os15.5.x86_64.rpm
zypper install ./netify-proc-aggregator_1.0.48-1.os15.5.x86_64.rpm
Ubuntu
Ubuntu 24.04 - Noble
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
cd /tmp/
wget https://client:secret@download.netify.ai/5/ubuntu/noble/binary-amd64/netify-proc-aggregator_1.0.48-1_amd64.deb
apt install ./netify-proc-aggregator_1.0.48-1_amd64.deb
Ubuntu 22.04 - Jammy
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
cd /tmp/
wget https://client:secret@download.netify.ai/5/ubuntu/jammy/binary-amd64/netify-proc-aggregator_1.0.48-1_amd64.deb
apt install ./netify-proc-aggregator_1.0.48-1_amd64.deb
Ubuntu 20.04 - Focal
The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example:
cd /tmp/
wget https://client:secret@download.netify.ai/5/ubuntu/focal/binary-amd64/netify-proc-aggregator_1.0.48-1_amd64.deb
apt install ./netify-proc-aggregator_1.0.48-1_amd64.deb
Configuration
Plugin Loader Configuration
All plugins are disabled by default, and the Netify Aggregator Processor plugin is no different. To enable:
netifyd --enable-plugin proc-aggregator
--disable-plugin
.Alternatively, you can edit /etc/netifyd/plugins.d/10-netify-proc-aggregator.conf and set enable to yes.
# Netify Aggregator Processor Plugin Loader
# Copyright (C) 2023 eGloo Incorporated
#
##############################################################################
[proc-aggregator]
enable = yes
plugin_library = /usr/lib64/libnetify-proc-aggregator.so.0.0.0
conf_filename = ${path_state_persistent}/netify-proc-aggregator.json
# vim: set ft=dosini :
Plugin Configuration
Once the plugin has been enabled, it can be configured using the defined JSON configuration file specified in the plugin loader configuration. Let's look at a configuration sample to review the syntax and parts of the file.
{
"aggregator": 1,
"log_interval": 60,
"privacy_mode": false,
"format": "json",
"compressor": "gz",
"batched_rows": 0,
"nested": false,
"sinks": {
"sink-log": {
"default": {
"format": "json",
"compressor": "gz"
}
}
}
}
Property | aggregator |
---|---|
Description | Aggregator format type - valid options are 1, 2 or 3 - see the types section for more information. |
Type | integer |
Default | 1 |
Property | log_internal |
---|---|
Description | Interval time (in seconds) between aggregate summary reports. |
Type | integer |
Default | 60 |
Property | privacy_mode |
---|---|
Description | If true , aggregation will not include a breakdown by MAC address or IP. |
Type | boolean |
Options | true, false |
Property | batched_rows |
---|---|
Description | For instances that are expected to process a very high volume of flows, batched_rows can be used to limit the number of aggregate records processed at one time resulting in a series of one or more batches of output. If there are 1,000 aggregate rows and batched_rows is set to 100, then there would be 10 sink payloads generated and dispatched. This option can be used to reduce memory consumption and CPU time of both the local Agent and whatever application(s) are configured to receive the payload(s). Set batched_rows to zero (default) to disable and process all records into a single payload. |
Type | integer |
Default | 0 |
Property | nested |
---|---|
Description | When encoding the aggregated rows into a sink payload, two layout formats are supported: nested and flat (default). In nested mode, the keys that are used for aggregation become the nesting levels. |
Type | boolean |
Options | true, false |
Default | false |
Flat
{
"key1": "key value1",
"key2": "key value2",
"key3": "key value3",
"data1": "aggregate data1",
"data2": "aggregate data2",
"data3": "aggregate data3"
}
Nested
{
"key value1": {
"key value2": {
"key value3": {
"data1": "aggregate data1",
"data2": "aggregate data2",
"data3": "aggregate data3"
}
}
}
}
Property | format |
---|---|
Description | The aggregator processor passes structured data to sink plugins for further handling. This field determines what type of data structure to output. The plugin supports: |
Type | string |
Options | json, msgpack |
Property | compressor |
---|---|
Description | If desired, data can be compressed using a compatible library. |
Type | string |
Options | none, gzip |
Property | sinks |
---|---|
Description | An object array that determines which Netify sink plugins to send data to. |
Type | object |
Options | Depends on local configuration (see Sink Objects section below) |
Sink Objects
The Aggregator Processor's sink object list determines which sinks receive the aggregator processor's data and what type of data should be sent to it. A single aggregator processor can be configured to send to any number of sinks. In the example above, we are sending to only one, the sink-log. Let's take a closer look at the configuration.
The sink-log
object key is critical - it must match the section name of the corresponding plugin loader. The nested key inside the sink refers
to a channel as defined in the sink plugin. In this example,
the keyed sink-log
reference means (very likely) that we have the sink log plugin installed and enabled and
that we find a section named sink-log
. We should also see a channel defined in the sink-log plugin with the name stats
.
To verify this, we need to first look at the sink log loader configuration file:
$ cat /etc/netifyd/plugins.d/10-netify-sink-log.conf
# Netify Agent Log Sink Plugin Loader
# Copyright (C) 2023 eGloo Incorporated
#
# This is free software, licensed under the GNU General Public License v3.
#
##############################################################################
[sink-log]
enable = yes
plugin_library = /usr/lib64/libnetify-sink-log.so.0.0.0
conf_filename = ${path_state_persistent}/netify-sink-log.json
# vim: set ft=dosini :
You can see the sink-log
section name does match our aggregator processor sink target configuration. The fact that the configuration file exists,
is a good indication it has been installed, and the enable = yes
confirms the plugin is enabled. From this file, we can also see the
JSON configuration file of the sink log from the conf_filename
option.
Next, we look at the JSON configuration file for the sink-log plugin.
$ cat /etc/netifyd/netify-sink-log.json
{
"overwrite": false,
"log_path": "/tmp",
"channels": {
"aggregate": {
"overwrite": true,
"log_path": "/tmp",
"log_name": "netify-aggregate-log-"
}
}
}
As seen above, the aggregate
channel exists. To further gain an understanding of the sink log configuration, please
go here.
Aggregator Types
Depending on your use case, different types of aggregation can be configured.
Aggregator Type 1 - Minimalist
The Type 1 Aggregator groups flows by Application and Protocol, with an option to include the local_ip/local_mac, depending on the privacy mode attribute.
Privacy Mode Enabled (true){
"application_id": "10310.netify.ubiquiti",
"download": 0,
"packets": 1,
"protocol_id": 31,
"upload": 204
},
{
"application_id": "0.netify.unclassified",
"download": 244,
"packets": 4,
"protocol_id": 91,
"upload": 242
},
{
"application_id": "10909.netify.google-play",
"download": 694,
"packets": 10,
"protocol_id": 188,
"upload": 769
}
Privacy Mode Disabled (false)
{
"application_id": "10910.netify.google-chat",
"download": 67,
"local_ip": "192.168.1.100",
"local_mac": "00:00:00:00:00:00",
"packets": 2,
"protocol_id": 188,
"upload": 71
},
{
"application_id": "10091.netify.amazon-aws",
"download": 68,
"local_ip": "192.168.1.101",
"local_mac": "11:11:11:11:11:11",
"packets": 2,
"protocol_id": 196,
"upload": 66
},
{
"application_id": "10017.netify.outlook",
"download": 84,
"local_ip": "192.168.1.100",
"local_mac": "00:00:00:00:00:00",
"packets": 2,
"protocol_id": 188,
"upload": 81
}
Aggregator Type 2 -
The Type 2 Aggregator groups flows by Application and Protocol, Remote IP and IP Protocol, with an option to include the local_ip/local_mac, depending on the privacy mode attribute.
Privacy Mode Enabled (true){
"application_id": "10256.netify.ubuntu",
"flow_count": 1,
"ip_protocol": 6,
"local_bytes": 491,
"other_bytes": 467,
"other_ip": "185.125.190.48",
"packets": 10,
"protocol_id": 7
},
{
"application_id": "126.netify.google",
"flow_count": 1,
"ip_protocol": 17,
"local_bytes": 497,
"other_bytes": 469,
"other_ip": "142.250.176.202",
"packets": 14,
"protocol_id": 188
},
{
"application_id": "10256.netify.ubuntu",
"flow_count": 1,
"ip_protocol": 17,
"local_bytes": 292,
"other_bytes": 100,
"other_ip": "192.168.71.173",
"packets": 2,
"protocol_id": 5
}
Privacy Mode Disabled (false)
{
"application_id": "10310.netify.ubiquiti",
"flow_count": 1,
"ip_protocol": 17,
"local_bytes": 204,
"local_ip": "192.168.1.100",
"local_mac": "00:00:00:00:00:00",
"other_bytes": 0,
"other_ip": "255.255.255.255",
"packets": 1,
"protocol_id": 31
},
{
"application_id": "10017.netify.outlook",
"flow_count": 1,
"ip_protocol": 17,
"local_bytes": 3639,
"local_ip": "192.168.1.101",
"local_mac": "11:11:11:11:11:11",
"other_bytes": 2093,
"other_ip": "52.96.165.210",
"packets": 23,
"protocol_id": 188
},
{
"application_id": "11116.netify.aws-accelerator",
"flow_count": 1,
"ip_protocol": 17,
"local_bytes": 194,
"local_ip": "192.168.1.100",
"local_mac": "00:00:00:00:00:00",
"other_bytes": 113,
"other_ip": "192.168.71.173",
"packets": 2,
"protocol_id": 5
}
Aggregator Type 3 - Advanced Metadata
The Type 3 Aggregator groups provides higher order DPI data and digest inclusion that allows tracking flows.
Privacy Mode Enabled (true){
"detected_application": 10036,
"detected_application_name": "10036.netify.internal-network",
"detected_protocol": 8,
"detected_protocol_name": "MDNS",
"digests": [
"0ff27788cfb55f7e103d20a55f6bd33664dbf68d"
],
"interface": "wlp1s0",
"internal": true,
"ip_protocol": 17,
"ip_version": 4,
"local_bytes": 440,
"local_origin": true,
"other_bytes": 0,
"other_ip": "224.0.0.251",
"other_port": 5353,
"other_type": "multicast",
"packets": 1
},
{
"detected_application": 10017,
"detected_application_name": "10017.netify.outlook",
"detected_protocol": 188,
"detected_protocol_name": "QUIC",
"digests": [
"fa3c6efb48ebec5835b83f35090772118efcbf9d"
],
"interface": "wlp1s0",
"internal": true,
"ip_protocol": 17,
"ip_version": 4,
"local_bytes": 3443,
"local_origin": true,
"other_bytes": 2050,
"other_ip": "52.96.165.210",
"other_port": 443,
"other_type": "remote",
"packets": 20
}
Privacy Mode Disabled (false)
{
"detected_application": 10033,
"detected_application_name": "10033.netify.netify",
"detected_protocol": 196,
"detected_protocol_name": "HTTP/S",
"digests": [
"5dd5bb2c827c677ee3f904d40ee0b0ce512234b8"
],
"interface": "wlp1s0",
"internal": true,
"ip_protocol": 6,
"ip_version": 4,
"local_bytes": 3095,
"local_ip": "192.168.1.100",
"local_mac": "00:00:00:00:00:00",
"local_origin": true,
"other_bytes": 457,
"other_ip": "148.113.141.168",
"other_port": 443,
"other_type": "remote",
"packets": 8
},
{
"detected_application": 10910,
"detected_application_name": "10910.netify.google-chat",
"detected_protocol": 188,
"detected_protocol_name": "QUIC",
"digests": [
"e5cf8332d97ff2f5d8e4a455e5c8e02a4895d8a8"
],
"interface": "wlp1s0",
"internal": true,
"ip_protocol": 17,
"ip_version": 4,
"local_bytes": 426,
"local_ip": "192.168.1.101",
"local_mac": "11:11:1b:11:11:11",
"local_origin": true,
"other_bytes": 476,
"other_ip": "142.251.40.206",
"other_port": 443,
"other_type": "remote",
"packets": 13
}
Upload and Download Conventions
The local designation indicates the endpoint's local side, and the other designation will be indicated by the other_type field. These designations do not indicate a flow's direction. To determine which side of a flow started the conversation, consult the local_origin field. When this field is true, it indicates that the local endpoint started transmitting first. When false, the opposite endpoint started the flow.
Below is a sample of JSON to consider.
"local_origin": true
"local_ip": "192.168.4.105",
"local_bytes": 2434,
"other_type": "remote",
"other_ip": "31.13.80.53",
"other_bytes": 6139
In this example, 192.168.4.105 transmitted 2434 bytes and received 6139 bytes from 31.13.80.53. The flow originated from 192.168.4.105.
Said another way, 192.168.4.105 uploaded 2434 bytes and downloaded 6139 bytes from 31.13.80.53.
Examples
Send aggregate to the Sink Log plugin.
Send uncompressed aggregate stats data every 15 seconds to the Sink Log plugin to a channel name aggregate
.
{
"aggregator": 1,
"log_interval": 60,
"privacy_mode": false,
"format": "json",
"compressor": "gz",
"batched_rows": 0,
"nested": false,
"sinks": {
"sink-log": {
"aggregate": {
"log_interval": 15,
"format": "json",
"compressor": "none"
}
}
}
}
Send aggregate stats data to the Sink Message Queue plugin.
Send data compressed with gzip and formatted using Message Pack to the Sink Message Queue plugin to a channel name data-center-1
.
{
"aggregator": 2,
"log_interval": 60,
"privacy_mode": false,
"format": "json",
"compressor": "gz",
"batched_rows": 0,
"nested": false,
"sinks": {
"sink-mqtt": {
"data-center-1": {
"format": "msgpack",
"compressor": "gz"
}
}
}
}