Aggregator Processor

Netify Aggregator Processor Plugin

Introduction

The Aggregator Processor plugin is responsible for creating structured data objects from the internals of the Netify Agent's memory by aggregating individual flow data. This data is typically used for creating high-level dashboards or returning usage statistics from within data centers and SD-WAN aggregation points.

License

Netify Flow Actions Plugin is a proprietary plugin requiring a license. Please contact us for details.

Installation

Netify plugins are packaged in the same workflow as the agent and can usually be installed using a similar syntax that was implemented during the installation of the Netify agent. Exceptions to this rule occur when the plugin requires a Software License Agreement or if an upgrade or downgrade to a different version of the plugin is required.

Plugins are compiled against specific Netify agent versions and require ABI compatibility. Click here for more information.
Download URLs in the commands below for example only and version specific. Use a browser to navigate to the repository to find and copy the package you wish to install.

Alma Linux

Alma Linux 9

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

cd /tmp/
wget https://client:secret@download.netify.ai/5/almalinux/9/stable/x86_64/netify-proc-aggregator_1.0.43-1.os9.x86_64.rpm
apt install ./netify-proc-aggregator_1.0.43-1.os9.x86_64.rpm
Alma Linux 8

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

cd /tmp/
wget https://client:secret@download.netify.ai/5/almalinux/8/stable/x86_64/netify-proc-aggregator_1.0.43-1.os8.x86_64.rpm
yum install ./netify-proc-aggregator_1.0.43-1.os8.x86_64.rpm

Debian

Debian 13 (Bookworm)

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

cd /tmp/
wget https://client:secret@download.netify.ai/5/debian/10/amd64/netify-proc-aggregator_1.0.43-1_amd64.deb 
apt install ./netify-proc-aggregator_1.0.43-1_amd64.deb
Debian 11 (Bullseye)

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

cd /tmp/
wget https://client:secret@download.netify.ai/5/debian/10/amd64/netify-proc-aggregator_1.0.43-1_amd64.deb 
apt install ./netify-proc-aggregator_1.0.43-1_amd64.deb
Debian 10 (Buster)

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

cd /tmp/
wget https://client:secret@download.netify.ai/5/debian/10/amd64/netify-proc-aggregator_1.0.43-1_amd64.deb 
apt install ./netify-proc-aggregator_1.0.43-1_amd64.deb

OpenWRT

OpenWRT 23.05

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

pkg add https://client:secret@download.netify.ai/5/openwrt/23.05/amd64/netify-proc-aggregator_1.0.43-1_x86_64.ipk
OpenWRT 22.03

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

pkg add https://client:secret@download.netify.ai/5/openwrt/22.03/amd64/netify-proc-aggregator_1.0.43-1_x86_64.ipk
OpenWRT 21.02

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

pkg add https://client:secret@download.netify.ai/5/openwrt/21.02/amd64/netify-proc-aggregator_1.0.43-1_x86_64.ipk
OpenWRT 19.07

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

pkg add https://client:secret@download.netify.ai/5/openwrt/19.07/amd64/netify-proc-aggregator_1.0.43-1_x86_64.ipk

OPNsense

OPNsense 24.1 - Savvy Shark

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

pkg add https://client:secret@download.netify.ai/5/freebsd/13.2/amd64/netify-proc-aggregator_1.0.43,1.pkg
OPNsense 23.7 - Restless Roadrunner

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

pkg add https://client:secret@download.netify.ai/5/freebsd/13.2/amd64/netify-proc-aggregator_1.0.43,1.pkg

pfSense

pfSense Plus 23.x

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

pkg add https://client:secret@download.netify.ai/5/freebsd/14.0/amd64/netify-proc-aggregator_1.0.43,1.pkg
pfSense CE 2.7.x

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

pkg add https://client:secret@download.netify.ai/5/freebsd/13.2/amd64/netify-proc-aggregator_1.0.43,1.pkg

Rocky Linux

Rocky Linux 9

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

cd /tmp/
wget https://client:secret@download.netify.ai/5/rockylinux/9/stable/x86_64/netify-proc-aggregator_1.0.43-1.os9.x86_64.rpm
apt install ./netify-proc-aggregator_1.0.43-1.os9.x86_64.rpm
Rocky Linux 8

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

cd /tmp/
wget https://client:secret@download.netify.ai/5/rockylinux/8/stable/x86_64/netify-proc-aggregator_1.0.43-1.os8.x86_64.rpm
yum install ./netify-proc-aggregator_1.0.43-1.os8.x86_64.rpm

Ubuntu

Ubuntu 22.04 - Jammy

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

cd /tmp/
wget https://client:secret@download.netify.ai/5/ubuntu/jammy/binary-amd64/netify-proc-aggregator_1.0.43-1_amd64.deb 
apt install ./netify-proc-aggregator_1.0.43-1_amd64.deb
Ubuntu 20.04 - Focal

The Netify Aggregator Processor plugin is a licensed (proprietary) plugin. Instructions to access this plugin will have been provided to you by a Netify representative. Typically, the package will be available via a web link with Basic authentication protection. Example: 

cd /tmp/
wget https://client:secret@download.netify.ai/5/ubuntu/jammy/binary-amd64/netify-proc-aggregator_1.0.43-1_amd64.deb 
apt install ./netify-proc-aggregator_1.0.43-1_amd64.deb

Configuration

Plugin Loader Configuration

All plugins are disabled by default, and the Netify Aggregator Processor plugin is no different. To enable: 

netifyd --enable-plugin proc-aggregator
A plugin can also be disabled in this way by substituting --disable-plugin.

Alternatively, you can edit /etc/netifyd/plugins.d/10-netify-proc-aggregator.conf and set enable to yes. 

# Netify Aggregator Processor Plugin Loader
# Copyright (C) 2023 eGloo Incorporated
#
##############################################################################

[proc-aggregator]
enable = yes
plugin_library = /usr/lib64/libnetify-proc-aggregator.so.0.0.0
conf_filename = ${path_state_persistent}/netify-proc-aggregator.json

# vim: set ft=dosini :

Plugin Configuration

Once the plugin has been enabled, it can be configured using the defined JSON configuration file specified in the plugin loader configuration. Let's look at a configuration sample to review the syntax and parts of the file. 

{
    "log_interval": 60,
    "privacy_mode": false,
    "format": "json",
    "compressor": "gz",
    "batched_rows": 0,
    "nested": false,
    "sinks": {
        "sink-log": {
            "default": {
                "format": "json",
                "compressor": "gz"
            }
        }
    }
}
Property log_internal
Description Interval time (in seconds) between aggregate summary reports.
Type integer
Default 60
Property privacy_mode
Description If true, aggregation will not include a breakdown by MAC address or IP.
Type boolean
Options true, false
Property batched_rows
Description For instances that are expected to process a very high volume of flows, batched_rows can be used to limit the number of aggregate records processed at one time resulting in a series of one or more batches of output. If there are 1,000 aggregate rows and batched_rows is set to 100, then there would be 10 sink payloads generated and dispatched. This option can be used to reduce memory consumption and CPU time of both the local Agent and whatever application(s) are configured to receive the payload(s). Set batched_rows to zero (default) to disable and process all records into a single payload.
Type integer
Default 0
Property nested
Description When encoding the aggregated rows into a sink payload, two layout formats are supported: nested and flat (default). In nested mode, the keys that are used for aggregation become the nesting levels.
Type boolean
Options true, false
Default false
Flat
{
    "key1": "key value1",
    "key2": "key value2",
    "key3": "key value3",
    "data1": "aggregate data1",
    "data2": "aggregate data2",
    "data3": "aggregate data3"
}
Nested
{
    "key value1": {
        "key value2": {
            "key value3": {
                "data1": "aggregate data1",
                "data2": "aggregate data2",
                "data3": "aggregate data3"
            }
        }
    }
}
Property format
Description The aggregator processor passes structured data to sink plugins for further handling. This field determines what type of data structure to output. The plugin supports:
Type string
Options json, msgpack
Property compressor
Description If desired, data can be compressed using a compatible library.
Type string
Options none, gzip
Property sinks
Description An object array that determines which Netify sink plugins to send data to.
Type object
Options Depends on local configuration (see Sink Objects section below)
Sink Objects

The Aggregator Processor's sink object list determines which sinks receive the aggregator processor's data and what type of data should be sent to it. A single aggregator processor can be configured to send to any number of sinks. In the example above, we are sending to only one, the sink-log. Let's take a closer look at the configuration.

The sink-log object key is critical - it must match the section name of the corresponding plugin loader. The nested key inside the sink refers to a channel as defined in the sink plugin. In this example, the keyed sink-log reference means (very likely) that we have the sink log plugin installed and enabled and that we find a section named sink-log. We should also see a channel defined in the sink-log plugin with the name stats. To verify this, we need to first look at the sink log loader configuration file: 

$ cat /etc/netifyd/plugins.d/10-netify-sink-log.conf 

# Netify Agent Log Sink Plugin Loader
# Copyright (C) 2023 eGloo Incorporated
#
# This is free software, licensed under the GNU General Public License v3.
#
##############################################################################

[sink-log]
enable = yes
plugin_library = /usr/lib64/libnetify-sink-log.so.0.0.0
conf_filename = ${path_state_persistent}/netify-sink-log.json

# vim: set ft=dosini :

You can see the sink-log section name does match our aggregator processor sink target configuration. The fact that the configuration file exists, is a good indication it has been installed, and the enable = yes confirms the plugin is enabled. From this file, we can also see the JSON configuration file of the sink log from the conf_filename option.

Next, we look at the JSON configuration file for the sink-log plugin. 

$ cat /etc/netifyd/netify-sink-log.json 

    {
        "overwrite": false,
        "log_path": "/tmp",
        "channels": {
            "aggregate": {
                "overwrite": true,
                "log_path": "/tmp",
                "log_name": "netify-aggregate-log-"
            }
        }
    }

As seen above, the aggregate channel exists. To further gain an understanding of the sink log configuration, please go here.

Examples

Send aggregate to the Sink Log plugin.

Send uncompressed aggregate stats data every 15 seconds to the Sink Log plugin to a channel name aggregate. 

{
    "log_interval": 60,
    "privacy_mode": false,
    "format": "json",
    "compressor": "gz",
    "batched_rows": 0,
    "nested": false,
    "sinks": {
        "sink-log": {
            "aggregate": {
                "log_interval": 15,
                "format": "json",
                "compressor": "none"
            }
        }
    }
}
Requires the installation and correct configuration of the Netify Sink Log plugin.

Send aggregate stats data to the Sink Message Queue plugin.

Send data compressed with gzip and formatted using Message Pack to the Sink Message Queue plugin to a channel name data-center-1. 

{
    "log_interval": 60,
    "privacy_mode": false,
    "format": "json",
    "compressor": "gz",
    "batched_rows": 0,
    "nested": false,
    "sinks": {
        "sink-mqtt": {
            "data-center-1": {
                "format": "msgpack",
                "compressor": "gz"
            }
        }
    }
}
Requires the installation and correct configuration of the Netify Sink MQTT plugin.

Documentation


Netify Agent Plugins


Technical Support

Haven't found the answers you're looking for?

Contact Us