Aggregator Processor

Netify Aggregator Processor Plugin

Introduction

The Aggregator Processor plugin is responsible for creating structured data objects from the internals of the Netify Agent's memory by aggregating individual flow data. This data is typically used for creating high-level dashboards or returning usage statistics from within data centers and SD-WAN aggregation points.

License

Netify Flow Actions Plugin is a proprietary plugin requiring a license. Please contact us for details.

Installation

Netify plugins are packaged in the same workflow as the agent and can be installed using a similar syntax that was implemented during the installation of the Netify agent. Plugins are compiled and made available for x86 on mirrors. For other architectures like ARM and MIPS, please contact us

Select your installation target for specific instructions on how to install this plugin.

Plugins are compiled against specific Netify agent versions and require ABI compatibility. Click here for more information.
Netify mirros can be found here.

Configuration

Plugin Loader Configuration

All plugins are disabled by default, and the Netify Aggregator Processor plugin is no different. To enable: 

netifyd --enable-plugin proc-aggregator
A plugin can also be disabled in this way by substituting --disable-plugin.

Alternatively, you can edit /etc/netifyd/plugins.d/10-netify-proc-aggregator.conf and set enable to yes. 

# Netify Aggregator Processor Plugin Loader
# Copyright (C) 2023 eGloo Incorporated
#
##############################################################################

[proc-aggregator]
enable = yes
plugin_library = /usr/lib64/libnetify-proc-aggregator.so.0.0.0
conf_filename = ${path_state_persistent}/netify-proc-aggregator.json

# vim: set ft=dosini :

Plugin Configuration

Once the plugin has been enabled, it can be configured using the defined JSON configuration file specified in the plugin loader configuration. Let's look at a configuration sample to review the syntax and parts of the file. 

{
    "aggregator": 1,
    "log_interval": 60,
    "privacy_mode": false,
    "format": "json",
    "compressor": "gz",
    "batched_rows": 0,
    "nested": false,
    "sinks": {
        "sink-log": {
            "default": {
                "format": "json",
                "compressor": "gz"
            }
        }
    }
}
Property aggregator
Description Aggregator format type - valid options are 1, 2 or 3 - see the types section for more information.
Type integer
Default 1
Property log_internal
Description Interval time (in seconds) between aggregate summary reports.
Type integer
Default 60
Property privacy_mode
Description If true, aggregation will not include a breakdown by MAC address or IP.
Type boolean
Options true, false
Property batched_rows
Description For instances that are expected to process a very high volume of flows, batched_rows can be used to limit the number of aggregate records processed at one time resulting in a series of one or more batches of output. If there are 1,000 aggregate rows and batched_rows is set to 100, then there would be 10 sink payloads generated and dispatched. This option can be used to reduce memory consumption and CPU time of both the local Agent and whatever application(s) are configured to receive the payload(s). Set batched_rows to zero (default) to disable and process all records into a single payload.
Type integer
Default 0
Property nested
Description When encoding the aggregated rows into a sink payload, two layout formats are supported: nested and flat (default). In nested mode, the keys that are used for aggregation become the nesting levels.
Type boolean
Options true, false
Default false
Flat
{
    "key1": "key value1",
    "key2": "key value2",
    "key3": "key value3",
    "data1": "aggregate data1",
    "data2": "aggregate data2",
    "data3": "aggregate data3"
}
Nested
{
    "key value1": {
        "key value2": {
            "key value3": {
                "data1": "aggregate data1",
                "data2": "aggregate data2",
                "data3": "aggregate data3"
            }
        }
    }
}
Property format
Description The aggregator processor passes structured data to sink plugins for further handling. This field determines what type of data structure to output. The plugin supports:
Type string
Options json, msgpack
Property compressor
Description If desired, data can be compressed using a compatible library.
Type string
Options none, gzip
Property sinks
Description An object array that determines which Netify sink plugins to send data to.
Type object
Options Depends on local configuration (see Sink Objects section below)
Sink Objects

The Aggregator Processor's sink object list determines which sinks receive the aggregator processor's data and what type of data should be sent to it. A single aggregator processor can be configured to send to any number of sinks. In the example above, we are sending to only one, the sink-log. Let's take a closer look at the configuration.

The sink-log object key is critical - it must match the section name of the corresponding plugin loader. The nested key inside the sink refers to a channel as defined in the sink plugin. In this example, the keyed sink-log reference means (very likely) that we have the sink log plugin installed and enabled and that we find a section named sink-log. We should also see a channel defined in the sink-log plugin with the name stats. To verify this, we need to first look at the sink log loader configuration file: 

$ cat /etc/netifyd/plugins.d/10-netify-sink-log.conf 

# Netify Agent Log Sink Plugin Loader
# Copyright (C) 2023 eGloo Incorporated
#
# This is free software, licensed under the GNU General Public License v3.
#
##############################################################################

[sink-log]
enable = yes
plugin_library = /usr/lib64/libnetify-sink-log.so.0.0.0
conf_filename = ${path_state_persistent}/netify-sink-log.json

# vim: set ft=dosini :

You can see the sink-log section name does match our aggregator processor sink target configuration. The fact that the configuration file exists, is a good indication it has been installed, and the enable = yes confirms the plugin is enabled. From this file, we can also see the JSON configuration file of the sink log from the conf_filename option.

Next, we look at the JSON configuration file for the sink-log plugin. 

$ cat /etc/netifyd/netify-sink-log.json 

    {
        "overwrite": false,
        "log_path": "/tmp",
        "channels": {
            "aggregate": {
                "overwrite": true,
                "log_path": "/tmp",
                "log_name": "netify-aggregate-log-"
            }
        }
    }

As seen above, the aggregate channel exists. To further gain an understanding of the sink log configuration, please go here.

Aggregator Types

Depending on your use case, different types of aggregation can be configured.

Aggregator Type 1 - Minimalist

The Type 1 Aggregator groups flows by Application and Protocol, with an option to include the local_ip/local_mac, depending on the privacy mode attribute.

Privacy Mode Enabled (true) 
{
    "application_id": "10310.netify.ubiquiti",
    "download": 0,
    "packets": 1,
    "protocol_id": 31,
    "upload": 204
},
{
    "application_id": "0.netify.unclassified",
    "download": 244,
    "packets": 4,
    "protocol_id": 91,
    "upload": 242
},
{
    "application_id": "10909.netify.google-play",
    "download": 694,
    "packets": 10,
    "protocol_id": 188,
    "upload": 769
}
Privacy Mode Disabled (false) 
{
    "application_id": "10910.netify.google-chat",
    "download": 67,
    "local_ip": "192.168.1.100",
    "local_mac": "00:00:00:00:00:00",
    "packets": 2,
    "protocol_id": 188,
    "upload": 71
},
{
    "application_id": "10091.netify.amazon-aws",
    "download": 68,
    "local_ip": "192.168.1.101",
    "local_mac": "11:11:11:11:11:11",
    "packets": 2,
    "protocol_id": 196,
    "upload": 66
},
{
    "application_id": "10017.netify.outlook",
    "download": 84,
    "local_ip": "192.168.1.100",
    "local_mac": "00:00:00:00:00:00",
    "packets": 2,
    "protocol_id": 188,
    "upload": 81
}

Aggregator Type 2 -

The Type 2 Aggregator groups flows by Application and Protocol, Remote IP and IP Protocol, with an option to include the local_ip/local_mac, depending on the privacy mode attribute.

Privacy Mode Enabled (true) 
{
    "application_id": "10256.netify.ubuntu",
    "flow_count": 1,
    "ip_protocol": 6,
    "local_bytes": 491,
    "other_bytes": 467,
    "other_ip": "185.125.190.48",
    "packets": 10,
    "protocol_id": 7
},
{
    "application_id": "126.netify.google",
    "flow_count": 1,
    "ip_protocol": 17,
    "local_bytes": 497,
    "other_bytes": 469,
    "other_ip": "142.250.176.202",
    "packets": 14,
    "protocol_id": 188
},
{
    "application_id": "10256.netify.ubuntu",
    "flow_count": 1,
    "ip_protocol": 17,
    "local_bytes": 292,
    "other_bytes": 100,
    "other_ip": "192.168.71.173",
    "packets": 2,
    "protocol_id": 5
}
Privacy Mode Disabled (false) 
{
    "application_id": "10310.netify.ubiquiti",
    "flow_count": 1,
    "ip_protocol": 17,
    "local_bytes": 204,
    "local_ip": "192.168.1.100",
    "local_mac": "00:00:00:00:00:00",
    "other_bytes": 0,
    "other_ip": "255.255.255.255",
    "packets": 1,
    "protocol_id": 31
},
{
    "application_id": "10017.netify.outlook",
    "flow_count": 1,
    "ip_protocol": 17,
    "local_bytes": 3639,
    "local_ip": "192.168.1.101",
    "local_mac": "11:11:11:11:11:11",
    "other_bytes": 2093,
    "other_ip": "52.96.165.210",
    "packets": 23,
    "protocol_id": 188
},
{
    "application_id": "11116.netify.aws-accelerator",
    "flow_count": 1,
    "ip_protocol": 17,
    "local_bytes": 194,
    "local_ip": "192.168.1.100",
    "local_mac": "00:00:00:00:00:00",
    "other_bytes": 113,
    "other_ip": "192.168.71.173",
    "packets": 2,
    "protocol_id": 5
}

Aggregator Type 3 - Advanced Metadata

The Type 3 Aggregator groups provides higher order DPI data and digest inclusion that allows tracking flows.

Privacy Mode Enabled (true) 
{
    "detected_application": 10036,
    "detected_application_name": "10036.netify.internal-network",
    "detected_protocol": 8,
    "detected_protocol_name": "MDNS",
    "digests": [
        "0ff27788cfb55f7e103d20a55f6bd33664dbf68d"
    ],
    "interface": "wlp1s0",
    "internal": true,
    "ip_protocol": 17,
    "ip_version": 4,
    "local_bytes": 440,
    "local_origin": true,
    "other_bytes": 0,
    "other_ip": "224.0.0.251",
    "other_port": 5353,
    "other_type": "multicast",
    "packets": 1
},
{
    "detected_application": 10017,
    "detected_application_name": "10017.netify.outlook",
    "detected_protocol": 188,
    "detected_protocol_name": "QUIC",
    "digests": [
        "fa3c6efb48ebec5835b83f35090772118efcbf9d"
    ],
    "interface": "wlp1s0",
    "internal": true,
    "ip_protocol": 17,
    "ip_version": 4,
    "local_bytes": 3443,
    "local_origin": true,
    "other_bytes": 2050,
    "other_ip": "52.96.165.210",
    "other_port": 443,
    "other_type": "remote",
    "packets": 20
}
Privacy Mode Disabled (false) 
{
    "detected_application": 10033,
    "detected_application_name": "10033.netify.netify",
    "detected_protocol": 196,
    "detected_protocol_name": "HTTP/S",
    "digests": [
        "5dd5bb2c827c677ee3f904d40ee0b0ce512234b8"
    ],
    "interface": "wlp1s0",
    "internal": true,
    "ip_protocol": 6,
    "ip_version": 4,
    "local_bytes": 3095,
    "local_ip": "192.168.1.100",
    "local_mac": "00:00:00:00:00:00",
    "local_origin": true,
    "other_bytes": 457,
    "other_ip": "148.113.141.168",
    "other_port": 443,
    "other_type": "remote",
    "packets": 8
},
{
    "detected_application": 10910,
    "detected_application_name": "10910.netify.google-chat",
    "detected_protocol": 188,
    "detected_protocol_name": "QUIC",
    "digests": [
        "e5cf8332d97ff2f5d8e4a455e5c8e02a4895d8a8"
    ],
    "interface": "wlp1s0",
    "internal": true,
    "ip_protocol": 17,
    "ip_version": 4,
    "local_bytes": 426,
    "local_ip": "192.168.1.101",
    "local_mac": "11:11:1b:11:11:11",
    "local_origin": true,
    "other_bytes": 476,
    "other_ip": "142.251.40.206",
    "other_port": 443,
    "other_type": "remote",
    "packets": 13
}

Aggregator Type 4 - Client Custom

The Type 4 Aggregator is a Netify client/customer requested format.

This feature is available in Netify Aggregator version 1.0.88 and above.
{
  "log_time_end": 1745002614,
  "log_time_start": 1745002604,
  "stats": [
    {
      "application_id": 10909,
      "download": 935,
      "ip_protocol": 17,
      "ip_version": 4,
      "other_ip": "142.250.80.14",
      "other_port": 443,
      "packets": 5,
      "protocol_id": 188,
      "upload": 748,
      "vlan_id": 0
    },
    {
      "application_id": 0,
      "download": 66,
      "ip_protocol": 6,
      "ip_version": 4,
      "other_ip": "104.18.43.204",
      "other_port": 443,
      "packets": 2,
      "protocol_id": 196,
      "upload": 66,
      "vlan_id": 0
    }
}

Aggregator Type 5 - High Level Stats

The Type 5 Aggregator tracks interface stats - the same type of stats that can be found written to /var/run/netifyd/status.json. The benefit of having this data available from the aggregator plugin is two-fold:

  • ability to direct ouput to any Netify Sink plugin (MQTT, HTTP, Socket, Log etc.)
  • ability to set any interval time

This latter ability can be useful when performing diagnostics or troubleshooting bandwidth-related incidents. Set to an interval of your choosing, you can easily obtain high level bandwidth metrics without having to parse any data or perform any calculations.

This feature is available in Netify Aggregator version 1.0.88 and above.
{
  "log_time_end": 1745002518,
  "log_time_start": 1745002508,
  "stats": [
    {
      "capture_dropped": 0,
      "capture_filtered": 0,
      "discarded": 11,
      "discarded_bytes": 552,
      "ethernet": 83084,
      "flow_dropped": 0,
      "fragmented": 0,
      "icmp": 1,
      "ifname": "wlp1s0",
      "igmp": 0,
      "ip": 83073,
      "ip_bytes": 153700217,
      "largest_bytes": 26130,
      "local_bytes": 151480246,
      "local_packets": 53358,
      "mpls": 0,
      "other_bytes": 2227891,
      "other_packets": 29738,
      "pppoe": 0,
      "queue_dropped": 0,
      "raw": 83084,
      "tcp": 83044,
      "tcp_resets": 2,
      "tcp_seq_errors": 0,
      "udp": 28,
      "vlan": 0,
      "wire_bytes": 155693969
    }
  ]
}

Attribute Definitions

Property capture_dropped
Description Packets dropped by the capture plane - agent couldn't service request fast enough. If you're using PCAP, need to add CPU's if possible. For TPACKETtv3, try increasing the ring buffer size.
Type integer
Property capture_filtered
Description Packets filtered out (removed) in the case where eBPF are in use.
Type integer
Property discarded
Description Packet counts being discarded prior to entering the DPI engine for analysis. An example is an unsupported protocol like NetBEUI
Type integer
Property discarded_bytes
Description Byte count associated with the discarded packets as described above.
Type integer
Property ethernet
Description Total number of supported Ethernet header counts seen.
Type integer
Property flow_dropped
Description Total number of new flows dropped because of a system configuration setting (unlimited by default). Setting max_flows in /etc/netifyd.conf to a non-zero value will limit the number of flows the agent tracks, thereby restrictiong CPU and memory in the event of traffic patterns that the hardware may not be able to support.
Type integer
Property icmp
Description Total number of ICMP packets.
Type integer
Property ifname
Description Interface name.
Type string
Property igmp
Description Total number of IGMP packets.
Type integer
Property ip
Description Total number of IPv4 or IPv6 packets.
Type integer
Property ip_bytes
Description Total number of IPv4 or IPv6 bytes.
Type integer
Property largest_bytes
Description Largest frame size, in bytes.
Type integer
Property local_bytes
Description Total transmitted (upload) bytes.
Type integer
Property local_packets
Description Total transmitted (upload) packets.
Type integer
Property mpls
Description Total number of MPLS packets.
Type integer
Property other_bytes
Description Total received (download) bytes.
Type integer
Property other_packets
Description Total received (download) packets.
Type integer
Property pppoe
Description Total number of PPPoE packets.
Type integer
Property raw
Description Total packet count on the interface.
Type integer
Property tcp
Description Total number of TCP packets.
Type integer
Property tcp_resets
Description Total number of TCP reset (RST) packets.
Type integer
Property tcp_seq_errors
Description Total number of TCP sequence errors.
Type integer
Property udp
Description Total number of UDP packets.
Type integer
Property vlan
Description Total number of VLAN tagged packets.
Type integer
Property wire_bytes
Description Total number bytes (upload and download) after discarded packets have applied.
Type integer

Upload and Download Conventions

The local designation indicates the endpoint's local side, and the other designation will be indicated by the other_type field. These designations do not indicate a flow's direction. To determine which side of a flow started the conversation, consult the local_origin field. When this field is true, it indicates that the local endpoint started transmitting first. When false, the opposite endpoint started the flow.

Below is a sample of JSON to consider. 

"local_origin": true
"local_ip": "192.168.4.105",
"local_bytes": 2434,
"other_type": "remote",
"other_ip": "31.13.80.53",
"other_bytes": 6139

In this example, 192.168.4.105 transmitted 2434 bytes and received 6139 bytes from 31.13.80.53. The flow originated from 192.168.4.105.

Said another way, 192.168.4.105 uploaded 2434 bytes and downloaded 6139 bytes from 31.13.80.53.

Examples

Send aggregate to the Sink Log plugin.

Send uncompressed aggregate stats data every 15 seconds to the Sink Log plugin to a channel name aggregate. 

{
    "aggregator": 1,
    "log_interval": 60,
    "privacy_mode": false,
    "format": "json",
    "compressor": "gz",
    "batched_rows": 0,
    "nested": false,
    "sinks": {
        "sink-log": {
            "aggregate": {
                "log_interval": 15,
                "format": "json",
                "compressor": "none"
            }
        }
    }
}
Requires the installation and correct configuration of the Netify Sink Log plugin.

Send aggregate stats data to the Sink Message Queue plugin.

Send data compressed with gzip and formatted using Message Pack to the Sink Message Queue plugin to a channel name data-center-1. 

{
    "aggregator": 2,
    "log_interval": 60,
    "privacy_mode": false,
    "format": "json",
    "compressor": "gz",
    "batched_rows": 0,
    "nested": false,
    "sinks": {
        "sink-mqtt": {
            "data-center-1": {
                "format": "msgpack",
                "compressor": "gz"
            }
        }
    }
}
Requires the installation and correct configuration of the Netify Sink MQTT plugin.

Documentation


Netify Agent Plugins


Technical Support

Haven't found the answers you're looking for?

Contact Us