Troubleshooting

Systemd

For systems using Systemd, check the status for any helpful information. 

sudo systemctl status netifyd
○ netifyd.service - Netify Agent
     Loaded: loaded (/lib/systemd/system/netifyd.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:netifyd(8)
             man:netifyd.conf(5)
             https://netify.ai/

Oct 13 10:40:53 netifyd[6590]: sink-socket: Netify Socket Sink Plugin v1.0.42 Copyright (C) 2021-2024 eGloo Incorporated.
Oct 13 10:40:53 netifyd[6590]: netify-proc-python: License status changed from no license to valid.
Oct 13 10:40:53 netifyd[6590]: Plugin has terminated: proc-aggregator: /usr/lib/x86_64-linux-gnu/libnetify-proc-aggregator.so
Oct 13 10:40:54 netifyd[6590]: sink-mqtt: MQ connect: 192.168.71.173:1883: Connection refused [14]
Oct 13 10:40:54 netifyd[6590]: Plugin has terminated: proc-core: /usr/lib/x86_64-linux-gnu/libnetify-proc-core.so.0.0.0
Oct 13 10:40:54 netifyd[6590]: Plugin has terminated: proc-nfa: /usr/lib/x86_64-linux-gnu/libnetify-proc-flow-actions.so
Oct 13 10:40:54 netifyd[6590]: Plugin has terminated: proc-python: /usr/lib/x86_64-linux-gnu//libnetify-proc-python.so.0.0.0
Oct 13 10:40:54 netifyd[6590]: nd-instance: Exit on error: 1
Oct 13 10:40:55 netifyd[6590]: sink-mqtt: MQ connect: 192.168.71.173:1883: Connection refused [14]
Oct 13 10:40:56 systemd[1]: netifyd.service: Deactivated successfully.

Init

For systems using Init, check the status using the Netify init script. 

/etc/init.d/netifyd status

Netify Daemon Flags

Netify's command line wrapper for the daemon, netifyd, can be a valuable source of information into the inner workings, configuration and status of the agent running in the background.

Version

Use the -V flag to display version information of the agent as well as plugin status. In many cases, unexpected results are simply a result of forgetting to enable a plugin.

sudo netifyd -V
Netify Agent/5.0.55-HEAD-2613-33cec546 (debian; linux-gnu; x86_64; conntrack; netlink; dns-cache; tpv3; tcmalloc; regex)
https://www.netify.ai/

This application uses nDPI v4.9.0
https://www.ntop.org/products/deep-packet-inspection/ndpi/
https://github.com/ntop/nDPI

  This program comes with ABSOLUTELY NO WARRANTY.
  Netifyd is dual-licensed under commercial and open source licenses. The
  commercial license gives you the full rights to create and distribute software
  on your own terms without any open source license obligations.

  Netifyd is also available under GPL and LGPL open source licenses.  The open
  source licensing is ideal for student/academic purposes, hobby projects,
  internal research project, or other projects where all open source license
  obligations can be met.

Report bugs to: https://gitlab.com/netify.ai/public/netify-agent/issues

Processor plugins:

 proc-core/1.0.64
    /etc/netifyd/netify-proc-core.json
    /usr/lib/x86_64-linux-gnu/libnetify-proc-core.so.0.0.0

Sink plugins:

 sink-http/1.0.43
    /etc/netifyd/netify-sink-http.json
    /usr/lib/x86_64-linux-gnu/libnetify-sink-http.so.0.0.0
 sink-socket/1.0.42
    /etc/netifyd/netify-sink-socket.json
    /usr/lib/x86_64-linux-gnu/libnetify-sink-socket.so.0.0.0

For example, if you are expecting the Netify Flow Actions plugin to be creating an ipset and populating it based on criteria you've set and it is not performing this action, the above output indicates the Flow Actions plugin is not enabled.

Status

Use the -s flag to display agent and plugin status information.

sudo netifyd -s

Debug

Use the -d flag to run the Netify agent in the foreground (instead of as a background daemond) - detailed information will be displayed at every stage of the startup sequence and continue into the traffic analysis. 

sudo netifyd -d

To stop this process, click CTRL-c.

Process Status

Use the ps command to show any running Netify processes. Note whether the PID is changing, indicating an agent that is restarting. 

ps afxw | grep netifyd

Logs

The Netify agent logs to standard tools in use, as per the platform it was installed on. Consult the log files frequently when making initial changes to configuration files and reloading/restarting the agent to catch any errors you may have introduced. It is recommended to tail the log file, using grep to filter out unnecesssary output.

Linux Mainstream Distros

cat /var/log/syslog | grep netifyd

OpenWRT

logread | grep netifyd

Common Issues

Below are the most common issues encountered when using Netify for the first time along with actions to take to resolve.

Poor Classification

If you are observing poort application classification rates, it possible you are using the Open Source application signatures. To determine if this is the case, run: 

netifyd -s | grep apps
• apps: 1861, domains: 17776, networks: 4639, soft-dissectors: 35, transforms: 1

An apps count in the first column of less than 200 indicates the use of the open source list. Subsribe to Netify Informatics or contact our sales team to inquire about licensing the commercial application signature list.

Daemon Restarting / Instability

The default Systemd and Init scripts are configured to restart the agent if the agent stops without any user interaction. This behaviour can be useful to ensure the daemon is running at all times, however, it can cause an adminstrator to incorrectly conclude the daemon is 'crashing', when in fact, there is a valid condition that causes the agent to terminate itself. The most common causes of the agent stopping are listed below.

No Capture Interfaces

Using the command netifyd -s, check for the existance of a warning message:

✗ No capture sources (interfaces) configured.

To resolve this issue, make sure one or more interfaces are defined.

Configuration Syntax

One or more configuration files have invalid JSON. Check your syslog (logread for OpenWRT platforms) for details...for example:

grep netifyd /var/log/syslog

Oct 13 10:40:53 [netifyd]: proc-aggregator: Netify Flow Aggregator Processor Plugin v1.0.55 Copyright (C) 2021-2024 eGloo Incorporated.
Oct 13 10:40:53 [netifyd]: proc-aggregator: Exception: proc-aggregator: /etc/netifyd/netify-proc-aggregator.json: JSON exception: [json.exception.parse_error.101] parse error at line 12
..
.
Oct 13 10:40:54 [netifyd]: nd-instance: Exit on error: 1

To resolve this issue, fix any configuration file that is not valid and restart the agent.

Foreground Netifyd Process

If you're like many admins, having multiple terminals open is common pratice, especially when evaluating a piece of software. If you are seeing 'strange' behaviour, do a quick check that you don't have any other netifyd processes running. 

ps afxw | grep netifyd
8406 pts/1    S+     0:00  |       \_ grep --color=auto netifyd
8374 ?        Ssl    0:00  \_ netifyd help

To resolve this issue, kill (by pid) any netifyd processes that you don't recognize: 

kill -9 8374

Licensing Issue

If you are using Netify plugins that require a license, check that your license:

  • Has not expired
  • Has been signed by eGloo

To check the status of your license, the Netify agent must be running, then run netifyd -s.

netifyd -s
[200~Netify Agent/5.0.55-HEAD-2613-33cec546 (debian; linux-gnu; x86_64; conntrack; netlink; dns-cache; tpv3; tcmalloc; regex)
✓ agent is running: PID 8759
..
.
proc-core/1.0.64 (processor)
proc-nfa/1.0.79 (processor)
✓ license status: valid
proc-python/1.0.14 (processor)
✓ license status: valid
sink-http/1.0.43 (sink)
sink-log/1.0.41 (sink)
sink-mqtt/1.0.51 (sink)
✓ license status: valid
sink-socket/1.0.42 (sink)

To resolve any licensing issue, please contact our sales team.

Technical Support

In order to help us help you, please include the following information with any support ticket:

  • Operating system and version of the host (ex. Debian 12)
  • Host environment (ex. Bare Metal [AWS EC2, x86-64, ARM64 etc.], Virtual Machine [KVM, VirtualBox etc.] or Container [Docker, LXC etc.])
  • Output from netifyd -V
  • Output from netifyd -s (while netifyd is running)
  • Output from any log, systemd or init output that appears relevant
  • Attach any relevant JSON configuration file (ex. if you are having an issue with socket output, include /etc/netifyd/netify-sink-socket.json as well as the configuration file from any processory sending data to the sink)
  • Your Netify Informatics account (if relevant)

To open a support ticket, please email support@netify.ai.

Further Reading

Documentation


Netify DPI Agent


Technical Support

Haven't found the answers you're looking for?

Contact Us