Setting Up SNMPv3 Traps
We have various articles already in our documentation for setting up SNMPv2 trap handling in Opsview Monitor, but SNMPv3 traps are a whole new ballgame. They can be quite confusing and complicated to set up the first time you go through the process, but when you understand what is going on, everything should make more sense.
SNMP has gone through several revisions to improve performance and security (version 1, 2c and 3). By default, it is a UDP based protocol where communication is based on a 'fire and forget' methodology in which network packets are sent to another device, but there is no check for receipt of that packet (versus TCP when a network packet must be acknowledged by the other end of the communication link).
There are two modes of operation with SNMP - get requests (or polling) where one device requests information from an SNMP enabled device on a regular basis (normally using UDP port 161), and traps where the SNMP enabled device sends a message to another device when an event occurs (normally using UDP port 162). The latter includes instances such as someone logging on, the device powering up or down, or a wide variety of other problems that would need this type of investigation.
This blog covers SNMP v3 traps, as polling and version 2c traps are covered elsewhere in our documentation.
Since SNMP is primarily a UDP based system, traps may be 'lost' when sending between devices; the sending device does not wait to see if the receiver got the trap. This means if the configuration on the sending device is wrong (using the wrong receiver IP address or port) or the receiver isn't listening for traps or rejecting them out of hand due to misconfiguration, the sender will never know.
The SNMP v2c specification introduced the idea of splitting traps into two types; the original 'hope it gets there' trap and the newer 'INFORM' traps. Upon receipt of an INFORM, the receiver must send an acknowledgement back. If the sender doesn't get the acknowledgement back, then it knows there is an existing problem and can log it for sysadmins to find when they interrogate the device.
Older versions of SNMP relied on a single unencrypted "community string" for both get requests and traps, which made it very insecure on the network (anyone with network experience could 'snoop' on the network and detect the unencrypted community strings). The only security options with SNMP v1 and v2c are to either disable it altogether (not very helpful for monitoring purposes) or make sure SNMP enabled devices are 'read only' (SNMP can allow you to update configuration on devices) so that if the connection details were obtained by a 3rd party, they could only look at what was happening rather than change device configuration. However, even that approach can provide too much information to the wrong type of party.
Version 3 uses the same base protocol as earlier versions, but introduces encryption and much improved authentication mechanisms. Depending on how you authorize with the SNMP agent on a device, you may be granted different levels of access.
The security level you use depends on what credentials you must provide to authenticate successfully:
|noAuthNoPriv||No authentication and no encryption||Required||Not Required||Not Required||Not Required||Not Required|
|authNoPriv||Messages are authenticated but not encrypted||Required||Required||Required||Not Required||Not Required|
|authPriv||Messages are authenticated and encrypted||Required||Required||Required||Required||Required|
As can be seen above, the best security scheme is 'authPriv', but it takes more to configure a working system.
For systems to communicate, both sides must use the same authProtocol (MD5 or SHA) and privProtocol (AES or DES). Some devices do not support all of these combinations, so you must check what can be used to ensure the trap receiver is configured in the same way.
If you have multiple devices that need to use multiple schemes, you should create multiple users, one for each scheme type. Do not try to shoe-horn all devices into using one type as this makes managing the systems harder than necessary. But the fewer schemes you need to set up on the SNMP devices, the less configuration and password management you need on the receiver.
Version 3 also introduces the concept of an 'SNMP Engine ID', which is a unique identifier for each SNMP device usually expressed as a hexadecimal string such as '0x8000123acd1ab43abbfff000fa'. If your SNMP sender has an engine ID, this must be taken into account when creating the authentication schemes on the receiver (note: engine ID's are not used for INFORMs).
Setting up SNMPv3 traps on an Opsview server
The first step is to configure the trap sender with the IP address of your master server (or with each node within the monitoring cluster so all nodes in the cluster receive traps) and the available authProtocl and privProtocol schemes. Set an appropriate secName (similar to a username) and secure passwords (use a utility such as 'pwgen' on Linux). You can then set up the matching scheme on the receiver side.
To configure the SNMP Trap daemon on the receiver correctly, it must be stopped on your machine (something like either 'service snmptrapd stop' or '/etc/init.d/snmptrapd stop', depending on your OS) before you edit the configuration files.
Locate the snmptrapd.conf file (it may be '/etc/snmp/snmptrapd.conf', '/var/lib/net-snmp/snmptrapd.conf' or somewhere else, OS dependant) and add in lines such as:
createUser -e <engineID> <securityName> <authProtocol> <privProtocol> <privKey>
authUser log,exec <securityName>
createUser -e 0x8000123acd1ab43abbfff000fa opsviewv3 SHA mySecureAuthPassword AES mySecurePrivPassword
authUser log,exec opsviewv3
As noted above, if the trap is being sent as an 'INFORM', you do not need to specify the engine ID and the '-e 0x8000123acd1ab43abbfff000fa ' can be omitted.
By default, snmptrapd must be told what to do with traps received from authenticated users- this is what the authUser line does; for the specific user, log the trap has been received and passed to an executable file. Without this line, the trap is received but nothing else will be done with it, not even logged.
You should also have a line in there to redirect the traps into Opsview - this is the executable file the trap is passed on to from the authUser line.
traphandle default /usr/local/nagios/bin/snmptrap2nagios
Also, make sure you do not have a line like the below (or if you do, it is set to 'no'):
this means no authentication is required to receive the traps and you have just spent time setting up all the authentication and authorization for no reason.
You can now restart snmptrapd and test your traps.
Note: On some systems when the snmptrap daemon is started, it may remove the 'createUser' lines from the snmptrapd.conf file and convert them to 'usmUser' lines, so don’t worry if what you have configured seems to disappear.
There are many ways to troubleshoot SNMP v3 issues, but these are the main methods we recommend:
First, confirm the receiver is getting the traps sent through. You can do this using a network snooper command as root:
tcpdump -vv -A -T snmp -s 0 "(dst port 161) or (dst port 162) and (host <sender IP address>)"
This will look for all network packets sent to either port 161 or 162 on the local machine from the senders IP address. You may have an option on the sender to generate a test trap, in which case you should see some activity from the tcpdump command (use <CTRL>-C to stop the trace). If you see traffic here, then you can move onto the next step. Otherwise, check firewalls between the sender and receiver to make sure traffic can be passed through.
The second stage would be to stop snmptapd from running as a daemon and restart it in the foreground with debug logging enabled. Again, run the daemon, send the test trap, wait a few seconds, then stop the daemon using <CTRL>-C and go through the log file to see if your trap has been received and if so, what was done with it:
snmptrapd -t -m ALL -Oa -M /usr/share/snmp/mibs:/usr/local/nagios/snmp/load -p /var/run/snmptrapd.pid -f -Lo -DALL -Lf /tmp/snmptrap.out
You can also send a test trap from the receiver into itself pretending to be the sending device, which may help in your debugging process to make sure traps can be received and processed:
snmptrap -e <engineID> -v3 -u <secName> -a <authProtocl> -A <authPassword> -x <pricProtocol> -X <privPassword> -l <scheme> localhost 1 0
snmptrap -e 0x8000123acd1ab43abbfff000fa -v 3 -u opsviewv3 -a SHA -A mySecureAuthPassword -x AES -XmySecurePrivPassword -l authPriv localhost 1 0
If you need to test an INFORM instead of a standard trap, use the following:
snmpinform -v 3 -u opsviewv3 -a SHA -A mySecureAuthPassword -x AES -XmySecurePrivPassword -l authPriv localhost 1 0
SNMP is by no means simple. However, it can be very useful when it is set up properly and to do that you need some understanding of how it all fits together. Hopefully the above will provide you with what you need and feel free to read more information about SNMP traps on our Knowledge Center or via a Net-SNMP tutorial.
Also, be sure to learn more about how Opsview can help you with your SNMP monitoring needs.