Azure Monitor Agent Syslog forwarding to Sentinel Troubleshooting and Disk usage

After some issues with a deployment I wanted to describe some tips to troubleshoot this since most docs were useless. Here I am going to cover how to configure Syslog to forward logs to Azure Monitor Agent and ultimately send them to Microsoft Sentinel. Also address common troubleshooting steps, particularly how to manage disk space if Syslog logs start filling up your disk.

Configure Azure Monitor to forward to Log Analytics (Sentinel) Workspace

To configure a Linux machine to collect information via syslog and forward to Log Analytics workspace, this one-liner from Microsoft is pretty useful.

sudo wget -O Forwarder_AMA_installer.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Forwarder_AMA_installer.py&&sudo python3 Forwarder_AMA_installer.py

This will configure Azure Monitor agent on the machine and do the correct configuration on it to forward it. However we also need a DCR (Data Collection Rule) to forward the data to Sentinel. The Data Collection rule can be configured from Azure Monitor, and should be applied to the Syslog server

Here we also need to define which log categories should be collected from the syslog machine. Here is a list of recommended log sources

kern (kernel messages):
Log warning and above. These messages are important as they may indicate hardware or low-level system issues.

auth/authpriv (security/authorization):
Log info and above. These logs capture authentication events, failed login attempts, and other security-related activities, which are essential for monitoring and auditing.

daemon (system daemons):
Log notice and above. Daemon logs should be monitored for service issues and operational failures.

mail (mail system):
Log info and above, unless running a mail server, in which case consider logging debug during troubleshooting.

cron (scheduled jobs):
Log info and above. Monitoring scheduled tasks can help identify failed or delayed jobs.

syslog (syslogd internal):
Log error and above to monitor the health of the logging system itself.

ftp (FTP daemon):
Log warning and above. If FTP is in use, ensure logs capture potential issues or security problems.

local0 to local7 (local use):
For custom applications, configure notice and above to capture meaningful events without overloading with debug-level information unless troubleshooting.

log audit (log audit):
Log info and above for tracking system activity audits, especially for compliance purposes.

log alert (log alert):
Log alert and emergency to capture critical conditions that need immediate action.

If you are collecting information from services such as VMware you should also add user logs, since vCenter uses this log facility to log authentication attempts.

Checking Disk Space Usage

If your disk is filling up due to Syslog data, follow these steps to check for available space and clean up excessive logs.

1. Check Disk Space:
   Run the following command to see how much space is available on your disk df -h This will display the disk usage in a human-readable format, showing you which partitions may be running low on space. If the usage is 100% well, then we need to do some cleaning.
2. Check Azure Monitor Agent Error Logs:
   If your disk is full, the Azure Monitor Agent will not be able to forward logs properly. You can check the agent’s error log
   cat /var/opt/microsoft/azuremonitoragent/log/mdsd.err If you see messages like the one below, it indicates that the disk is full and the agent cannot write logs Error while inserting item to Local persistent store local6.info: IO error: No space left on device: While appending to file: /var/opt/microsoft/azuremonitoragent/events/local6.info/215765.sst: No space left on device

Cleaning Up Excessive Logs

To free up disk space, we can remove large or unnecessary logs that are stored locally, since our goal is to forward them to Sentinel. Start by inspecting disk usage in the /var/log directory:

1. List Disk Usage in /var/log:
   Use the following command to list the size of log files in the /var/log directory ls -l /var/log
2. Delete Unnecessary Logs:
   If you find that specific logs are consuming a lot of space, such as kernel or syslog logs, you can delete them sudo rm /var/log/kern* sudo rm /var/log/syslog*

Troubleshooting the Azure Monitor Agent

If you encounter other issues and if the disk usage is not the culprit, you can run a built-in troubleshooter to diagnose and resolve issues:

1. Run the Troubleshooter:
   Navigate to the directory where the agent is installed and run the troubleshooter cd /var/lib/waagent/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent-{version}/ama_tst/ sudo sh ama_troubleshooter.sh -A Replace {version} with the actual version number of the agent installed on your machine. This will give you an interactive CLI that will check for DCR, Communication and other issues with the agent.
2. Disable and Re-enable the Azure Monitor Agent:
   Sometimes, disabling and re-enabling the agent can resolve issues
3. /var/lib/waagent/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent-1.31.1# ./shim.sh -disable
4. /var/lib/waagent/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent-1.31.1# ./shim.sh -enable

Configuring Syslog for Forwarding to Azure Monitor and Sentinel

To configure your Syslog to stop logging locally and forward logs directly to Azure Monitor Agent (which will forward them to Sentinel), modify your Syslog configuration as follows:

1. Edit the Syslog Configuration:
   Open the Syslog configuration file, typically located at /etc/rsyslog.conf or /etc/rsyslog.d/50-default.conf sudo vi /etc/rsyslog.d/50-default.conf
2. Disable Local Log Storage:
   Comment out the lines that log to local files to prevent Syslog from storing logs locally. For example, you can comment out lines like the following:bashCopy code#*.*;auth,authpriv.none -/var/log/syslog #cron.* /var/log/cron.log #daemon.* -/var/log/daemon.log #kern.* -/var/log/kern.log #lpr.* -/var/log/lpr.log mail.*-/var/log/mail.log #user.* -/var/log/user.log This will prevent logs from being stored in the corresponding files (such as /var/log/syslog and /var/log/kern.log).

1. Restart the Syslog Service:
   After editing the configuration, restart the Syslog service to apply the changes:bashCopy codesudo systemctl restart rsyslog