Am I Hacked? Run An Intrusion Audit

Am I Hacked? Run An Intrusion Audit

What is an audit?

An audit, in IT terms, is an evaluation and an exhaustive examination of services, health, processes, connections, applications, practices for optimization and possible vulnerabilities in the server. It provides the best information to take action and patch up insecurities that are funded in this examination. This audit involves gathering information about attacks that arrive over the TCP/IP network.

Once we’ve known what an audit is, we are going to recognize the importance of making it one of our servers. This is because computer security has become quite important in every organization due to the high quality and correct functionally required by users or customers in all current web services.

Intrusion Audit:

It’s a process that verifies if certain standards are being accomplished and it detects those areas that need remediation or improvement. A security audit involves a systematic and a measurable technical assessment of how an organization’s security policy is employed at a specific site or location to find information that drives to the suspect infiltration, suspect user, injected bad code, and system infection.

First of all, in an intrusion Audit, we need to identify technical vulnerabilities in computers and networks, as well as weaknesses in policies and practices related to the operation of these systems, read logs, suspicious services and processes.

How to know if my site has been compromised by an attacker? (or “if my server has been hacked?”).

A system that has been compromised (“or hacked”) is not always notable, it usually runs in an invisible way, but we can recognize it by the following clues:

  • Multiple Code files seem modified.
  • Deployed several uncommon Programs, services in the server such as game servers, bots services, IRC services.
  • Server with high CPU spikes.
  • Spam is sent from one or several users.
  • Files are modified or deleted.
  • Databases are modified or removed.
  • Verify received emails in the root inbox with mutt or pine mail.
  • A lot of bandwidth is consumed.
  • Pages take a lot of time to load.
  • Plugins and modules are added to some CMS.
  • Your site is blacklisted by Google, Yahoo or Bing.
  • Users created by an attacker execute suspicious commands with root access.
  • Attackers can take control of mail accounts, databases, WordPress, Joomla, Drupal or any other account.

How to detect an intruder?

Some common practices to start the Forensics analysis are to review uncommon processes, review how many users is the site running, review uncommon Unix users, Review docroot and /tmp folder to upload folder from the application and look for suspicious files to analyze permissions. Additionally, you should check the emails received in the root inbox, to find the intruder. For this you must know how your actual Linux system works, which services are running, the structure of the application code and a good sense of your IT system architecture. Keeping this in mind and looking for discrepancies, uncommon activities will lead you to the root cause, vulnerability and intruder.

One thing we have to take in consideration is that intrusions are not always in the same way; Almost every attack is executed by using different methods. The way to remove an intruder and its tools may vary depending on the case.

Check users on Linux server with this command:

$ cut -d: -f1 /etc/passwd

This command will list your server users*.

*If there is someone that isn’t a system user like httpd, MySQL, Nginx, bin, mail, www-data, Syslog, man, etc. If there is another name that you do not recognize, you can remove it by typing the command. Make sure that the user you are removing isn’t really a system user.

$ sudo userdel user-that-you-find

To check processes, sockets and ports use a tool called Netstat and lsof. To see sockets and ports in actual, use:

$ netstat -ant

Same as previous, except that it gives you more details like PID and Program name:

$ netstat -plant

To see sockets and ports in actual use every 5 seconds:

$ netstat -n -o 5

To see what program and PID, are using sockets, open ports, files, scripts, etc. use:

$ lsof -i

If you noticed some “suspicious” processes, then you can see what files are executing with the next command:

$ lsof -p Program-PID

Let’s go to one example of a server that was compromised

The site was infected with injected code PHP and was sending Spam. For this reason, the email domain was banned and blacklisted by Google, Bing, and Yahoo.

First I needed to check which files, script, sockets and ports were being used. I used “lsof” and then filtered the results with “grep” command:

$ lsof -Pni |grep -vE ':80|:22|named|dovecot|sendm|minis|mysql|pure'
COMMAND     PID             USER   FD   TYPE   DEVICE SIZE/OFF NODE NAMEphp        1525            gozer    4u  IPv4 10972645      0t0  TCP 50.97.123.10:55993->23.21.148.239:443 (CLOSE_WAIT)sshd       1746             root    3u  IPv4    10011      0t0  TCP *:30001 (LISTEN)sshd       1746             root    4u  IPv6    10013      0t0  TCP *:30001 (LISTEN)/usr/loca  1987             root    5u  IPv4    10627      0t0  TCP 127.0.0.1:783 (LISTEN)spamd      1998             root    5u  IPv4    10627      0t0  TCP 127.0.0.1:783 (LISTEN)httpd      2048             root    5u  IPv4    10869      0t0  TCP *:443 (LISTEN)httpd      2048             root    6u  IPv6    10871      0t0  TCP *:443 (LISTEN)cpsrvd     2268             root    4u  IPv4    14295      0t0  TCP *:2087 (LISTEN)cpsrvd     2268             root    6u  IPv4    14298      0t0  TCP *:2086 (LISTEN)cpsrvd     2268             root    8u  IPv4    14301      0t0  TCP *:2095 (LISTEN)cpsrvd     2268             root    9u  IPv4    14302      0t0  TCP *:2082 (LISTEN)cpsrvd     2268             root   10u  IPv4    14303      0t0  TCP *:2083 (LISTEN)cpsrvd     2268             root   11u  IPv4    14304      0t0  TCP *:2096 (LISTEN)cpdavd     2357             root    7u  IPv4    15073      0t0  TCP *:2078 (LISTEN)cpdavd     2357             root    8u  IPv4    15074      0t0  TCP *:2080 (LISTEN)cpdavd     2357             root    9u  IPv4    15075      0t0  TCP *:2077 (LISTEN)cpdavd     2357             root   10u  IPv4    15076      0t0  TCP *:2079 (LISTEN)sshd       5769             root    3r  IPv4 69662948      0t0  TCP 50.97.123.10:30001->189.219.232.49:52994 (ESTABLISHED)perl      12432 wherechristispre   20u  IPv4 17614303      0t0  TCP 50.97.123.10:39331->194.54.81.162:43158 (CLOSE_WAIT)perl      12432 wherechristispre   22u  IPv4 19126496      0t0  TCP 50.97.123.10:39331->194.54.81.162:40990 (CLOSE_WAIT)perl      12432 wherechristispre   24u  IPv4 19269835      0t0  TCP 50.97.123.10:39331->194.54.81.162:60506 (CLOSE_WAIT)perl      12432 wherechristispre   26u  IPv4 21012134      0t0  TCP 50.97.123.10:39331->194.54.81.162:54509 (CLOSE_WAIT)perl      12432 wherechristispre   28u  IPv4 21244803      0t0  TCP 50.97.123.10:39331->194.54.81.162:53758 (CLOSE_WAIT)perl      12432 wherechristispre   30u  IPv4 21288232      0t0  TCP 50.97.123.10:39331->194.54.81.162:34472 (CLOSE_WAIT)perl      12432 wherechristispre   32u  IPv4 22234865      0t0  TCP 50.97.123.10:39331->194.54.81.162:59834 (CLOSE_WAIT)perl      12432 wherechristispre   34u  IPv4 22791575      0t0  TCP 50.97.123.10:39331->194.54.81.162:54283 (CLOSE_WAIT)

As we can see, PHP and PERL scripts were being executed, so I needed to check if an IP address connected to the server was on the blacklist with an online tool called “mxtoolbox” (http://mxtoolbox.com).

With “lsof” I got IP 23.21.148.239 and IP 189.219.232.49, so I checked it on mxtoolbox.

ip

I noticed that the IP was on the blacklist and it was executing a PHP script, so I had to see which documents were being executed.

$  lsof -p 1525
COMMAND  PID  USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAMEphp     1525 gozer  cwd    DIR              202,2     4096    92370 /home/gozer/public_htmlphp     1525 gozer  rtd    DIR              202,2     4096        2 /php     1525 gozer  txt    REG              202,2 27664289  2772504 /usr/bin/phpphp     1525 gozer  mem    REG              202,2  1965856  2771095 /usr/lib64/libcrypto.so.1.0.1ephp     1525 gozer  mem    REG              202,2   444184  2772290 /usr/lib64/libssl.so.1.0.1ephp     1525 gozer  mem    REG              202,2  7707967  2771651 /usr/lib64/libmysqlclient.so.18.0.0php     1525 gozer  mem    REG              202,2   157960  2771757 /usr/lib64/libpng12.so.0.49.0php     1525 gozer  mem    REG              202,2   157072  4112697 /lib64/ld-2.12.sophp     1525 gozer  mem    REG              202,2  1926520  4112698 /lib64/libc-2.12.sophp     1525 gozer  mem    REG              202,2   145896  4112641 /lib64/libpthread-2.12.sophp     1525 gozer  mem    REG              202,2    22536  4112402 /lib64/libdl-2.12.sophp     1525 gozer  mem    REG              202,2    47112  4112817 /lib64/librt-2.12.sophp     1525 gozer  mem    REG              202,2   599392  4112830 /lib64/libm-2.12.sophp     1525 gozer    5r   DIR              202,2     4096   181107 /home/gozer/public_html/wp-content/plugins/supermenu/admin-deprecated/redux-framework/sample/patterns

In this case, we can see that the plugin “redux-framework” was being executed by the IP blacklisted, so it was very likely that the Plugin was infected. The best option was to remove and install it again, and not allowing it to reconnect to the server with an iptables rule.

$ iptables -A INPUT -s 23.21.148.239 -j DROP

Once I discovered this, I had to know what file was running with the next command where “1525” is the program PID.

$ cat /proc/1525/environFILENAME=/home/gozer/public_html/xmlrpc.phpSCRIPT_NAME=/xmlrpc.php

“xmlrpc” is very vulnerable and has to be continuously upgraded. It must have a good configuration to protect the .htaccess file with the next configuration on the end line:

<Files "xmlrpc.php">Order Allow,Denydeny from all

I recommend always to have the ultimate version.
Now, I had to see what was happening with Perl scripts, using lsof again.

$ lsof -p 16786
COMMAND   PID             USER   FD   TYPE   DEVICE SIZE/OFF     NODE NAMEperl    16786 wherechristispre  cwd    DIR    202,2     4096        2 /perl    16786 wherechristispre  rtd    DIR    202,2     4096        2 /perl    16786 wherechristispre  txt    REG    202,2    13304  2770784 /usr/bin/perlperl    16786 wherechristispre  mem    REG    202,2  1488544  2794989 /usr/lib64/perl5/CORE/libperl.soperl    16786 wherechristispre  mem    REG    202,2   157072  4112697 /lib64/ld-2.12.soperl    16786 wherechristispre  mem    REG    202,2  1926520  4112698 /lib64/libc-2.12.soperl    16786 wherechristispre  mem    REG    202,2   145896  4112641 /lib64/libpthread-2.12.soperl    16786 wherechristispre  mem    REG    202,2    22536  4112402 /lib64/libdl-2.12.soperl    16786 wherechristispre  mem    REG    202,2    17520  4112783 /lib64/libutil-2.12.soperl    16786 wherechristispre  mem    REG    202,2   599392  4112830 /lib64/libm-2.12.soperl    16786 wherechristispre  mem    REG    202,2   113952  4112885 /lib64/libresolv-2.12.soperl    16786 wherechristispre  mem    REG    202,2    12776  4112745 /lib64/libfreebl3.soperl    16786 wherechristispre  mem    REG    202,2   116368  4112903 /lib64/libnsl-2.12.soperl    16786 wherechristispre  mem    REG    202,2    43392  4112749 /lib64/libcrypt-2.12.soperl    16786 wherechristispre  mem    REG    202,2    21056  2801666 /usr/lib64/perl5/auto/File/Glob/Glob.soperl    16786 wherechristispre  mem    REG    202,2   120008  2801701 /usr/lib64/perl5/auto/POSIX/POSIX.soperl    16786 wherechristispre  mem    REG    202,2    17976  2803461 /usr/lib64/perl5/auto/Fcntl/Fcntl.soperl    16786 wherechristispre  mem    REG    202,2    25624  2801884 /usr/lib64/perl5/auto/Socket/Socket.soperl    16786 wherechristispre  mem    REG    202,2    19336  2801685 /usr/lib64/perl5/auto/IO/IO.soperl    16786 wherechristispre  mem    REG    202,2 99154480  2772010 /usr/lib/locale/locale-archiveperl    16786 wherechristispre    0r   CHR      1,3      0t0     3800 /dev/nullperl    16786 wherechristispre    1w   CHR      1,3      0t0     3800 /dev/nullperl    16786 wherechristispre    2w   CHR      1,3      0t0     3800 /dev/nullperl    16786 wherechristispre    3u  IPv4 69906812      0t0      TCP *:39331 (LISTEN)

So I was able to see the modules, sockets, etc. You could do Forensic Analysis (TCT, The Coroner’s Toolkit) and see if Perl was replaced or if it is a rootkit doing a specifically scan in the found path with ClamAV or Sophos. Another option is re-installing Perl.

To get more information I executed a nmap scan with the next IP 194.54.81.162 using nmap tool.

$ nmap 194.54.81.162
Starting Nmap 6.47 ( http://nmap.org ) at 2015-12-27 20:33 CST
Nmap scan report for 162.81.54.194.static.server.ua (194.54.81.162)
Host is up (0.19s latency).
Not shown: 993 closed ports
PORT     STATE    SERVICE
21/tcp   open     ftp
25/tcp   filtered smtp
53/tcp   open     domain
80/tcp   open     http
88/tcp   open     kerberos-sec
5222/tcp open     xmpp-client
5269/tcp open     xmpp-server

As you can see, this scan provided me enough information about this IP. The IP had a PTR record (reverse DNS) which concludes, it was a server with domain “UA” that is the Internet country code for Ukraine. I didn’t need this connection, so it was necessary to block this IP address, with the next command:

$ iptables -A INPUT -s 194.54.81.162 -j DROP

To save IPTABLES rules use:

$ iptables-save

Now, I had to do a scan check manually on the document root, looking for suspicious code in PHP files. I had to find PHP scripts that used “eval (“ or “base64” functions. This is not always malware, but with the names and with the type of scripts I was able to identify the ones that were actually malware. You can use an online tool called VirusTotal.com to check the signatures in the code and know if it is malware. However, I searched “manually” to find eval values:

$ grep -lir “eval (”

And got this results:

Sin títuloññ

Then, I opened every suspicious file. I opened this one with vim:

$ vim wp-content/themes/betheme/functions/img/items/dir.php.suspected.
Sin título

And I discovered the first Malware. In this case, I had to delete the file using the command:

$ rm -f wp-content/themes/betheme/functions/img/items/dir.php.suspected

All the files that had a similar structure had to be deleted with the RM command.

Then, I had to find malicious signatures in the document root, so I have “malware signatures” saved for this cases. I checked all of them with grep command.

Next signature was contained by two PHP files:

$ grep "setTineout(1e); var defau1t_keyword = encodeURIConponent(document.tit1e); var se_referrer =
eQFodeURIConponent(document.referrer); var host =encodeURIConponent(window.location.host)" -lir

Here the results:

wp-content/themes/betheme/header.php
wp-content/themes/twentyfifteen/header.php

I had to open the file and find the string, in this case, I used Vim to open the result files and localize the malicious script.

Sin título1

I commented the script to ignore this line of code. Then I reviewed whether the page was loading properly, sometimes scripts call other script and the page shows errors.
So as I noticed that there was no problem, I proceeded to eliminate this malicious script in all files that were found.

Sin título2

Now, the files were disinfected. But the real problem wasn’t resolved yet. The infected server was using Exim. In this case, an effective way to find intrusion and email sent without authorization was to check logs files. I found this worthy information in Exim logs that was located in: /var/log/exim_mainlog and other logs created time before. I identified the email subjects that were sent by the malware by typing this:

$ awk -F"T="" '/<=/ {print $2}' /var/log/exim_mainlog-20151220 | cut -d" -f1 | sort | uniq -c | sort -n
3

The subject “You Have 1 SnapF#ckMsg” was sent 1063 times, so I found the user that was sending this Spam and stopped it. By using this command, I was able to see the user:

$ grep "You Have 1 SnapF#ckMsg" /var/log/exim_mainlog-20151220 | sort -n | uniq -c | sort -n

Result:

1063 U=wherechristispre

Because of this, I got the IP that was sending the mails (I’m not writing the address here because it was the server’s IP and for privacy reasons). But with this, I noticed that the user was sending SPAM locally. The first step to fix this was to change all Email password accounts and then to localize the script that was sending SPAM.

I had to find it by checking the logs and filtered with the next command:

$ grep cwd /var/log/exim_mainlog-20151220 | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n

I located the PHP script that was sending SPAM and with this command, I found the path script that was sending emails.

Sin título5

This script is a malicious one because there shouldn’t be any PHP script here. By opening the file, I found the malicious script that was sending SPAM.

Sin título6

And finally, I deleted it. You could use lsof to see if there are any more PHP scripts executing and discover other possible malicious scripts by using for example:

$ lsof /home/wherechrisispresent/public_html/wp-admin/css/midnight/alias57.php

As I said before, it is necessary to change all the email passwords, WordPress password, update the theme that is being used and deletes unused themes.

How to detect Intrusions?

A host-based intrusion detection system (HIDS), works by collecting details about your computer’s file system and configuration. It then stores this information to reference and validate the current state of the system. If changes are found between the known-good state and the current state, it could be a sign that your security has been compromised. Use these softwares to detect and avoid intrusions:

HIDS tools

1) Tiger:

Tiger is a security tool that can be used both as a security audit and as an intrusion detection system. It supports multiple UNIX platforms and it is free and provided under a GPL license. Unlike other tools, Tiger needs only of POSIX tools and is written entirely in shell language.
Functions:

  • Sniffer bandwidth to detect possible intrusions.
  • Run in background analyzing possible malware.
  • Daily generation of issues report.
  • Detecting suspicious users in the network.
  • Detecting possible malware identified by the sniffing tool.

2)Snort:

Snort is a free open source network and works as an intrusion of detection and prevention system. It is a tool that is widely used in TCP/IP traffic sniffers and analyzers. Through protocol analysis, content is searching and matching.
Functions:

  • Snort detects attack methods, including:
  • denial of service,
  • buffer overflow,
  • CGI attacks,
  • Stealth port scans, and SMB probes.

When suspicious behavior is detected, Snort sends a real-time alert to syslog, a separate ‘alerts’ file, or in a pop-up window.

3) Suricata:

Suricata is a high-performance Network IDS, IPS and Network Security Monitoring engine.
Functions:

  • Highly scalable: you can run one instance and it will balance a load of processing across every processor on a sensor Suricata is configured to use.
  • Protocol identification.
  • File identification, MD5 checksums and File.
  • Extraction.
  • Built in hardware acceleration.

4) Bro:

In a way Bro is both, a signature and an anomaly-based IDS. Its analysis engine will convert traffic captured into a series of events. An event could be a user login to FTP, a connection to a website or practically anything.
Functions:

  • Adaptable: Bro’s domain-specific scripting language enables site-specific monitoring policies.
  • Efficient: targets high-performance networks and is used operationally at a variety of large sites.
  • It is not restricted to any particular detection approach and does not reply on traditional signatures.
  • Enabling high-level semantic analysis at the application layer.

4) Chkrootkit:

Is a shell script that checks system binaries for rootkit modifications.
Functions:

  • Checks if the interface is in promiscuous mode.
  • Checks for last log deletions.
  • Checks for wtmp deletions.
  • Checks for signs of LKM Trojans.
  • Quick and dirty strings replacement.
  • Checks for utmp deletions.

5) Rkhunter:

Functions:

  • MD5 hash compare.
  • Look for default files used by rootkits.
  • Wrong file permissions for binaries.
  • Look for suspected strings in LKM and KLD modules.
  • Look for hidden files.
  • Optional scan within plaintext and binary files.

Information security is the most important practice around the internet network, more specific practices like audits, intrusion detection, hardening in the server, forensics analysis and pen testing because the services that our servers provide are public. We must make them safe, because if we aren’t, then we are vulnerable to be compromised, like in the example above. For this reason, an audit could avoid this uncomfortable situation in which our server has been hacked. A vital part of intrusion detection is gathering information, and detecting and preventing the system attack.

As you can see, in this case of intrusion audit, it is a process that involves many stages and in each one, you need to be careful not to lose any detail (to find information that drives to the suspect infiltration, suspect user, injected bad code, and system infection). The tools that are needed performance and intrusion audit may vary depending on each case. In this case, we started this Forensic analysis reviewing what was happening with the PHP code and why was sending spam. Using Isof was an excellent idea to then filtered the results and started checking for uncommon information. We hope this example helps you understand better what is an Intrusion Audit and how to start a forensic analysis.

secure your website and application

Subscribe

to our newsletter

Table of Contents

We Make DevOps Easier

Weekly DevOps Newsletter

Subscribe to our DevOps News

Subscribe to a monthly newsletter to receive the IT best practices, startup-related insights & emerging technologies.

Join hundreds of business leaders and entrepreneurs, who are part of our growing tech community.

We guarantee 100% privacy. Your information will not be shared.