As we all know, there is a huge DDoS attack recently which influences lots of websites. At first beginning, our server didn’t get influenced by it. But last week, I allow ssh server by password.(In the past, we only allow the user to use the public key to ssh server. But last week, we just want to allow a user to log in simply and fast. We open it temporarily and forget to close it) This week, the server is totally attacked. Anyway, the final problem is that our server is blocked by increasing useless threads and bandwidth is used up. So here I list how I find these problems and how we try to fix it.
Step1: check network status
According to check network status, you will find bandwidth is too high which influences other normal customers to use this server’s resources.
sudo apt-get install nethogs sudo nethogs sudo nethogs eth0 eth1
Step2: find exact threads
Except knowing the bandwidth status, you also need to know thread status. By viewing real-time thread status, you will know there are many malicious threads which take too many resources. For my case, I find there are 300 malicious threads which are created every 2 minutes. It is not hard to understand that the final server is slow enough to undertake these increasing malicious threads.
sudo apt-get install htop htop
Step3: kill useless threads
After known these malicious threads, we need to kill them. In fact, killing them can solve this problem by root cause. Because for now I only know its effect, not the root cause. Luckily we find a bash script which causes it. So I kill this bash script together. Until now, it looks like we already finish everything. But things haven’t done. After 8 hours stable network, the server is attacked again. I can’t ssh into the server. So final solution is to shut down and rebuild the server. So for now, I don’t know the root cause.
sudo kill -9 $(pgrep <useless_threads_main_name>)
Step4: add your own public key to ssh
In order to avoid the attack happens again, I involve public key back. Because I’m sure this attack happened when I open password login.
ssh <user_name>@<server_ip> 'mkdir -p ~/.ssh;cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub
Here id_rsa.pub is your own public key file.
Step5: disable password to ssh
sudo vi /etc/ssh/sshd_config
# change to no to disable tunnelled clear text passwords PasswordAuthentication no PubkeyAuthentication yes
service ssh restart
Using high-level security configuration is needed to avoid this kind of attack. Once this malicious attack happens, the best way is to backup your data as soon as possible and rebuild your server. (During investigation process, I also install several kinds of maldetect tools. I’m trying to use these tools to scan out the malicious scripts/codes/software. But unfortunately, they all failed.)
I still need more knowledge to help me understand and find out the root cause. Keep learning.