On 16.03.2017 my server was hacked for 2 days and created 632 Gb traffic. Luckily netcup does not charge for traffic.
The reason was that I had two security leaks, I guess it was "DirtyCow", but the Struts framework was also vulnerable.

What I did:

1. The root password was changed > reset a new via the netcup console
2. Updated struts libraries in two web applications
3. updated Ubuntu to 3.13.0-112 to, see DigitalOcean

apt-get update
apt-get upgrade

Note: if upgrade fails due to disk capacity, delete old update files in /boot

4. Analysis, searched for changed files the last two days / last 60 min:

find . -mtime -2
find . -mmin -60

5. found user "cloudy_root", removed

userdel -r cloudy_root

6. found changing init.d processes, removed like

update-rc.d -f lmpjqnxuao remove

7. found changing executables in /usr/bin, removed, like

rm lmpjqnxuao 

8. the processes reinstalled itself, so I blocked traffic, which came from IP 198.*, 69.*, 204.*, 46.*

netstat -pant

iptables -I INPUT -s 198.0.0.0/8 -j DROP
iptables -I INPUT -s 69.0.0.0/8 -j DROP
iptables -I INPUT -s 204.0.0.0/8 -j DROP
iptables -I INPUT -s 46.0.0.0/8 -j DROP
iptables -I OUTPUT -s 198.0.0.0/8 -j DROP
iptables -I OUTPUT -s 69.0.0.0/8 -j DROP
iptables -I OUTPUT -s 204.0.0.0/8 -j DROP
iptables -I OUTPUT -s 46.0.0.0/8 -j DROP

9. Finally found cron file, which was responsible for that. It used a lib, so this was suspicious and it recreated itself again and again with differnt pids

ps -ef | grep -i libudev.so.6
vi crontab
rm /etc/cron.daily/gcc.sh

10. Restarted by the way again and again in between

Other useful commands:

who -a
ls -lat
/etc/init.d/collectl start
collectl -p /var/log/collectl/filename --top 
collectl --top
skill -KILL -u <username>
locate acdnfhruvx