Find the source of spam-script

Question

We have a shared hosting cgi-server with apache2+php fcgi and dma as mta (it forwards messages to mail relay) - it's Debian Wheezy - on it and with ability for clients to run perl/cgi scripts.There's one client with 70+ sites, and he had been spamming from his ftp-account like crazy. Thing is he doesn't know where the spam script originates from and so do we.

The process(es) who is sending out spam is perl-script hidden as crond - when you check /proc/$PID/cwd it's always /tmp and the file who started the process is already deleted. strace on the process doesn't help - all you see is system calls to make another mail message and headers etc. Searching through the access logs of his few most visited sites for repetitive/suspicious GET/POST requests gets us nowhere.

Should I say regular grep on .php/.cgi/*.pl for base64,eval,fopen,gzinflate and their combinations gives zero result.

The question is - are there any more effective methods/software to find the spam script(s) or to watch which script ran what? Thanks.

Answer 1

I think, the mal-script here, is not in the usual .pl/.cgi/.fcgi/.fpl extension form, but still executing on system level as a cgi script. You need to check Apache Handlers/Apache MIME types, to see what other extensions are being run as cgi script. Once you narrow down on that, a simple grep should work.

Answer 2

I had a similar problem, and following ITroubs's hint, found a cron job, which in this case scheduled a command to be run every 15 minutes, in /var/spool/cron/apache:

*/15 * * * * /var/tmp/lkvnTiofU >/dev/null 2>&1

Commenting this out, changing the file owner and group to root, and finally killing all processes by the apache user with the command perl and deleting /var/tmp/lkvnTiofU seems to have solved the problem.

Looking at open files of the offending processes, with lsof -p [PID] | more where [PID] is the process ID, it seemed that two of the three processes were busy establishing SMTP connections to lots of different servers to send spam, bypassing my servers MTA entirely, and a third process was listening on port 39331, probably a backdoor handling the spamming.

In my case, this was a leftover from an attack I patched a few weeks ago, but in your case, you may need to first identify the original exploit, which may be using a different language altogether, before going through the above.