Rootkits are particularly difficult to detect in the infected environment. This makes rootkits more dangerous than most other forms of malware.
AIDE and tripwire are checksum monitoring programs that can be run periodically to detect changes of programs and library files. Both of these are packages in Debian Linux systems. Of these two tools, AIDE is considered more modern and flexible.
Samhain is another checksum-based monitoring system designed for intrusion detection. Samhain takes monitoring to the next step by offering networked monitoring and the signing of important database entries, configuration files and email messages.
STEALTH is a checksum monitor that is designed to work with remotely checking the integrity of a system via SSH.
All checksum monitors rely on a clean baseline installation. In other words, when a system was first installed, the checksum database should be created and saved on some form of read-only storage for future references.
But, what if a system does not have a baseline checksum database when it was first installed? There are generally two ways to remedy.
The first method is to create a clean installation on a separate system, possibly a virtual machine. Run the checksum monitor program on the clean system, then import the checksum database to the production system. This method will generally generate a long list of differences because the “stock” system used for reference purposes does not include all the customization of the production system. Nonetheless, assuming both systems are updated by the same repository of packages, this method will detect rootkit installed files.
The second method is to run the monitor on the production system directly. This approach relies on the assumption that the system is clean when the monitor was run the first time. If you choose this approach, at least run a rootkit detection program first!
More specific detectors for rootkits include chkrootkit and rkhunter. These programs look for signatures of known rootkits and signs that a rootkit is installed. While these programs will not be effective against new and unknown rootkits, they are effective against most rootkits in circulation.