Teng-Chuan Hsiao

In the late 20th century, computer viruses emerged as powerful malware that resides permanently in target hosts. For a virus to function, it must load into memory from persistent storage, such as a file on a hard drive. Due to the significant destructive potential of viruses, numerous defense measures have been developed to protect computer systems. Among these, antivirus software is one of the most recognized and widely used. Typically, antivirus solutions rely on static analysis (signature-based) technologies to detect infections in files stored on permanent storage devices, such as hard drives or USB (Universal Serial Bus) flash drives. However, a new breed of malware, fileless malware, has been designed to evade detection and enhance durability. Fileless malware resides solely in the memory of the target hosts, circumventing traditional antivirus software, which cannot access or analyze processes executed directly from memory. This study proposes the Check-on-Execution (CoE) kernel-based approach to detect fileless malware on Linux systems. CoE intervenes by suspending code execution before a program executes code from a process’s writable and executable memory area. To prevent the execution of fileless malware, CoE extracts the code from memory, packages it with an ELF (Executable and Linkable Format) header to create an ELF file, and uses VirusTotal for analysis. Experimental results demonstrate that CoE significantly enhances a Linux system’s ability to defend against fileless malware. Additionally, CoE effectively protects against shell code injection attacks, including buffer and memory overflows, and can handle packed malware. However, it is important to note that this study focuses exclusively on fileless malware, and further research is needed to address other types of malware.

The first appearance of viruses can date back to the late last century. As an effective form of malware, viruses reside in the permanent storage of target hosts. Before a virus can execute, that must load into memory from the persistent storage included in the associated file. Due to the reliable destructive power of viruses, many mechanisms have been developed to defend computer systems against these hazardous threats. Antivirus software is one of the most famous and popular among these mechanisms. Most antivirus software uses static analysis (signature-based) technology on files stored in permanent storage, such as hard disks or USB flashes, to detect viruses hidden in files. Fileless malware was developed to enhance the survivability of malware by circumventing detection. Fileless malware only exists in the target hosts’ memory, not files. Antivirus software cannot even access the fileless malware code, much less analyze it, since it may be performed in memory directly without needing to load it from a disk. As a result, it is difficult for an antivirus engine to defend a system against fileless malware attacks. This paper proposes a kernel-based solution called Check-on-Execution (CoE) to detect fileless malware on a Linux system. When a program is going to execute a piece of code in a writable and executable memory area of a process, CoE suspends the code execution first. Coe retrieves the code from memory, packs the code with an ELF header to create an ELF file, and uses VirusTotal to check the file to prevent a Linux system from executing fileless malware. Experimental results show that CoE noticeably enhances the ability of a Linux system to defend itself again fileless malware. CoE is also suitable for protecting a system from shell code injection attacks, such as buffer and heap overflow attacks. It is capable of handling even packed malware. But in this paper, we only focus on fileless malware.