Enterprise AI Analysis
DrWASI: LLM-assisted Differential Testing for WebAssembly System Interface Implementations
This analysis explores DrWASI, a pioneering framework for detecting implementation bugs in WebAssembly System Interface (WASI). Leveraging LLMs for seed generation and differential testing across major Wasm runtimes, DrWASI uncovers critical vulnerabilities and inconsistencies, offering invaluable insights for enhancing the security and reliability of the Wasm ecosystem.
Executive Impact Summary
WebAssembly (Wasm) is an emerging binary format that serves as a compilation target for over 40 programming languages. Wasm runtimes provide execution environments that enhance portability by abstracting away operating systems and hardware details. A key component in these runtimes is the WebAssembly System Interface (WASI), which manages interactions with operating systems, like file operations. Considering the critical role of Wasm runtimes, the community has aimed to detect their implementation bugs. However, no work has focused on WASI-specific bugs that can affect the original functionalities of running Wasm binaries and cause unexpected results. To fill the void, we present DRWASI, the first general-purpose differential testing framework for WASI implementations. Our approach uses a large language model to generate seeds and applies variant and environment mutation strategies to expand and enrich the test case corpus. We then perform differential testing across major Wasm runtimes. By leveraging dynamic and static information collected during and after the execution, DRWASI can identify bugs. Our evaluation shows that DRWASI uncovered 33 unique bugs, with all confirmed and 7 fixed by developers. This research represents a pioneering step in exploring a promising yet under-explored area of the Wasm ecosystem, providing valuable insights for stakeholders.
0
Bugs Identified
0
Bugs Fixed
0
Test Cases Generated
0
Differential Tests Performed
0
Acceptance Rate
0
LLM Seed Gen Time
0
Total Downloads
0
Total Citations
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Authority Bugs
Modification Bugs
Reading Bugs
Understanding WASI Authority Bugs
As shown in Table 4, we have identified eight bugs that violate strict authority rules, seven in Wasmer and one in Wasmtime. As shown in Figure 6, it can be divided into three subcategories. A1. Unexpected Denial of Service. There is a bug when a file is expected to open with the specified access mode and permission, but the Wasm runtime fails as it demands more access modes beyond the required ones [24]. As the Case 1 shown in Figure 7, though the file permission of hello.txt is changed to 0400, i.e., read-only, Wasmer fails to open the file and outputs the error message at Line 5 of the test case code [24]. This bug violates the authority rules (see Section 2.2). We examined the implementation of Wasmer and found it requires write access for a file in default, regardless of the actual needs. As a result, if Wasmer attempts to open a file in read-only mode, an exception is raised because it cannot acquire the write access mode. This situation could lead to interruptions in file-related operations right when the file is opened, thereby hindering the completion of subsequent tasks. A2. File Permission Escape. Six bugs [35, 38, 50, 51, 53, 57] allow access to files without necessary permissions or access modes. Notably, one of these bugs even permits access to files or directories without any permissions granted by the OS. For example, as Line 2 of the Case 2 shown in Figure 8, even when the OS sets the permission of the directory as 0400, i.e., read-only, Wasmer can still access the file with the O_WRONLY mode [35]. In other words, Wasmer escapes the file permissions the OS sets. Another example is shown in Figure 9 (Case 3), where Wasmtime grants the access mode which should be prohibited [57]. Specifically, the test case gets the file access mode with O_RDWR, i.e., read and write at Line 15, although it is opened with O_WRONLY (Line 1). After our promptly feedback, developers of Wasmtime have confirmed and fixed this bug in Line 1–Line 4 of Wasmtime source code. Specifically, Wasmtime first checks for the READ flag in Line 1. If the file does not have the READ flag, Wasmtime then clears the FD_READ bit by performing a bitwise AND operation, thus removing the FD_READ permission in Lines 2 and 3. This bug could enable Wasm binaries to bypass the OS permissions and access files, potentially leading to problems such as malicious file tampering and alterations to critical files. A3. Misleading Error Message. A bug leads to the return of a misleading error message related to authorization rules [45]. Specifically, Wasmer attempts to open a file with the access mode O_RDWR under the insufficient permission 0200 (-w- — -), which should not be allowed [45]. The expected return message is permission denied, but Wasmer returns no such file or directory. This issue may mislead the developers and users of the Wasm application, making them believe that the file they are trying to access does not exist, when in fact, the problem is due to insufficient permissions.
8
Total Authority Bugs Identified
Case Study: A1 - Unexpected Denial of Service (Figure 7)
Bug description: The Wasm runtime fails to get the file descriptor. Wasmer 4.3.1 and 4.2.2 fail to open a file with O_RDONLY even when expected to succeed. This occurs because Wasmer requires write access by default, causing exceptions during read-only operations.
Impact: Leads to interruptions in file-related operations, hindering task completion and affecting application stability.
Case Study: A2 - File Permission Escape (Figure 8, Case 2)
Bug description: Wasmer 4.3.1 allows access to files with O_WRONLY mode even when the OS sets read-only permissions (0400) on the directory. This indicates a bypass of OS-level file permissions.
Impact: Critical security vulnerability, potentially allowing malicious Wasm binaries to tamper with or alter critical files, compromising data integrity.
Case Study: A2 - Grant Additional Access Mode (Figure 9, Case 3)
Bug description: Wasmtime 19.0.2 grants additional access modes (O_RDWR) when a file is opened with O_WRONLY, a prohibited action. This bug was confirmed and fixed by Wasmtime developers after feedback.
Impact: Could enable Wasm binaries to bypass OS permissions, leading to unauthorized file access and potential system compromise.
Understanding WASI Modification Bugs
As shown in Table 4, we have identified 17 modification bugs, where 4 of them have been fixed already. These bugs make up the majority of the bugs we found. They are primarily related to the implementation of WASI functions such as fd_allocate, fd_datasync, and path_filestat_set_times. We will detail them in the following. B1. Unanticipated File Size Change. Four bugs [25, 36, 37, 52] are associated with such incorrect and unexpected file size changes. An example is shown in Figure 12 (Case 4), when allocating file spaces, if start_value + len is less than file_size, the file size and file content should remain unchanged. This is also the behavior observed in all Wasm runtimes except for Wasmer [25]. Specifically, Wasmer directly truncates the file at Line 5 of the test case code, reducing its size from 49 to 1. Meanwhile, a portion of the file content is lost. After reviewing the Wasmer implementation, we found that this issue occurs because Wasmer only checks whether the file size would overflow but does not perform other checks, such as verifying whether start_value + len is less than file_size. Interestingly, we also observed that Wasmer fails to truncate a file when it is opened with the access mode O_TRUNC [36]. Furthermore, when allocating new size for a file or setting the file size using fd_filestat_set_size, Wasmer returns a success message but the file size remains unchanged [37, 52]. These bugs can result in unintended modifications on file sizes and contents, potentially causing data loss or corruption. Such issues can compromise the reliability and integrity of file handling in Wasm runtimes, affecting the stability and correctness of applications running in these environments. B2. Unpredictable File Offset. There are two bugs [41, 54] related to the file offset, i.e., the position of the file cursor. Specifically, when setting the file offset using the WASI function fd_seek, Wasmer fails to update the offset value [41]. Additionally, when opening a file with O_APPEND and writing content, it is expected that the new content will be appended to the end of the file and the offset will be updated to the end of the file after writing. However, in Wasmer, though the new content is successfully appended, the offset does not move to the end of the file [54]. Incorrect file offset settings can result in data to be written to the wrong location, impacting the consistency and correctness of Wasm applications that depend on file contents. B3. Erroneous File Content Update. These two bugs are related to the issues of writing new content into a file and exist in Wasmer [42] and Wasmtime [58]. One of the bug is found by parsing the collected static information. Similar to other Wasm runtimes, Wasmer also prints Successfully write 388 bytes for a write request. However, it does not actually write any bytes into the opened file [42]. As for another one, it writes the incorrect content to the file, which test case is shown in Figure 11 (Case 5) [58]. As we can see, the file is opened with 0_APPEND in Line 1, and the content is expected to be written from the end of the file. However, Wasmtime ignores whether the file is opened in the append mode, as illustrated at Line 3, where the offset is hard-encoded as 0, i.e., the beginning of the file. Intuitively, the patch ignores the specified offset when the file is opened in the append mode, as illustrated at the bottom of Figure 11, such bugs can result in incorrect modifications to file content, which may, in turn, affect the functionality and consistency of Wasm applications. B4. Unable to Create or Delete Files. There are four bugs related to failures in creating and deleting files [27, 44, 48, 49]. Specifically, Wasmer fails to create hard/soft link on files [44, 48]. Besides, it is also found that Wasmer fails to delete a file using the WASI function path_unlink_file, whereas other Wasm runtimes can complete successfully [49]. These bugs may affect file access and are specific to Wasmer, potentially leading to compatibility issues when migrating or running the same Wasm applications across different Wasm runtimes. This can increase the challenges for Wasm application developers during development and debugging. B5. Immutable File Metadata Update. These two bugs [47, 55] emerge when modifying file metadata, such as renaming a file or changing its last access time. These bugs can result in inaccurate file information, which, in turn, impacts file organization and management. B6. Misguiding Return Message. Similar to A3, even though the behaviors among Wasm runtimes are consistent, the return values could be different. There are three bugs that show this symptom [20, 28, 59]. For example, as shown in Figure 10 (Case 6), the data synchronization at Line 3 is expected to be successful, while Wasmer indicates a data synchronization failure at Line 4 of the test case code [28]. This is because Wasmer checks whether the current file descriptor has the FD_DATASYNC availability flag before calling fd_datasync. If it does, it returns Errno::Access, which is exactly contrary to expectations. We have corrected this issue and pull our patch into the master branch as shown at the bottom of Figure 10 [12]. Such errors can cause Wasm application developers significant difficulties in debugging and maintenance, leading them to search for issues in the wrong places and wasting valuable time and effort. If these bugs are not resolved before deployment to the production environment, they can undermine the application's stability and reliability, ultimately negatively impacting the user experience of Wasm applications.
17
Total Modification Bugs Identified
Case Study: B1 - File Truncated (Figure 12, Case 4)
Bug description: Wasmer 4.2.2 truncates the file content unexpectedly when allocating file spaces, specifically when start_value + len is less than file_size. The file size is reduced from 49 to 1, causing data loss.
Impact: Results in unintended modifications to file sizes and contents, potentially leading to data loss or corruption, compromising data reliability.
Case Study: B3 - Content Writing Bug (Figure 11, Case 5)
Bug description: Wasmtime 19.0.2 ignores the O_APPEND mode when writing to a file, hard-encoding the offset as 0. Content expected to be appended is instead written from the beginning of the file.
Impact: Incorrect modifications to file content, affecting data consistency and the functionality of Wasm applications relying on accurate file state.
Case Study: B6 - Fail to Flush to Disk (Figure 10, Case 6)
Bug description: Wasmer 4.2.2 reports data synchronization failure for fd_datasync even when successful. This happens because Wasmer incorrectly checks for the FD_DATASYNC availability flag before the call, leading to an Errno::Access return.
Impact: Misleading error messages cause significant debugging difficulties for developers, wasting time and potentially undermining application stability if not resolved.
Understanding WASI Reading Bugs
As shown in Table 4, eight bugs fall into this category, which account for 24.2% of the total. These bugs are associated with the implementation of WASI functions such as fd_read and fd_filestat_get, resulting in getting incorrect information about files or directories. C1. Link File Read Error. Five out of eight such reading bugs are related to link files, including reading the source file via a soft link [34, 46], retrieving the number of hard link files [21, 39] under a directory, and displaying the file size of a link file [37]. For example, as shown in Figure 14 (Case 7), Wasmer fails to print the file size of a hard link file correctly [37]. Specifically, the file hardfile_1 is a hard link file which points to subdir_2/subdir_1/subfile_2, which file is indicated as empty by Wasmer. Besides, Wasmer also fails to parse the source file of a link file [46]. As shown in Figure 15 (Case 8), when retrieving the source file information via a soft link, the expected result is to correctly output the file it points to. However, both Wasmer and WasmEdge fail to obtain the source file information. Such bugs may cause link file-related functionality in Wasm applications to fail. For example, they could mislead Wasm application users or developers into making incorrect judgments when handling files, leading to subsequent operation failures or incorrect data processing. C2. Unpredictable File Metadata Reading. Two bugs related to failed file metadata parsing, including file size, access time, and owner. These two bugs are all found in Wasmer [26, 43]. C3. Erroneous File Content Reading. Wasmer also fails to read the file content into the buffer [40]. Both C2 and C3 related to mistakenly reading file metadata and file content can result in incorrect or unusable file information being displayed, negatively impacting the user experience for Wasm application users.
8
Total Reading Bugs Identified
Case Study: C1 - Print File Size Bug (Figure 14, Case 7)
Bug description: Wasmer 4.3.1 fails to print the correct file size for a hard link file (hardfile_1). It reports "0 bytes" instead of the expected "26 bytes", indicating an issue in retrieving accurate file metadata.
Impact: Misleads Wasm application users and developers regarding file information, potentially causing incorrect judgments and subsequent operational failures or data processing errors.
Case Study: C1 - Source File Read Bug (Figure 15, Case 8)
Bug description: Wasmer 4.3.1, 4.2.2, WasmEdge 0.13.5, and 0.13.3 fail to get the source file information when reading via a soft link (softfile_1). The expected result is the link contents, but it returns "Fails."
Impact: Causes link file-related functionality to fail in Wasm applications, leading to incorrect file handling and poor user experience.
Enterprise Process Flow
LLM-assisted Seed Generation
→
Variant Mutation & C Test Cases
→
Compile to Wasm Binaries
→
Environment Setup & Perm. Mutation
→
Differential Execution (Runtimes)
→
Dynamic & Static Info Collection
→
Report Deduplication
→
Final Bug Report Generation
→
GitHub Issue Reporting
| Runtime | Language | Version | Stars | #Commits |
|---|---|---|---|---|
| Wasmer | Rust | 4.3.1 and 4.2.2 | 18.4k | 17,865 |
| Wasmtime | Rust | 19.0.2 and 13.0.1 | 15.1k | 13,566 |
| WAMR | C/C++ | 1.3.2 and 1.2.3 | 4.8k | 2,014 |
| WasmEdge | C/C++ | 0.13.5 and 0.13.3 | 8.4k | 3,599 |
Calculate Your Potential ROI
Estimate the annual savings and efficiency gains your enterprise could achieve by addressing WASI implementation challenges with our solutions.
Estimated Annual Savings
$0
Annual Hours Reclaimed
0
Your Path to Robust WASI Implementations
Our proven methodology ensures a seamless integration of best practices for WebAssembly System Interface. Here's a typical roadmap:
Phase 1: Initial Assessment & Gap Analysis
We begin with a comprehensive review of your current Wasm runtime environment and WASI usage. Our experts identify existing inconsistencies, potential vulnerabilities, and areas for optimization based on DrWASI's findings.
Phase 2: Custom Test Suite Development & Deployment
Leveraging LLM-assisted techniques, we generate a tailored test suite specific to your WASI functions and use cases. This suite is then deployed for differential testing against your chosen Wasm runtimes.
Phase 3: Bug Identification & Root Cause Analysis
Our framework performs automated differential testing, identifying discrepancies in WASI implementations. We conduct deep static and dynamic analysis to pinpoint root causes and provide actionable insights for developers.
Phase 4: Patching, Validation & Continuous Integration
We collaborate with your development team to implement fixes and validate their effectiveness using our enhanced test suite. We also establish a continuous testing framework to prevent regressions and ensure ongoing WASI compliance and security.
Ready to Fortify Your WebAssembly Ecosystem?
Don't let WASI implementation inconsistencies compromise your applications. Partner with us to ensure security, portability, and reliability.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat