MoCo: Fuzzing Deep Learning Libraries via Assembling Code

Pin Ji; Yang Feng; Duo Wu; Lingyue Yan; Pengling Chen; Jia Liu; Zhihong Zhao

Home/Archive/arXiv

arXiv · 2405.07744

MoCo: Fuzzing Deep Learning Libraries via Assembling Code

Pin Ji·Yang Feng·Duo Wu·Lingyue Yan·Pengling Chen·Jia Liu·Zhihong Zhao

View original source ↗Read source document ↗Export metadata ↓

Abstract

The rapidly developing deep learning (DL) techniques have been applied in software systems with various application scenarios. However, they could also pose new safety threats with potentially serious consequences, especially in safety-critical domains. DL libraries serve as the underlying foundation for DL systems, and bugs in them can have unpredictable impacts that directly affect the behaviors of DL systems. Previous research on fuzzing DL libraries still has limitations in the diversity of test inputs, the construction of test oracles, and the precision of detection. In this paper, we propose MoCo, a novel fuzzing testing method for DL libraries via assembling code. MoCo first disassembles the seed code file to obtain the template and code blocks, and then employs code block mutation operators (e.g., API replacement, random generation, and boundary checking) to generate more new code blocks adapted to the template. By inserting context-appropriate code blocks into the template step by step, MoCo can generate a tree of code files with intergenerational relations. According to the derivation relations in this tree and the applied mutation operators, we construct the test oracle based on the execution state consistency. Since the granularity of code assembly and mutation is controlled rather than randomly divergent, we can quickly pinpoint the lines of code where the bugs are located and the corresponding triggering conditions. We conduct a comprehensive experiment to evaluate the efficiency and effectiveness of MoCo using three widely-used DL libraries (i.e., TensorFlow, PyTorch, and Jittor). During the experiment, MoCo detects 64 new bugs of four types in three DL libraries, where 51 bugs have been confirmed, and 13 bugs have been fixed by developers.

Explore related subjects

cs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Pin Ji, Yang Feng, Duo Wu, Lingyue Yan, Pengling Chen, Jia Liu, Zhihong Zhao. 2024-05-13. MoCo: Fuzzing Deep Learning Libraries via Assembling Code. https://arxiv.org/abs/2405.07744

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Combining Type Inference and Automated Unit Test Generation for Python

Automated unit test generation is an established research field that has so far focused on statically-typed programming languages. The lack of type information in dynamically-typed programming languages, such as Python, inhibits test generators, which heavily rely on information about parameter and return types of functions to select suitable arguments when constructing test cases. Since automated test generators inherently rely on frequent execution of candidate tests, we make use of these frequent executions to address this problem by introducing type tracing, which extracts type-related information during execution and gradually refines the available type information. We implement type tracing as an extension of the Pynguin test-generation framework for Python, allowing it (i) to infer parameter types by observing how parameters are used during runtime, (ii) to record the types of values that function calls return, and (iii) to use this type information to increase code coverage. The approach leads to up to 87.8 % more branch coverage, improved mutation scores, and to type information of similar quality to that produced by other state-of-the-art type-inference tools.

cs.SE↗

arXiv2026

Guidelines for Empirical Studies in Software Engineering involving Large Language Models

Large Language Models (LLMs) are widely used in software engineering (SE) research and practice, yet their non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical studies. We address this challenge through a collaborative effort of 22 researchers, presenting a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (must) from recommendations (should) and is contextualized by the study types it applies to. Our guidelines recommend that researchers: (1) declare LLM usage and role; (2) report model versions, configurations, and customizations; (3) document the system and prompt design beyond the model; (4) report session traces, i.e., interaction logs and runtime traces; (5) use suitable baselines, benchmarks, and metrics; (6) include an open LLM as a baseline; (7) validate LLM outputs against human judgment; and (8) articulate limitations and mitigations. We complement the guidelines with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. We maintain the study types and guidelines online as a living resource for the community to use and shape (llm-guidelines$.$org).

cs.SE↗

arXiv2026

MigrateLib: a tool for end-to-end Python library migration

Library migration is the process of replacing a library with a similar one in a software project. Manual library migration is time consuming and error prone, as it requires developers to understand the Application Programming Interfaces (API) of both libraries, map equivalent APIs, and perform the necessary code transformations. Due to the difficulty of the library migration process, most of the existing automated techniques and tooling stop at the API mapping stage or support a limited set of libraries and code transformations. In this paper, we develop an end-to-end solution that can automatically migrate code between any arbitrary pair of Python libraries that provide similar functionality. Due to the promising capabilities of Large Language Models (LLMs) in code generation and transformation, we use LLMs as the primary engine for migration. Before building the tool, we first study the capabilities of LLMs for library migration on a benchmark of 321 real-world library migrations. We find that LLMs can effectively perform library migration, but some post-processing steps can further improve the performance. Based on this, we develop MigrateLib, a command line application that combines the power of LLMs, static analysis, and dynamic analysis to provide accurate library migration. We evaluate MigrateLib on 717 real-world Python applications that are not from our benchmark. We find that MigrateLib can migrate 32% of the migrations with complete correctness. Of the remaining migrations, only 14% of the migration-related changes are left for developers to fix for more than half of the projects.

cs.SE↗