Assessing Language Models for Salient Class Identification

Abstract

Code review requires reviewers to understand the core intent of code changes, which becomes difficult when a commit modifies multiple classes. In such commits, one or more primarily modified classes, referred to as salient classes, may induce modifications in other classes. Accurate identification of salient classes offers reviewers an effective entry point to navigate code changes and facilitates program comprehension. Existing state-of-the-art approaches rely on complex program-analysis procedures, including Abstract Syntax Tree (AST) parsing, class relation extraction, handcrafted feature engineering, or dependency graph construction. To this end, we study whether language models (LMs) can identify salient classes directly from commits without feature engineering, graph construction, or training. We first construct a new dataset ApacheJavaCM, derived from the ApacheCM dataset, containing 7,911 commits and 25,914 labeled classes. On this dataset, we systematically evaluate whether LMs can identify salient classes directly from commits and compare with the strongest reproducible state-of-the-art (SOTA) baseline. The evaluation covers two large language models (LLMs), GPT-5.4 and DeepSeek-V3.2, one small language model (SLM), Qwen3.5-9B, and three prompting strategies: zero-shot, few-shot, and chain-of-thought. The LMs substantially outperform the baseline while remaining stable across commit characteristics and selected LMs. We also found that, for salient class identification tasks, a 9B-parameter open-source SLM, Qwen3.5-9B, under few-shot prompting, achieves performance comparable to that of a much larger closed-source LLM, GPT-5.4. These results suggest that lightweight, locally deployable SLMs are feasible options for the salient class identification task and can reduce both cost and privacy barriers associated with relying on closed-source LLMs.