arXiv subjects

Jeonghyeon Kim

Publications and source records attributed to Jeonghyeon Kim.

9 recordsLinked to original sources

arXiv2026

RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues: long, structured, and information-rich. Real user requests, however, are typically far shorter and less structured. To characterize this gap, we define a six-category information taxonomy and four dimensions of linguistic style, and apply them to real user prompts from SWE-chat and problem statements from SWE-bench Verified and Pro. We find that requests carrying only a problem statement, alone or with limited additional context, account for 88% of real prompts but just 7% of benchmark problems. Furthermore, 87% of real prompts are casually written whereas 94% of benchmark problems are formal. Guided by these observations, we introduce RealSWE, 381 multi-variant task families derived from SWE-bench Verified and Pro. Variants within each family share the same underlying task and gold patch while differing only in information composition and linguistic style. Evaluating seven contemporary LLMs with RealSWE, we find that i) realistic inputs reduce resolution rates by 6.4 pp on average and can change model rankings. Controlled analysis further shows that ii) including Desired Behavior and Motivation significantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit; iii) linguistic style has only small, model-dependent effects. These findings provide actionable guidance for users and agents: explicitly stating the desired behavior and motivation, which most real prompts omit, substantially improves the LLM's software engineering performance.

cs.AI↗

arXiv2026

AgentLens: Adaptive Visual Modalities for Human-Agent Interaction in Mobile GUI Agents

Mobile GUI agents can automate smartphone tasks by interacting directly with app interfaces, but how they should communicate with users during execution remains underexplored. Existing systems rely on two extremes: foreground execution, which maximizes transparency but prevents multitasking, and background execution, which supports multitasking but provides little visual awareness. Through iterative formative studies, we found that users prefer a hybrid model with just-in-time visual interaction, but the most effective visualization modality depends on the task. Motivated by this, we present AgentLens, a mobile GUI agent that adaptively uses three visual modalities during human-agent interaction: Full UI, Partial UI, and GenUI. AgentLens extends a standard mobile agent with adaptive communication actions and uses Virtual Display to enable background execution with selective visual overlays. In a controlled study with 21 participants, AgentLens was preferred by 85.7% of participants and achieved the highest usability (1.94 Overall PSSUQ) and adoption-intent (6.43/7).

cs.HC↗

arXiv2026

Localized Concept Erasure in Text-to-Image Diffusion Models via High-Level Representation Misdirection

Recent advances in text-to-image (T2I) diffusion models have seen rapid and widespread adoption. However, their powerful generative capabilities raise concerns about potential misuse for synthesizing harmful, private, or copyrighted content. To mitigate such risks, concept erasure techniques have emerged as a promising solution. Prior works have primarily focused on fine-tuning the denoising component (e.g., the U-Net backbone). However, recent causal tracing studies suggest that visual attribute information is localized in the early self-attention layers of the text encoder, indicating a potential alternative for concept erasing. Building on this insight, we conduct preliminary experiments and find that directly fine-tuning early layers can suppress target concepts but often degrades the generation quality of non-target concepts. To overcome this limitation, we propose High-Level Representation Misdirection (HiRM), which misdirects high-level semantic representations of target concepts in the text encoder toward designated vectors such as random directions or semantically defined directions (e.g., supercategories), while updating only early layers that contain causal states of visual attributes. Our decoupling strategy enables precise concept removal with minimal impact on unrelated concepts, as demonstrated by strong results on UnlearnCanvas and NSFW benchmarks across diverse targets (e.g., objects, styles, nudity). HiRM also preserves generative utility at low training cost, transfers to state-of-the-art architectures such as Flux without additional training, and shows synergistic effects with denoiser-based concept erasing methods.

cs.CV↗

arXiv2025

Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations

Prior research on out-of-distribution detection (OoDD) has primarily focused on single-modality models. Recently, with the advent of large-scale pretrained vision-language models such as CLIP, OoDD methods utilizing such multi-modal representations through zero-shot and prompt learning strategies have emerged. However, these methods typically involve either freezing the pretrained weights or only partially tuning them, which can be suboptimal for downstream datasets. In this paper, we highlight that multi-modal fine-tuning (MMFT) can achieve notable OoDD performance. Despite some recent works demonstrating the impact of fine-tuning methods for OoDD, there remains significant potential for performance improvement. We investigate the limitation of na\"ive fine-tuning methods, examining why they fail to fully leverage the pretrained knowledge. Our empirical analysis suggests that this issue could stem from the modality gap within in-distribution (ID) embeddings. To address this, we propose a training objective that enhances cross-modal alignment by regularizing the distances between image and text embeddings of ID data. This adjustment helps in better utilizing pretrained textual information by aligning similar semantics from different modalities (i.e., text and image) more closely in the hyperspherical representation space. We theoretically demonstrate that the proposed regularization corresponds to the maximum likelihood estimation of an energy-based model on a hypersphere. Utilizing ImageNet-1k OoD benchmark datasets, we show that our method, combined with post-hoc OoDD approaches leveraging pretrained knowledge (e.g., NegLabel), significantly outperforms existing methods, achieving state-of-the-art OoDD performance and leading ID accuracy.

cs.CV↗

arXiv2024

Understanding active learning of molecular docking and its applications

With the advancing capabilities of computational methodologies and resources, ultra-large-scale virtual screening via molecular docking has emerged as a prominent strategy for in silico hit discovery. Given the exhaustive nature of ultra-large-scale virtual screening, active learning methodologies have garnered attention as a means to mitigate computational cost through iterative small-scale docking and machine learning model training. While the efficacy of active learning methodologies has been empirically validated in extant literature, a critical investigation remains in how surrogate models can predict docking score without considering three-dimensional structural features, such as receptor conformation and binding poses. In this paper, we thus investigate how active learning methodologies effectively predict docking scores using only 2D structures and under what circumstances they may work particularly well through benchmark studies encompassing six receptor targets. Our findings suggest that surrogate models tend to memorize structural patterns prevalent in high docking scored compounds obtained during acquisition steps. Despite this tendency, surrogate models demonstrate utility in virtual screening, as exemplified in the identification of actives from DUD-E dataset and high docking-scored compounds from EnamineReal library, a significantly larger set than the initial screening pool. Our comprehensive analysis underscores the reliability and potential applicability of active learning methodologies in virtual screening campaigns.

cs.LG↗

arXiv2023

Deep Active Learning with Contrastive Learning Under Realistic Data Pool Assumptions

Active learning aims to identify the most informative data from an unlabeled data pool that enables a model to reach the desired accuracy rapidly. This benefits especially deep neural networks which generally require a huge number of labeled samples to achieve high performance. Most existing active learning methods have been evaluated in an ideal setting where only samples relevant to the target task, i.e., in-distribution samples, exist in an unlabeled data pool. A data pool gathered from the wild, however, is likely to include samples that are irrelevant to the target task at all and/or too ambiguous to assign a single class label even for the oracle. We argue that assuming an unlabeled data pool consisting of samples from various distributions is more realistic. In this work, we introduce new active learning benchmarks that include ambiguous, task-irrelevant out-of-distribution as well as in-distribution samples. We also propose an active learning method designed to acquire informative in-distribution samples in priority. The proposed method leverages both labeled and unlabeled data pools and selects samples from clusters on the feature space constructed via contrastive learning. Experimental results demonstrate that the proposed method requires a lower annotation budget than existing active learning methods to reach the same level of accuracy.

cs.CV↗

arXiv2022

Mind the gap between theory and experiment

We discuss some examples where numerical simulations based on effectively fabricated nanostructures can provide additional insights into an experiment. Focusing on plasmonics, we study Fano resonant systems for optical trapping, realistic dipole antennas for near-field enhancement, and hybrid nanostructures that combine plasmonic metals with dielectrics refractive index sensing. For those systems, the same experimental detail can play a very different role, depending on the type of physical observable. For example, roughness can significantly influence the near-field, but be totally unnoticed in the far-field. It can affect molecules adsorbed on the surface, while refractive index sensing can be fully immune to such roughness. Approaching the experimental situation as closely as possible is certainly a challenging task and we demonstrate a simple approach based on SEM images for that. Altogether, bridging the gap between theory and experiment is not such a trivial task. However, some of the simple steps illustrated in this chapter can help build numerical models that match the experiment better.

physics.optics↗

arXiv2022

Probing surfactant bilayer interactions by tracking optically trapped single nanoparticles

Single-particle tracking and optical tweezers are powerful techniques for studying diverse processes at the microscopic scale. The stochastic behavior of a microscopically observable particle contains information about its interaction with surrounding molecules, and an optical tweezer can further facilitate this observation with its ability to constrain the particle to an area of interest. Although these techniques found their initial applications in biology, they can also shed new light on colloid and interface phenomena by unveiling nanoscale morphologies and molecular-level interactions in real time, which have been obscured in traditional ensemble analysis. Here we demonstrate the application of single-particle tracking and optical tweezers for studying molecular interactions at solid-liquid interfaces. Specifically, we investigate the behavior of surfactants at the water-glass interface by tracing their interactions with gold nanoparticles that are optically trapped on these molecules. We discover the underlying mechanisms governing the particle motion, which can be explained by hydrophobic interactions, disruptions, and rearrangements among surfactant monomers at the interfaces. Such interpretations are further supported by statistical analysis of an individual trajectory and comparison to theoretical predictions. Our findings provide new insights into the surfactant dynamics in this specific system but also illustrate the promise of single-particle tracking and optical manipulation in studying nanoscale physics and chemistry of surfaces and interfaces.

cond-mat.soft↗

arXiv2021

Low temperature annealing method for fabricating alloy nanostructures and metasurfaces: Unlocking a novel degree of freedom

The material and exact shape of a nanostructure determine its optical response, which is especially strong for plasmonic metals. Unfortunately, only a very few plasmonic metals are available, which limits the spectral range where these strong optical effects can be utilized. Alloying different plasmonic metals can overcome this limitation, at the expense of using a high temperature alloying process, which adversely destroys the nanostructure shape. Here, we develop a low temperature alloying process at only 300{\deg}C and fabricate Au-Ag nanostructures with a broad diversity of shapes, aspect ratios and stoichiometries. EDX and XPS analyses confirm the homogeneous alloying through the entire sample. Varying the alloy stoichiometry tunes the optical response of the nanostructure and controls spectral features such as Fano resonances. Binary metasurfaces that combine nanostructures with different stoichiometries are fabricated using multiple-step electron beam lithography, and their optical function as hologram or Fresnel zone plate is demonstrated at the visible wavelength of 532 nm. This low temperature annealing technique provides a versatile and cost-effective way of fabricating complex Au-Ag nanostructures with arbitrary stoichiometry.

cond-mat.mes-hall↗