arXiv subjects

Yi Yang

Publications and source records attributed to Yi Yang.

At least 451 records · Page 25Linked to original sources

arXiv2024

Noise Self-Regression: A New Learning Paradigm to Enhance Low-Light Images Without Task-Related Data

Deep learning-based low-light image enhancement (LLIE) is a task of leveraging deep neural networks to enhance the image illumination while keeping the image content unchanged. From the perspective of training data, existing methods complete the LLIE task driven by one of the following three data types: paired data, unpaired data and zero-reference data. Each type of these data-driven methods has its own advantages, e.g., zero-reference data-based methods have very low requirements on training data and can meet the human needs in many scenarios. In this paper, we leverage pure Gaussian noise to complete the LLIE task, which further reduces the requirements for training data in LLIE tasks and can be used as another alternative in practical use. Specifically, we propose Noise SElf-Regression (NoiSER) without access to any task-related data, simply learns a convolutional neural network equipped with an instance-normalization layer by taking a random noise image, $\mathcal{N}(0,σ^2)$ for each pixel, as both input and output for each training pair, and then the low-light image is fed to the trained network for predicting the normal-light image. Technically, an intuitive explanation for its effectiveness is as follows: 1) the self-regression reconstructs the contrast between adjacent pixels of the input image, 2) the instance-normalization layer may naturally remediate the overall magnitude/lighting of the input image, and 3) the $\mathcal{N}(0,σ^2)$ assumption for each pixel enforces the output image to follow the well-known gray-world hypothesis when the image size is big enough. Compared to current state-of-the-art LLIE methods with access to different task-related data, NoiSER is highly competitive in enhancement quality, yet with a much smaller model size, and much lower training and inference cost. Besides, NoiSER also excels in mitigating overexposure and handling joint tasks.

cs.CV↗

arXiv2024

MC-NeRF: Multi-Camera Neural Radiance Fields for Multi-Camera Image Acquisition Systems

Neural Radiance Fields (NeRF) use multi-view images for 3D scene representation, demonstrating remarkable performance. As one of the primary sources of multi-view images, multi-camera systems encounter challenges such as varying intrinsic parameters and frequent pose changes. Most previous NeRF-based methods assume a unique camera and rarely consider multi-camera scenarios. Besides, some NeRF methods that can optimize intrinsic and extrinsic parameters still remain susceptible to suboptimal solutions when these parameters are poor initialized. In this paper, we propose MC-NeRF, a method that enables joint optimization of both intrinsic and extrinsic parameters alongside NeRF. The method also supports each image corresponding to independent camera parameters. First, we tackle coupling issue and the degenerate case that arise from the joint optimization between intrinsic and extrinsic parameters. Second, based on the proposed solutions, we introduce an efficient calibration image acquisition scheme for multi-camera systems, including the design of calibration object. Finally, we present an end-to-end network with training sequence that enables the estimation of intrinsic and extrinsic parameters, along with the rendering network. Furthermore, recognizing that most existing datasets are designed for a unique camera, we construct a real multi-camera image acquisition system and create a corresponding new dataset, which includes both simulated data and real-world captured images. Experiments confirm the effectiveness of our method when each image corresponds to different camera parameters. Specifically, we use multi-cameras, each with different intrinsic and extrinsic parameters in real-world system, to achieve 3D scene representation without providing initial poses.

cs.CV↗

arXiv2024

Topological quantum walk in synthetic non-Abelian gauge fields

We theoretically introduce synthetic non-Abelian gauge fields for topological quantum walks. The photonic mesh lattice configuration is generalized with polarization multiplexing to achieve a four-dimensional Hilbert space, based on which we provide photonic building blocks for realizing various quantum walks in non-Abelian gauge fields. It is found that SU(2) gauge fields can lead to Peierls substitution in both momenta and quasienergy. In one and two dimensions, we describe detailed photonic setups to realize topological quantum walk protocols whose Floquet winding numbers and Rudner-Lindner-Berg-Levin invariants can be effectively controlled by the gauge fields. Finally, we show how non-Abelian gauge fields facilitate convenient simulation of entanglement in conjunction with polarization-dependent and spatial-mode-dependent coin operations. Our results shed light on the study of synthetic non-Abelian gauge fields in photonic Floquet systems.

physics.optics↗

arXiv2024

Simulating Composite Fermion Excitons by Density Functional Theory and Monte Carlo on a Disk

The Kohn-Sham density functional method for the fractional quantum Hall (FQH) effect has recently been developed by mapping the strongly interacting electrons into an auxiliary system of weakly interacting composite fermions (CFs) that experience a density-dependent effective magnetic field. This approach has been successfully applied to explore the edge rescontruction, fractional charge and fractional braiding statistics of quasiparticle excitations. In this work, we investigate composite fermion excitons in the bulk of the disk geometry. By varying the separation of the quasiparticle-quasihole pairs and calculating their energy, we compare the dispersion of the magnetoroton mode with results from other numerical methods, such as exact diagonalization (ED) and Monte Carlo (MC) simulation. Furthermore, through an evaluation of the spectral function, we identify chiral ``graviton'' excitations: a spin $-2$ mode for the particle-like Laughlin state and a spin $2$ mode for the hole-like Laughlin state. This method can be extended to construct neutral collective excitations for other fractional quantum Hall states in disk geometry.

cond-mat.str-el↗

arXiv2024

AnySynth: Harnessing the Power of Image Synthetic Data Generation for Generalized Vision-Language Tasks

Diffusion models have recently been employed to generate high-quality images, reducing the need for manual data collection and improving model generalization in tasks such as object detection, instance segmentation, and image perception. However, the synthetic framework is usually designed with meticulous human effort for each task due to various requirements on image layout, content, and annotation formats, restricting the application of synthetic data on more general scenarios. In this paper, we propose AnySynth, a unified framework integrating adaptable, comprehensive, and highly controllable components capable of generating an arbitrary type of synthetic data given diverse requirements. Specifically, the Task-Specific Layout Generation Module is first introduced to produce reasonable layouts for different tasks by leveraging the generation ability of large language models and layout priors of real-world images. A Uni-Controlled Image Generation Module is then developed to create high-quality synthetic images that are controllable and based on the generated layouts. In addition, user specific reference images, and style images can be incorporated into the generation to task requirements. Finally, the Task-Oriented Annotation Module offers precise and detailed annotations for the generated images across different tasks. We have validated our framework's performance across various tasks, including Few-shot Object Detection, Cross-domain Object Detection, Zero-shot Composed Image Retrieval, and Multi-modal Image Perception and Grounding. The specific data synthesized by our framework significantly improves model performance in these tasks, demonstrating the generality and effectiveness of our framework.

cs.CV↗

arXiv2024

Imagine and Seek: Improving Composed Image Retrieval with an Imagined Proxy

The Zero-shot Composed Image Retrieval (ZSCIR) requires retrieving images that match the query image and the relative captions. Current methods focus on projecting the query image into the text feature space, subsequently combining them with features of query texts for retrieval. However, retrieving images only with the text features cannot guarantee detailed alignment due to the natural gap between images and text. In this paper, we introduce Imagined Proxy for CIR (IP-CIR), a training-free method that creates a proxy image aligned with the query image and text description, enhancing query representation in the retrieval process. We first leverage the large language model's generalization capability to generate an image layout, and then apply both the query text and image for conditional generation. The robust query features are enhanced by merging the proxy image, query image, and text semantic perturbation. Our newly proposed balancing metric integrates text-based and proxy retrieval similarities, allowing for more accurate retrieval of the target image while incorporating image-side information into the process. Experiments on three public datasets demonstrate that our method significantly improves retrieval performances. We achieve state-of-the-art (SOTA) results on the CIRR dataset with a Recall@K of 70.07 at K=10. Additionally, we achieved an improvement in Recall@10 on the FashionIQ dataset, rising from 45.11 to 45.74, and improved the baseline performance in CIRCO with a mAPK@10 score, increasing from 32.24 to 34.26.

cs.CV↗

arXiv2024

Density-wave-like gap evolution in La$_3$Ni$_2$O$_7$ under high pressure revealed by ultrafast optical spectroscopy

Density wave (DW) order is believed to be correlated with superconductivity in the recently discovered high-temperature superconductor La$_3$Ni$_2$O$_7$. However, experimental investigations of its evolution under high pressure are still lacking. Here, we explore the quasiparticle dynamics in bilayer nickelate La$_3$Ni$_2$O$_7$ single crystals using ultrafast optical pump-probe spectroscopy under high pressures up to 34.2 GPa. At ambient pressure, the temperature-dependent relaxation dynamics demonstrate a phonon bottleneck effect due to the opening of an energy gap around 151 K. The energy scale of the DW-like gap is determined to be 66 meV by the Rothwarf-Taylor model. Combined with recent experiential results, we propose that this DW-like transition at ambient pressure and low temperature is spin density wave (SDW). With increasing pressure, this SDW order is significantly suppressed up to 13.3 GPa before it completely disappears around 26 GPa. Remarkably, at pressures above 29.4 GPa, we observe the emergence of another DW-like order with a transition temperature of approximately 135 K, which is probably related to the predicted charge density wave (CDW) order. Our study provides the experimental evidence of the evolution of the DW-like gap under high pressure, offering critical insights into the correlation between DW order and superconductivity in La$_3$Ni$_2$O$_7$.

cond-mat.supr-con↗

arXiv2024

Broadband measurement of Feibelman's quantum surface response functions

The Feibelman $d$-parameter, a mesoscopic complement to the local bulk permittivity, describes quantum optical surface responses for interfaces, including nonlocality, spill-in and-out, and surface-enabled Landau damping. It has been incorporated into the macroscopic Maxwellian framework for convenient modeling and understanding of nanoscale electromagnetic phenomena, calling for the compilation of a $d$-parameter database for interfaces of interest in nano-optics. However, accurate first-principles calculations of $d$-parameters face computational challenges, whereas existing measurements of $d$-parameters are scarce and restricted to narrow spectral windows. We demonstrate a general broadband ellipsometric approach to measure $d$-parameters at a gold--air interface across the visible--ultraviolet regimes. Gold is found to spill in and spill out at different frequencies. We also observe gold's Bennett mode, a surface-dipole resonance associated with a pole of the $d$-parameter, around 2.5 eV. Our measurements give rise to and are further validated by the passivity and Kramers--Kronig causality analysis of $d$-parameters. Our work advances the understanding of quantum surface response and may enable applications like enhanced electron field emission.

physics.optics↗

arXiv2024

Aggregating Nearest Sharp Features via Hybrid Transformers for Video Deblurring

Video deblurring methods, aiming at recovering consecutive sharp frames from a given blurry video, usually assume that the input video suffers from consecutively blurry frames. However, in real-world scenarios captured by modern imaging devices, sharp frames often interspersed within the video, providing temporally nearest sharp features that can aid in the restoration of blurry frames. In this work, we propose a video deblurring method that leverages both neighboring frames and existing sharp frames using hybrid Transformers for feature aggregation. Specifically, we first train a blur-aware detector to distinguish between sharp and blurry frames. Then, a window-based local Transformer is employed for exploiting features from neighboring frames, where cross attention is beneficial for aggregating features from neighboring frames without explicit spatial alignment. To aggregate nearest sharp features from detected sharp frames, we utilize a global Transformer with multi-scale matching capability. Moreover, our method can easily be extended to event-driven video deblurring by incorporating an event fusion module into the global Transformer. Extensive experiments on benchmark datasets demonstrate that our proposed method outperforms state-of-the-art video deblurring methods as well as event-driven video deblurring methods in terms of quantitative metrics and visual quality. The source code and trained models are available at https://github.com/shangwei5/STGTN.

cs.CV↗

arXiv2024

Synthetic gain for electron-beam spectroscopy

Electron-beam microscopy and spectroscopy featuring atomic-scale spatial resolution have become essential tools used daily in almost all branches of nanoscale science and technology. As a natural supercontinuum source of light, free electrons couple with phonons, plasmons, electron-hole pairs, inter- and intra-band transitions, and inner-shell ionization. The multiple excitations, intertwined with the intricate nature of nanostructured samples, present significant challenges in isolating specific spectral characteristics amidst complex experimental backgrounds. Here we introduce the approach of synthetic complex frequency waves to mitigate these challenges in free-electron--light interaction. The complex frequency waves, created through causality-informed coherent superposition of real-frequency waves induced by free electrons, offer virtual gain to offset material losses. This amplifies and enhances spectral features, as confirmed by our electron energy loss and cathodoluminescence measurements on multi-layer membranes, suspended nanoparticles, and film-coupled nanostructures. Strikingly, we reveal that our approach can retrieve resonance excitation completely buried underneath the zero-loss peak, substantially enhance the quality of hyperspectral imaging, and resolve entangled multiple-photon-electron events in their quantum interaction. Our findings indicate the versatile utility of complex frequency waves in various electron-beam spectroscopy and their promising diagnostic capabilities in free-electron quantum optics.

physics.optics↗

arXiv2024

MaGS: Reconstructing and Simulating Dynamic 3D Objects with Mesh-adsorbed Gaussian Splatting

3D reconstruction and simulation, although interrelated, have distinct objectives: reconstruction requires a flexible 3D representation that can adapt to diverse scenes, while simulation needs a structured representation to model motion principles effectively. This paper introduces the Mesh-adsorbed Gaussian Splatting (MaGS) method to address this challenge. MaGS constrains 3D Gaussians to roam near the mesh, creating a mutually adsorbed mesh-Gaussian 3D representation. Such representation harnesses both the rendering flexibility of 3D Gaussians and the structured property of meshes. To achieve this, we introduce RMD-Net, a network that learns motion priors from video data to refine mesh deformations, alongside RGD-Net, which models the relative displacement between the mesh and Gaussians to enhance rendering fidelity under mesh constraints. To generalize to novel, user-defined deformations beyond input video without reliance on temporal data, we propose MPE-Net, which leverages inherent mesh information to bootstrap RMD-Net and RGD-Net. Due to the universality of meshes, MaGS is compatible with various deformation priors such as ARAP, SMPL, and soft physics simulation. Extensive experiments on the D-NeRF, DG-Mesh, and PeopleSnapshot datasets demonstrate that MaGS achieves state-of-the-art performance in both reconstruction and simulation.

cs.CV↗

arXiv2024

Synthetic non-Abelian Electric Fields and Spin-Orbit Coupling in Photonic Synthetic Dimensions

We theoretically propose a scheme to synthesize photonic non-Abelian electric field and spin-orbit coupling (SOC) in the synthetic frequency dimension based on a polarization-multiplexed time-modulated ring resonator. Inside the ring resonator, the cascade of polarization-dependent phase modulation, polarization rotation, and phase retardation enables a photonic realization of minimal SOC composed of equal Rashba and Dresselhaus terms. The synthetic bands and associated spin textures can be effectively probed by the continuous-wave polarization- and time-resolved band structure measurements. Meanwhile, the pulse dynamics in the frequency dimension can be manipulated by both synthetic Abelian and non-Abelian electric fields, leading to Bloch oscillation, Zitterbewegung, and their interplay dynamics. Our proposed setup can be realized with free-space optics, fibers, and integrated photonics, and provides pseudospin control knobs for large-scale synthetic lattices based on frequency combs.

physics.optics↗

arXiv2024

Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models

Diffusion models have revitalized the image generation domain, playing crucial roles in both academic research and artistic expression. With the emergence of new diffusion models, assessing the performance of text-to-image models has become increasingly important. Current metrics focus on directly matching the input text with the generated image, but due to cross-modal information asymmetry, this leads to unreliable or incomplete assessment results. Motivated by this, we introduce the Image Regeneration task in this study to assess text-to-image models by tasking the T2I model with generating an image according to the reference image. We use GPT4V to bridge the gap between the reference image and the text input for the T2I model, allowing T2I models to understand image content. This evaluation process is simplified as comparisons between the generated image and the reference image are straightforward. Two regeneration datasets spanning content-diverse and style-diverse evaluation dataset are introduced to evaluate the leading diffusion models currently available. Additionally, we present ImageRepainter framework to enhance the quality of generated images by improving content comprehension via MLLM guided iterative generation and revision. Our comprehensive experiments have showcased the effectiveness of this framework in assessing the generative capabilities of models. By leveraging MLLM, we have demonstrated that a robust T2M can produce images more closely resembling the reference image.

cs.CV↗

arXiv2024

Moving Off-the-Grid: Scene-Grounded Video Representations

Current vision models typically maintain a fixed correspondence between their representation structure and image space. Each layer comprises a set of tokens arranged "on-the-grid," which biases patches or tokens to encode information at a specific spatio(-temporal) location. In this work we present Moving Off-the-Grid (MooG), a self-supervised video representation model that offers an alternative approach, allowing tokens to move "off-the-grid" to better enable them to represent scene elements consistently, even as they move across the image plane through time. By using a combination of cross-attention and positional embeddings we disentangle the representation structure and image structure. We find that a simple self-supervised objective--next frame prediction--trained on video data, results in a set of latent tokens which bind to specific scene structures and track them as they move. We demonstrate the usefulness of MooG's learned representation both qualitatively and quantitatively by training readouts on top of the learned representation on a variety of downstream tasks. We show that MooG can provide a strong foundation for different vision tasks when compared to "on-the-grid" baselines.

cs.CV↗

arXiv2024

NeuroFly: A framework for whole-brain single neuron reconstruction

Neurons, with their elongated, tree-like dendritic and axonal structures, enable efficient signal integration and long-range communication across brain regions. By reconstructing individual neurons' morphology, we can gain valuable insights into brain connectivity, revealing the structure basis of cognition, movement, and perception. Despite the accumulation of extensive 3D microscopic imaging data, progress has been considerably hindered by the absence of automated tools to streamline this process. Here we introduce NeuroFly, a validated framework for large-scale automatic single neuron reconstruction. This framework breaks down the process into three distinct stages: segmentation, connection, and proofreading. In the segmentation stage, we perform automatic segmentation followed by skeletonization to generate over-segmented neuronal fragments without branches. During the connection stage, we use a 3D image-based path following approach to extend each fragment and connect it with other fragments of the same neuron. Finally, human annotators are required only to proofread the few unresolved positions. The first two stages of our process are clearly defined computer vision problems, and we have trained robust baseline models to solve them. We validated NeuroFly's efficiency using in-house datasets that include a variety of challenging scenarios, such as dense arborizations, weak axons, images with contamination. We will release the datasets along with a suite of visualization and annotation tools for better reproducibility. Our goal is to foster collaboration among researchers to address the neuron reconstruction challenge, ultimately accelerating advancements in neuroscience research. The dataset and code are available at https://github.com/beanli161514/neurofly

cs.CV↗

arXiv2024

LCP-Fusion: A Neural Implicit SLAM with Enhanced Local Constraints and Computable Prior

Recently the dense Simultaneous Localization and Mapping (SLAM) based on neural implicit representation has shown impressive progress in hole filling and high-fidelity mapping. Nevertheless, existing methods either heavily rely on known scene bounds or suffer inconsistent reconstruction due to drift in potential loop-closure regions, or both, which can be attributed to the inflexible representation and lack of local constraints. In this paper, we present LCP-Fusion, a neural implicit SLAM system with enhanced local constraints and computable prior, which takes the sparse voxel octree structure containing feature grids and SDF priors as hybrid scene representation, enabling the scalability and robustness during mapping and tracking. To enhance the local constraints, we propose a novel sliding window selection strategy based on visual overlap to address the loop-closure, and a practical warping loss to constrain relative poses. Moreover, we estimate SDF priors as coarse initialization for implicit features, which brings additional explicit constraints and robustness, especially when a light but efficient adaptive early ending is adopted. Experiments demonstrate that our method achieve better localization accuracy and reconstruction consistency than existing RGB-D implicit SLAM, especially in challenging real scenes (ScanNet) as well as self-captured scenes with unknown scene bounds. The code is available at https://github.com/laliwang/LCP-Fusion.

cs.RO↗

arXiv2024

Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Survey

World models and video generation are pivotal technologies in the domain of autonomous driving, each playing a critical role in enhancing the robustness and reliability of autonomous systems. World models, which simulate the dynamics of real-world environments, and video generation models, which produce realistic video sequences, are increasingly being integrated to improve situational awareness and decision-making capabilities in autonomous vehicles. This paper investigates the relationship between these two technologies, focusing on how their structural parallels, particularly in diffusion-based models, contribute to more accurate and coherent simulations of driving scenarios. We examine leading works such as JEPA, Genie, and Sora, which exemplify different approaches to world model design, thereby highlighting the lack of a universally accepted definition of world models. These diverse interpretations underscore the field's evolving understanding of how world models can be optimized for various autonomous driving tasks. Furthermore, this paper discusses the key evaluation metrics employed in this domain, such as Chamfer distance for 3D scene reconstruction and Fréchet Inception Distance (FID) for assessing the quality of generated video content. By analyzing the interplay between video generation and world models, this survey identifies critical challenges and future research directions, emphasizing the potential of these technologies to jointly advance the performance of autonomous driving systems. The findings presented in this paper aim to provide a comprehensive understanding of how the integration of video generation and world models can drive innovation in the development of safer and more reliable autonomous vehicles.

cs.AI↗

arXiv2024

Luminous Type II Short-Plateau SN 2023ufx: Asymmetric Explosion of a Partially-Stripped Massive Progenitor

We present supernova (SN) 2023ufx, a unique Type IIP SN with the shortest known plateau duration ($t_\mathrm{PT}$ $\sim$47 days), a luminous V-band peak ($M_{V}$ = $-$18.42 $\pm$ 0.08 mag), and a rapid early decline rate ($s1$ = 3.47 $\pm$ 0.09 mag (50 days)$^{-1}$). By comparing observed photometry to a hydrodynamic MESA+STELLA model grid, we constrain the progenitor to be a massive red supergiant with M$_\mathrm{ZAMS}$ $\simeq$19 - 25 M$_{\odot}$. Independent comparisons with nebular spectral models also suggest an initial He-core mass of $\sim$6 M$_{\odot}$, and thus a massive progenitor. For a Type IIP, SN 2023ufx produced an unusually high amount of nickel ($^{56}$Ni) $\sim$0.14 $\pm$ 0.02 M$_{\odot}$, during the explosion. We find that the short plateau duration in SN 2023ufx can be explained with the presence of a small hydrogen envelope (M$_\mathrm{H_\mathrm{env}}$ $\simeq$1.2 M$_{\odot}$), suggesting partial stripping of the progenitor. About $\simeq$0.09 M$_{\odot}$ of CSM through mass loss from late-time stellar evolution of the progenitor is needed to fit the early time ($\lesssim$10 days) pseudo-bolometric light curve. Nebular line diagnostics of broad and multi-peak components of [O I] $λλ$6300, 6364, H$α$, and [Ca II] $λλ$7291, 7323 suggest that the explosion of SN 2023ufx could be inherently asymmetric, preferentially ejecting material along our line-of-sight.

astro-ph.HE↗