arXiv subjects

Jialin Li

Publications and source records attributed to Jialin Li.

At least 109 records · Page 6Linked to original sources

arXiv2024

CAR: Controllable Autoregressive Modeling for Visual Generation

Controllable generation, which enables fine-grained control over generated outputs, has emerged as a critical focus in visual generative models. Currently, there are two primary technical approaches in visual generation: diffusion models and autoregressive models. Diffusion models, as exemplified by ControlNet and T2I-Adapter, offer advanced control mechanisms, whereas autoregressive models, despite showcasing impressive generative quality and scalability, remain underexplored in terms of controllability and flexibility. In this study, we introduce Controllable AutoRegressive Modeling (CAR), a novel, plug-and-play framework that integrates conditional control into multi-scale latent variable modeling, enabling efficient control generation within a pre-trained visual autoregressive model. CAR progressively refines and captures control representations, which are injected into each autoregressive step of the pre-trained model to guide the generation process. Our approach demonstrates excellent controllability across various types of conditions and delivers higher image quality compared to previous methods. Additionally, CAR achieves robust generalization with significantly fewer training resources compared to those required for pre-training the model. To the best of our knowledge, we are the first to propose a control framework for pre-trained autoregressive visual generation models.

cs.CV↗

arXiv2024

Safe Time-Varying Optimization based on Gaussian Processes with Spatio-Temporal Kernel

Ensuring safety is a key aspect in sequential decision making problems, such as robotics or process control. The complexity of the underlying systems often makes finding the optimal decision challenging, especially when the safety-critical system is time-varying. Overcoming the problem of optimizing an unknown time-varying reward subject to unknown time-varying safety constraints, we propose TVSafeOpt, a new algorithm built on Bayesian optimization with a spatio-temporal kernel. The algorithm is capable of safely tracking a time-varying safe region without the need for explicit change detection. Optimality guarantees are also provided for the algorithm when the optimization problem becomes stationary. We show that TVSafeOpt compares favorably against SafeOpt on synthetic data, both regarding safety and optimality. Evaluation on a realistic case study with gas compressors confirms that TVSafeOpt ensures safety when solving time-varying optimization problems with unknown reward and safety functions.

cs.LG↗

arXiv2024

Blocks as Probes: Dissecting Categorization Ability of Large Multimodal Models

Categorization, a core cognitive ability in humans that organizes objects based on common features, is essential to cognitive science as well as computer vision. To evaluate the categorization ability of visual AI models, various proxy tasks on recognition from datasets to open world scenarios have been proposed. Recent development of Large Multimodal Models (LMMs) has demonstrated impressive results in high-level visual tasks, such as visual question answering, video temporal reasoning, etc., utilizing the advanced architectures and large-scale multimodal instruction tuning. Previous researchers have developed holistic benchmarks to measure the high-level visual capability of LMMs, but there is still a lack of pure and in-depth quantitative evaluation of the most fundamental categorization ability. According to the research on human cognitive process, categorization can be seen as including two parts: category learning and category use. Inspired by this, we propose a novel, challenging, and efficient benchmark based on composite blocks, called ComBo, which provides a disentangled evaluation framework and covers the entire categorization process from learning to use. By analyzing the results of multiple evaluation tasks, we find that although LMMs exhibit acceptable generalization ability in learning new categories, there are still gaps compared to humans in many ways, such as fine-grained perception of spatial relationship and abstract category understanding. Through the study of categorization, we can provide inspiration for the further development of LMMs in terms of interpretability and generalization.

cs.CV↗

arXiv2024

Anno-incomplete Multi-dataset Detection

Object detectors have shown outstanding performance on various public datasets. However, annotating a new dataset for a new task is usually unavoidable in real, since 1) a single existing dataset usually does not contain all object categories needed; 2) using multiple datasets usually suffers from annotation incompletion and heterogeneous features. We propose a novel problem as "Annotation-incomplete Multi-dataset Detection", and develop an end-to-end multi-task learning architecture which can accurately detect all the object categories with multiple partially annotated datasets. Specifically, we propose an attention feature extractor which helps to mine the relations among different datasets. Besides, a knowledge amalgamation training strategy is incorporated to accommodate heterogeneous features from different sources. Extensive experiments on different object detection datasets demonstrate the effectiveness of our methods and an improvement of 2.17%, 2.10% in mAP can be achieved on COCO and VOC respectively.

cs.CV↗

arXiv2024

Distribution-Aware Calibration for Object Detection with Noisy Bounding Boxes

Large-scale well-annotated datasets are of great importance for training an effective object detector. However, obtaining accurate bounding box annotations is laborious and demanding. Unfortunately, the resultant noisy bounding boxes could cause corrupt supervision signals and thus diminish detection performance. Motivated by the observation that the real ground-truth is usually situated in the aggregation region of the proposals assigned to a noisy ground-truth, we propose DIStribution-aware CalibratiOn (DISCO) to model the spatial distribution of proposals for calibrating supervision signals. In DISCO, spatial distribution modeling is performed to statistically extract the potential locations of objects. Based on the modeled distribution, three distribution-aware techniques, i.e., distribution-aware proposal augmentation (DA-Aug), distribution-aware box refinement (DA-Ref), and distribution-aware confidence estimation (DA-Est), are developed to improve classification, localization, and interpretability, respectively. Extensive experiments on large-scale noisy image datasets (i.e., Pascal VOC and MS-COCO) demonstrate that DISCO can achieve state-of-the-art detection performance, especially at high noise levels. Code is available at https://github.com/Correr-Zhou/DISCO.

cs.CV↗

arXiv2024

Building a Verifiable Logical Clock for P2P Networks

Logical clocks are a fundamental tool to establish causal ordering of events in a distributed system. They have been applied in weakly consistent storage systems, causally ordered broadcast, distributed snapshots, deadlock detection, and distributed system debugging. However, prior logical clock constructs fail to work in an open network with Byzantine participants. In this work, we present Chrono, a novel logical clock system that targets such challenging environment. We first redefine causality properties among distributed processes under the Byzantine failure model. To enforce these properties, Chrono defines a new validator abstraction for building fault-tolerant logical clocks. Furthermore, our validator abstraction is customizable: Chrono includes multiple backend implementations for the abstraction, each with different security-performance trade-offs. We have applied Chrono to build two decentralized applications, a mutual exclusive service and a weakly consistent key-value store. Chrono adds only marginal overhead compared to systems that tolerate no Byzantine faults. It also out-performs state-of-the-art BFT total order protocols by significant margins.

cs.DC↗

arXiv2024

How Well Do LLMs Identify Cultural Unity in Diversity?

Much work on the cultural awareness of large language models (LLMs) focuses on the models' sensitivity to geo-cultural diversity. However, in addition to cross-cultural differences, there also exists common ground across cultures. For instance, a bridal veil in the United States plays a similar cultural-relevant role as a honggaitou in China. In this study, we introduce a benchmark dataset CUNIT for evaluating decoder-only LLMs in understanding the cultural unity of concepts. Specifically, CUNIT consists of 1,425 evaluation examples building upon 285 traditional cultural-specific concepts across 10 countries. Based on a systematic manual annotation of cultural-relevant features per concept, we calculate the cultural association between any pair of cross-cultural concepts. Built upon this dataset, we design a contrastive matching task to evaluate the LLMs' capability to identify highly associated cross-cultural concept pairs. We evaluate 3 strong LLMs, using 3 popular prompting strategies, under the settings of either giving all extracted concept features or no features at all on CUNIT Interestingly, we find that cultural associations across countries regarding clothing concepts largely differ from food. Our analysis shows that LLMs are still limited to capturing cross-cultural associations between concepts compared to humans. Moreover, geo-cultural proximity shows a weak influence on model performance in capturing cross-cultural associations.

cs.CL↗

arXiv2024

Columbo: Low Level End-to-End System Traces through Modular Full-System Simulation

Fully understanding performance is a growing challenge when building next-generation cloud systems. Often these systems build on next-generation hardware, and evaluation in realistic physical testbeds is out of reach. Even when physical testbeds are available, visibility into essential system aspects is a challenge in modern systems where system performance depends on often sub-$μs$ interactions between HW and SW components. Existing tools such as performance counters, logging, and distributed tracing provide aggregate or sampled information, but remain insufficient for understanding individual requests in-depth. In this paper, we explore a fundamentally different approach to enable in-depth understanding of cloud system behavior at the software and hardware level, with (almost) arbitrarily fine-grained visibility. Our proposal is to run cloud systems in detailed full-system simulations, configure the simulators to collect detailed events without affecting the system, and finally assemble these events into end-to-end system traces that can be analyzed by existing distributed tracing tools.

cs.PF↗

arXiv2024

DPDPU: Data Processing with DPUs

Improving the performance and reducing the cost of cloud data systems is increasingly challenging. Data processing units (DPUs) are a promising solution, but utilizing them for data processing needs characterizing the new hardware and recognizing their capabilities and constraints. We hence propose DPDPU, a platform for holistically exploiting DPUs to optimize data processing tasks that are critical to performance and cost. It seeks to fill the semantic gap between DPUs and data processing systems and handle DPU heterogeneity with three engines dedicated to compute, networking, and storage. This paper describes our vision, DPDPU's key components, their associated utilization challenges, as well as the current progress and future plans.

cs.DC↗

arXiv2024

Challenge of direct imaging of exoplanets within structures: disentangling real signal from point source from background light

The high contrast and spatial resolution requirements for directly imaging exoplanets requires effective coordination of wavefront control, coronagraphy, observation techniques, and post-processing algorithms. However, even with this suite of tools, identifying and retrieving exoplanet signals embedded in resolved scattered light regions can be extremely challenging due to the increased noise from scattered light off the circumstellar disk and the potential misinterpretation of the true nature of the detected signal. This issue pertains not only to imaging terrestrial planets in habitable zones within zodiacal and exozodiacal emission but also to young planets embedded in circumstellar, transitional, and debris disks. This is particularly true for Hα detection of exoplanets in transitional disks. This work delves into recent Hα observations of three transitional disks systems with MagAO-X, an extreme adaptive optics system for the 6.5-meter Magellan Clay telescope. We employed angular differential imaging (ADI) and simultaneous spectral differential imaging (SSDI) in combination with KLIP, a PCA algorithm in post-processing, for optimal starlight suppression and quasi-static noise removal. We discuss the challenges in protoplanet identification with MagAO-X in environments rich with scattered and reflected light from disk structures and explore a potential solution for removing noise contributions from real astronomical objects with current observation and post-processing techniques.

astro-ph.EP↗

arXiv2024

MagAO-X: Commissioning Results and Status of Ongoing Upgrades

MagAO-X is the coronagraphic extreme adaptive optics system for the 6.5 m Magellan Clay Telescope. We report the results of commissioning the first phase of MagAO-X. Components now available for routine observations include: the >2 kHz high-order control loop consisting of a 97 actuator woofer deformable mirror (DM), a 2040 actuator tweeter DM, and a modulated pyramid wavefront sensor (WFS); classical Lyot coronagraphs with integrated low-order (LO) WFS and control using a third 97-actuator non-common path correcting (NCPC) DM; broad band imaging in g, r, i, and z filters with two EMCCDs; simultaneous differential imaging in H-alpha; and integral field spectroscopy with the VIS-X module. Early science results include the discovery of an H-alpha jet, images of accreting protoplanets at H-alpha, images of young extrasolar giant planets in the optical, discovery of new white dwarf companions, resolved images of evolved stars, and high-contrast images of circumstellar disks in scattered light in g-band (500 nm). We have commenced an upgrade program, called "Phase II", to enable high-contrast observations at the smallest inner working angles possible. These upgrades include a new 952 actuator NCPC DM to enable coronagraphic wavefront control; phase induced amplitude apodization coronagraphs; new fast cameras for LOWFS and Lyot-LOWFS; and real-time computer upgrades. We will report the status of these upgrades and results of first on-sky testing in March-May 2024.

astro-ph.IM↗

arXiv2024

More data than you want, less data than you need: machine learning approaches to starlight subtraction with MagAO-X

High-contrast imaging data analysis depends on removing residual starlight from the host star to reveal planets and disks. Most observers do this with principal components analysis (i.e. KLIP) using modes computed from the science images themselves. These modes may not be orthogonal to planet and disk signals, leading to over-subtraction. The wavefront sensor data recorded during the observation provide an independent signal with which to predict the instrument point-spread function (PSF). MagAO-X is an extreme adaptive optics (ExAO) system for the 6.5-meter Magellan Clay telescope and a technology pathfinder for ExAO with GMagAO-X on the upcoming Giant Magellan Telescope. MagAO-X is designed to save all sensor information, including kHz-speed wavefront measurements. Our software and compressed data formats were designed to record the millions of training samples required for machine learning with high throughput. The large volume of image and sensor data lets us learn a PSF model incorporating all the information available. This will eventually allow us to probe smaller star-planet separations at greater sensitivities, which will be needed for rocky planet imaging.

astro-ph.IM↗

arXiv2024

MagAO-X Phase II Upgrades: Implementation and First On-Sky Results of a New Post-AO 1000 Actuator Deformable Mirror

MagAO-X is the extreme coronagraphic adaptive optics (AO) instrument for the 6.5-meter Magellan Clay telescope and is currently undergoing a comprehensive batch of upgrades. One innovation that the instrument features is a deformable mirror (DM) dedicated for non-common path aberration correction (NCPC) within the coronagraph arm. We recently upgraded the 97 actuator NCPC DM with a 1000 actuator Boston Micromachines Kilo-DM which serves to (1) correct non-common path aberrations which hamper performance at small inner-working angles, (2) facilitate focal-plane wavefront control algorithms (e.g., electric field conjugation) and (3) enable 10 kHz correction speeds (up from 2 kHz) to assist post-AO, real-time low-order wavefront control. We present details on the characterization and installation of this new DM on MagAO-X as part of our efforts to improve deep contrast performance for imaging circumstellar objects in reflected light. Pre-installation procedures included use of a Twyman-Green interferometer to build an interaction matrix for commanding the DM surface, in closed-loop, to a flat state for seamless integration into the instrument. With this new NCPC DM now installed, we report on-sky results from the MagAO-X observing run in March -- May 2024 for the Focus Diversity Phase Retrieval and implicit Electric Field Conjugation algorithms for quasistatic speckle removal and in-situ Strehl ratio optimization, respectively.

astro-ph.IM↗

arXiv2024

On-sky, real-time optical gain calibration on MagAO-X using incoherent speckles

The next generation of extreme adaptive optics (AO) must be calibrated exceptionally well to achieve the desired contrast for ground-based direct imaging exoplanet targets. Current wavefront sensing and control system responses deviate from lab calibration throughout the night due to non linearities in the wavefront sensor (WFS) and signal loss. One cause of these changes is the optical gain (OG) effect, which shows that the difference between actual and reconstructed wavefronts is sensitive to residual wavefront errors from partially corrected turbulence. This work details on-sky measurement of optical gain on MagAO-X, an extreme AO system on the Magellan Clay 6.5m. We ultimately plan on using a method of high-temporal frequency probes on our deformable mirror to track optical gain on the Pyramid WFS. The high-temporal frequency probes, used to create PSF copies at 10-22 lambda /D, are already routinely used by our system for coronagraph centering and post-observation calibration. This method is supported by the OG measurements from the modal response, measured simultaneously by sequenced pokes of each mode. When tracked with DIMM measurements, optical gain calibrations show a clear dependence on Strehl Ratio, and this relationship is discussed. This more accurate method of calibration is a crucial next step in enabling higher fidelity correction and post processing techniques for direct imaging ground based systems.

astro-ph.IM↗

arXiv2024

Decision Boundary-aware Knowledge Consolidation Generates Better Instance-Incremental Learner

Instance-incremental learning (IIL) focuses on learning continually with data of the same classes. Compared to class-incremental learning (CIL), the IIL is seldom explored because IIL suffers less from catastrophic forgetting (CF). However, besides retaining knowledge, in real-world deployment scenarios where the class space is always predefined, continual and cost-effective model promotion with the potential unavailability of previous data is a more essential demand. Therefore, we first define a new and more practical IIL setting as promoting the model's performance besides resisting CF with only new observations. Two issues have to be tackled in the new IIL setting: 1) the notorious catastrophic forgetting because of no access to old data, and 2) broadening the existing decision boundary to new observations because of concept drift. To tackle these problems, our key insight is to moderately broaden the decision boundary to fail cases while retain old boundary. Hence, we propose a novel decision boundary-aware distillation method with consolidating knowledge to teacher to ease the student learning new knowledge. We also establish the benchmarks on existing datasets Cifar-100 and ImageNet. Notably, extensive experiments demonstrate that the teacher model can be a better incremental learner than the student model, which overturns previous knowledge distillation-based methods treating student as the main role.

cs.LG↗

arXiv2024

Improving Personalisation in Valence and Arousal Prediction using Data Augmentation

In the field of emotion recognition and Human-Machine Interaction (HMI), personalised approaches have exhibited their efficacy in capturing individual-specific characteristics and enhancing affective prediction accuracy. However, personalisation techniques often face the challenge of limited data for target individuals. This paper presents our work on an enhanced personalisation strategy, that leverages data augmentation to develop tailored models for continuous valence and arousal prediction. Our proposed approach, Distance Weighting Augmentation (DWA), employs a weighting-based augmentation method that expands a target individual's dataset, leveraging distance metrics to identify similar samples at the segment-level. Experimental results on the MuSe-Personalisation 2023 Challenge dataset demonstrate that our method significantly improves the performance of features sets which have low baseline performance, on the test set. This improvement in poor-performing features comes without sacrificing performance on high-performing features. In particular, our method achieves a maximum combined testing CCC of 0.78, compared to the reported baseline score of 0.76 (reproduced at 0.72). It also achieved a peak arousal and valence scores of 0.81 and 0.76, compared to reproduced baseline scores of 0.76 and 0.67 respectively. Through this work, we make significant contributions to the advancement of personalised affective computing models, enhancing the practicality and adaptability of data-level personalisation in real world contexts.

cs.LG↗

arXiv2024

IIDM: Inter and Intra-domain Mixing for Semi-supervised Domain Adaptation in Semantic Segmentation

Despite recent advances in semantic segmentation, an inevitable challenge is the performance degradation caused by the domain shift in real applications. Current dominant approach to solve this problem is unsupervised domain adaptation (UDA). However, the absence of labeled target data in UDA is overly restrictive and limits performance. To overcome this limitation, a more practical scenario called semi-supervised domain adaptation (SSDA) has been proposed. Existing SSDA methods are derived from the UDA paradigm and primarily focus on leveraging the unlabeled target data and source data. In this paper, we highlight the significance of exploiting the intra-domain information between the labeled target data and unlabeled target data. Instead of solely using the scarce labeled target data for supervision, we propose a novel SSDA framework that incorporates both Inter and Intra Domain Mixing (IIDM), where inter-domain mixing mitigates the source-target domain gap and intra-domain mixing enriches the available target domain information, and the network can capture more domain-invariant features. We also explore different domain mixing strategies to better exploit the target domain information. Comprehensive experiments conducted on the GTA5 to Cityscapes and SYNTHIA to Cityscapes benchmarks demonstrate the effectiveness of IIDM, surpassing previous methods by a large margin.

cs.CV↗

arXiv2024

LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking

Deep learning models, particularly those based on transformers, often employ numerous stacked structures, which possess identical architectures and perform similar functions. While effective, this stacking paradigm leads to a substantial increase in the number of parameters, posing challenges for practical applications. In today's landscape of increasingly large models, stacking depth can even reach dozens, further exacerbating this issue. To mitigate this problem, we introduce LORS (LOw-rank Residual Structure). LORS allows stacked modules to share the majority of parameters, requiring a much smaller number of unique ones per module to match or even surpass the performance of using entirely distinct ones, thereby significantly reducing parameter usage. We validate our method by applying it to the stacked decoders of a query-based object detector, and conduct extensive experiments on the widely used MS COCO dataset. Experimental results demonstrate the effectiveness of our method, as even with a 70\% reduction in the parameters of the decoder, our method still enables the model to achieve comparable or

cs.CV↗