arXiv subjects

Tianshi Chen

Publications and source records attributed to Tianshi Chen.

At least 19 recordsLinked to original sources

arXiv2026

Kernel-based linear system identification using augmented Krylov subspaces

We propose a novel Krylov subspace method for estimating the finite impulse response (FIR) of a one-dimensional linear time-invariant systems. The method approximates the system's FIR using a kernel-based formulation combined with hyperparameter selection based on maximum likelihood estimation (MLE), which requires repeated evaluation of two terms: The data fit $\boldsymbol{y}^{\top} (\lambda \boldsymbol{I} + \boldsymbol{A})^{-1} \boldsymbol{y}$ and the model complexity $\log(\det (\lambda \boldsymbol{I} + \boldsymbol{A}))$, where $\boldsymbol{A}$ is a certain positive semidefinite matrix that admits fast matrix--vector products and $\lambda > 0$ is a regularization parameter. Instead of approximating these two quantities separately, we jointly approximate them using a single augmented Krylov subspace for $\boldsymbol{A}$. One major benefit of augmentation is that we obtain accelerated convergence when approximating the data fit quadratic form, through implicit preconditioning. Thanks to the shift invariance of Krylov subspaces, the extracted approximations can be used to evaluate the MLE objective for many values of $\lambda$ at little additional cost. We derive error bounds for the approximations, reflecting the benefits of augmentation demonstrated through multiple numerical experiments.

math.NA↗

arXiv2025

Numerically Efficient and Stable Algorithms for Kernel-Based Regularized System Identification Using Givens-Vector Representation

Numerically efficient and stable algorithms are essential for kernel-based regularized system identification. The state of art algorithms exploit the semiseparable structure of the kernel and are based on the generator representation of the kernel matrix. However, as will be shown from both the theory and the practice, the algorithms based on the generator representation are sometimes numerically unstable, which limits their application in practice. This paper aims to address this issue by deriving and exploiting an alternative Givens-vector representation of some widely used kernel matrices. Based on the Givens-vector representation, we derive algorithms that yield more accurate results than existing algorithms without sacrificing efficiency. We demonstrate their usage for the kernel-based regularized system identification. Monte Carlo simulations show that the proposed algorithms admit the same order of computational complexity as the state-of-the-art ones based on generator representation, but without issues with numerical stability.

math.NA↗

arXiv2025

Hardwired-Neurons Language Processing Units as General-Purpose Cognitive Substrates

The rapid advancement of Large Language Models (LLMs) has established language as a core general-purpose cognitive substrate, driving the demand for specialized Language Processing Units (LPUs) tailored for LLM inference. To overcome the growing energy consumption of LLM inference systems, this paper proposes a Hardwired-Neurons Language Processing Unit (HNLPU), which physically hardwires LLM weight parameters into the computational fabric, achieving several orders of magnitude computational efficiency improvement by extreme specialization. However, a significant challenge still lies in the scale of modern LLMs. A straightforward hardwiring of gpt-oss 120 B would require fabricating photomask sets valued at over 6 billion dollars, rendering this straightforward solution economically impractical. Addressing this challenge, we propose the novel Metal-Embedding methodology. Instead of embedding weights in a 2D grid of silicon device cells, Metal-Embedding embeds weight parameters into the 3D topology of metal wires. This brings two benefits: (1) a 15x increase in density, and (2) 60 out of 70 photomask layers are homogeneous across chips, including all EUV photomasks. In total, Metal-Embedding reduced the photomask cost by 112x, bringing the Non-Recurring Engineering (NRE) cost of HNLPU into an economically viable range. Experimental results show that HNLPU achieved 249,960 tokens/s (5,555x/85x that of GPU/WSE), 36 tokens/J (1,047x/283x that of GPU/WSE), 13,232 mm2 total die area, $59.46 M-123.5 M estimated NRE at 5 nm technology. Analysis shows that HNLPU achieved 41.7-80.4x improvement in cost-effectiveness and 357x reduction in carbon footprint compared to OpenAI-scale H100 clusters, under an annual weight updating assumption.

cs.AR↗

arXiv2025

On Kernel Design for Regularized Volterra Series Identification of Wiener-Hammerstein Systems

There have been increasing interests on the Volterra series identification with the kernel-based regularization method. The major difficulties are on the kernel design and efficiency of the corresponding implementation. In this paper, we first assume that the underlying system to be identified is the Wiener-Hammerstein (WH) system with polynomial nonlinearity. We then show how to design kernels with nonzero off-diagonal blocks for Volterra maps by taking into account the prior knowledge of the linear blocks and the structure of WH systems. Moreover, exploring the structure of the designed kernels leads to the same computational complexity as the state-of-the-art result, i.e., $O(N^3)$, where $N$ is the sample size, but with a significant difference that the proposed kernels are designed in a direct and flexible way. In addition, for a special case of the kernel and a class of widely used input signals, further exploring the separable structure of the output kernel matrix can lower the computational complexity from $O(N^3)$ to $O(N\gamma^2)$, where $\gamma$ is the separability rank of the output kernel matrix and can be much smaller than $N$. We finally run Monte Carlo simulations to demonstrate the proposed kernels and the obtained theoretical results.

eess.SY↗

arXiv2025

QiMeng-Xpiler: Transcompiling Tensor Programs for Deep Learning Systems with a Neural-Symbolic Approach

Heterogeneous deep learning systems (DLS) such as GPUs and ASICs have been widely deployed in industrial data centers, which requires to develop multiple low-level tensor programs for different platforms. An attractive solution to relieve the programming burden is to transcompile the legacy code of one platform to others. However, current transcompilation techniques struggle with either tremendous manual efforts or functional incorrectness, rendering "Write Once, Run Anywhere" of tensor programs an open question. We propose a novel transcompiler, i.e., QiMeng-Xpiler, for automatically translating tensor programs across DLS via both large language models (LLMs) and symbolic program synthesis, i.e., neural-symbolic synthesis. The key insight is leveraging the powerful code generation ability of LLM to make costly search-based symbolic synthesis computationally tractable. Concretely, we propose multiple LLM-assisted compilation passes via pre-defined meta-prompts for program transformation. During each program transformation, efficient symbolic program synthesis is employed to repair incorrect code snippets with a limited scale. To attain high performance, we propose a hierarchical auto-tuning approach to systematically explore both the parameters and sequences of transformation passes. Experiments on 4 DLS with distinct programming interfaces, i.e., Intel DL Boost with VNNI, NVIDIA GPU with CUDA, AMD MI with HIP, and Cambricon MLU with BANG, demonstrate that QiMeng-Xpiler correctly translates different tensor programs at the accuracy of 95% on average, and the performance of translated programs achieves up to 2.0x over vendor-provided manually-optimized libraries. As a result, the programming productivity of DLS is improved by up to 96.0x via transcompiling legacy tensor programs.

cs.CL↗

arXiv2024

Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM

Deploying advanced large language models on edge devices, such as smartphones and robotics, is a growing trend that enhances user data privacy and network connectivity resilience while preserving intelligent capabilities. However, such a task exhibits single-batch computing with incredibly low arithmetic intensity, which poses the significant challenges of huge memory footprint and bandwidth demands on limited edge resources. To address these issues, we introduce Cambricon-LLM, a chiplet-based hybrid architecture with NPU and a dedicated NAND flash chip to enable efficient on-device inference of 70B LLMs. Such a hybrid architecture utilizes both the high computing capability of NPU and the data capacity of the NAND flash chip, with the proposed hardware-tiling strategy that minimizes the data movement overhead between NPU and NAND flash chip. Specifically, the NAND flash chip, enhanced by our innovative in-flash computing and on-die ECC techniques, excels at performing precise lightweight on-die processing. Simultaneously, the NPU collaborates with the flash chip for matrix operations and handles special function computations beyond the flash's on-die processing capabilities. Overall, Cambricon-LLM enables the on-device inference of 70B LLMs at a speed of 3.44 token/s, and 7B LLMs at a speed of 36.34 token/s, which is over 22X to 45X faster than existing flash-offloading technologies, showing the potentiality of deploying powerful LLMs in edge devices.

cs.AR↗

arXiv2024

Kernel-Based Regularized Continuous-Time System Identification from Sampled Data

The identification of continuous-time (CT) systems from discrete-time (DT) input and output signals, i.e., the sampled data, has received considerable attention for half a century. The state-of-the-art methods are parametric methods and thus subject to the typical issues of parametric methods. In the last decade, a major advance in system identification is the so-called kernel-based regularization method (KRM), which is free of the issues of parametric methods. It is interesting to test the potential of KRM on CT system identification. However, very few results have been reported, mainly because the estimators have no closed forms for general CT input signals, except for some very special cases. In this paper, we show for KRM that the estimators have closed forms when the DT input signal has the typical intersample behavior, i.e., zero-order hold or band-limited, and this paves the way for the application of KRM for CT system identification. Numerical Monte Carlo simulations show that the proposed method is more robust than the state-of-the-art methods and more accurate when the sample size is small.

eess.SY↗

arXiv2024

Ex3: Automatic Novel Writing by Extracting, Excelsior and Expanding

Generating long-term texts such as novels using artificial intelligence has always been a challenge. A common approach is to use large language models (LLMs) to construct a hierarchical framework that first plans and then writes. Despite the fact that the generated novels reach a sufficient length, they exhibit poor logical coherence and appeal in their plots and deficiencies in character and event depiction, ultimately compromising the overall narrative quality. In this paper, we propose a method named Extracting Excelsior and Expanding. Ex3 initially extracts structure information from raw novel data. By combining this structure information with the novel data, an instruction-following dataset is meticulously crafted. This dataset is then utilized to fine-tune the LLM, aiming for excelsior generation performance. In the final stage, a tree-like expansion method is deployed to facilitate the generation of arbitrarily long novels. Evaluation against previous methods showcases Ex3's ability to produce higher-quality long-form novels.

cs.CL↗

arXiv2024

A Local Gaussian Process Regression Approach to Frequency Response Function Estimation

Frequency response function (FRF) estimation is a classical subject in system identification. In the past two decades, there have been remarkable advances in developing local methods for this subject, e.g., the local polynomial method, local rational method, and iterative local rational method. The recent concentrations for local methods are two issues: the model order selection and the identification of lightly damped systems. To address these two issues, we propose a new local method called local Gaussian process regression (LGPR). We show that the frequency response function locally is either analytic or resonant, and this prior knowledge can be embedded into a kernel-based regularized estimate through a dot-product kernel plus a resonance kernel induced by a second-order resonant system. The LGPR provides a new route to tackle the aforementioned issues. In the numerical simulations, the LGPR shows the best FRF estimation accuracy compared with the existing local methods, and moreover, the LGPR is more robust with respect to sample size and noise level.

eess.SY↗

arXiv2024

A Class of Convex Optimization-Based Recursive Algorithms for Identification of Stochastic Systems

Focusing on identification, this paper develops a class of convex optimization-based criteria and correspondingly the recursive algorithms to estimate the parameter vector $\theta^{*}$ of a stochastic dynamic system. Not only do the criteria include the classical least-squares estimator but also the $L_l=|\cdot|^l, l\geq 1$, the Huber, the Log-cosh, and the Quantile costs as special cases. First, we prove that the minimizers of the convex optimization-based criteria converge to $\theta^{*}$ with probability one. Second, the recursive algorithms are proposed to find the estimates, which minimize the convex optimization-based criteria, and it is shown that these estimates also converge to the true parameter vector with probability one. Numerical examples are given, justifying the performance of the proposed algorithms including the strong consistency of the estimates, the robustness against outliers in the observations, and higher efficiency in online computation compared with the kernel-based regularization method due to the recursive nature.

math.OC↗

arXiv2023

On Kernel Design for Regularized Non-Causal System Identification

Through one decade's development, the kernel-based regularization method (KRM) has become a complement to the classical maximum likelihood/prediction error method and an emerging new system identification paradigm. One recent example is its application in the non-causal system identification, and the key issue lies in the design and analysis of kernels for non-causal systems. In this paper, we develop systematic ways to deal with this issue. In particular, we first introduce the guidelines for kernel design and then extend the system theoretic framework to design the so-called non-causal simulation-induced (NCSI) kernel, and we also study its structural properties, including stability and semiseparability. Finally, we consider some special cases of the NCSI kernel and show their advantage over the existing kernels through numerical simulations.

eess.SY↗

arXiv2023

Pushing the Limits of Machine Design: Automated CPU Design with AI

Design activity -- constructing an artifact description satisfying given goals and constraints -- distinguishes humanity from other animals and traditional machines, and endowing machines with design abilities at the human level or beyond has been a long-term pursuit. Though machines have already demonstrated their abilities in designing new materials, proteins, and computer programs with advanced artificial intelligence (AI) techniques, the search space for designing such objects is relatively small, and thus, "Can machines design like humans?" remains an open question. To explore the boundary of machine design, here we present a new AI approach to automatically design a central processing unit (CPU), the brain of a computer, and one of the world's most intricate devices humanity have ever designed. This approach generates the circuit logic, which is represented by a graph structure called Binary Speculation Diagram (BSD), of the CPU design from only external input-output observations instead of formal program code. During the generation of BSD, Monte Carlo-based expansion and the distance of Boolean functions are used to guarantee accuracy and efficiency, respectively. By efficiently exploring a search space of unprecedented size 10^{10^{540}}, which is the largest one of all machine-designed objects to our best knowledge, and thus pushing the limits of machine design, our approach generates an industrial-scale RISC-V CPU within only 5 hours. The taped-out CPU successfully runs the Linux operating system and performs comparably against the human-designed Intel 80486SX CPU. In addition to learning the world's first CPU only from input-output observations, which may reform the semiconductor industry by significantly reducing the design cycle, our approach even autonomously discovers human knowledge of the von Neumann architecture.

cs.AI↗

arXiv2023

Kernel-based Regularized Iterative Learning Control of Repetitive Linear Time-varying Systems

For data-driven iterative learning control (ILC) methods, both the model estimation and controller design problems are converted to parameter estimation problems for some chosen model structures. It is well-known that if the model order is not chosen carefully, models with either large variance or large bias would be resulted, which is one of the obstacles to further improve the modeling and tracking performances of data-driven ILC in practice. An emerging trend in the system identification community to deal with this issue is using regularization instead of the statistical tests, e.g., AIC, BIC, and one of the representatives is the so-called kernel-based regularization method (KRM). In this paper, we integrate KRM into data-driven ILC to handle a class of repetitive linear time-varying systems, and moreover, we show that the proposed method has ultimately bounded tracking error in the iteration domain. The numerical simulation results show that in contrast with the least squares method and some existing data-driven ILC methods, the proposed one can give faster convergence speed, better accuracy and robustness in terms of the tracking performance.

eess.SY↗

arXiv2022

On Embeddings and Inverse Embeddings of Input Design for Regularized System Identification

Input design is an important problem for system identification and has been well studied for the classical system identification, i.e., the maximum likelihood/prediction error method. For the emerging regularized system identification, the study on input design has just started, and it is often formulated as a non-convex optimization problem that minimizes a scalar measure of the Bayesian mean squared error matrix subject to certain constraints, and the state-of-art method is the so-called quadratic mapping and inverse embedding (QMIE) method, where a time domain inverse embedding (TDIE) is proposed to find the inverse of the quadratic mapping. In this paper, we report some new results on the embeddings/inverse embeddings of the QMIE method. Firstly, we present a general result on the frequency domain inverse embedding (FDIE) that is to find the inverse of the quadratic mapping described by the discrete-time Fourier transform. Then we show the relation between the TDIE and the FDIE from a graph signal processing perspective. Finally, motivated by this perspective, we further propose a graph induced embedding and its inverse, which include the previously introduced embeddings as special cases. This deepens the understanding of input design from a new viewpoint beyond the real domain and the frequency domain viewpoints.

eess.SY↗

arXiv2022

An Efficient Implementation for Spatial-Temporal Gaussian Process Regression and Its Applications

Spatial-temporal Gaussian process regression is a popular method for spatial-temporal data modeling. Its state-of-art implementation is based on the state-space model realization of the spatial-temporal Gaussian process and its corresponding Kalman filter and smoother, and has computational complexity $\mathcal{O}(NM^3)$, where $N$ and $M$ are the number of time instants and spatial input locations, respectively, and thus can only be applied to data with large $N$ but relatively small $M$. In this paper, our primary goal is to show that by exploring the Kronecker structure of the state-space model realization of the spatial-temporal Gaussian process, it is possible to further reduce the computational complexity to $\mathcal{O}(M^3+NM^2)$ and thus the proposed implementation can be applied to data with large $N$ and moderately large $M$. The proposed implementation is illustrated over applications in weather data prediction and spatially-distributed system identification. Our secondary goal is to design a kernel for both the Colorado precipitation data and the GHCN temperature data, such that while having more efficient implementation, better prediction performance can also be achieved than the state-of-art result.

eess.SY↗

arXiv2022

Asymptotic Theory for Regularized System Identification Part I: Empirical Bayes Hyper-parameter Estimator

Regularized system identification is the major advance in system identification in the last decade. Although many promising results have been achieved, it is far from complete and there are still many key problems to be solved. One of them is the asymptotic theory, which is about convergence properties of the model estimators as the sample size goes to infinity. The existing related results for regularized system identification are about the almost sure convergence of various hyper-parameter estimators. A common problem of those results is that they do not contain information on the factors that affect the convergence properties of those hyper-parameter estimators, e.g., the regression matrix. In this paper, we tackle problems of this kind for the regularized finite impulse response model estimation with the empirical Bayes (EB) hyper-parameter estimator and filtered white noise input. In order to expose and find those factors, we study the convergence in distribution of the EB hyper-parameter estimator, and the asymptotic distribution of its corresponding model estimator. For illustration, we run Monte Carlo simulations to show the efficacy of our obtained theoretical results.

eess.SY↗

arXiv2022

Identifiability Analysis of Noise Covariances for LTI Stochastic Systems with Unknown Inputs

Most existing works on optimal filtering of linear time-invariant (LTI) stochastic systems with arbitrary unknown inputs assume perfect knowledge of the covariances of the noises in the filter design. This is impractical and raises the question of whether and under what conditions one can identify the process and measurement noise covariances (denoted as $Q$ and $R$, respectively) of systems with unknown inputs. This paper considers the identifiability of $Q$/$R$ using the correlation-based measurement difference approach. More specifically, we establish (i) necessary conditions under which $Q$ and $R$ can be uniquely jointly identified; (ii) necessary and sufficient conditions under which $Q$ can be uniquely identified, when $R$ is known; (iii) necessary conditions under which $R$ can be uniquely identified, when $Q$ is known. It will also be shown that for achieving the results mentioned above, the measurement difference approach requires some decoupling conditions for constructing a stationary time series, which are proved to be sufficient for the well-known strong detectability requirements established by Hautus.

math.OC↗

arXiv2022

Real-Time Robust Video Object Detection System Against Physical-World Adversarial Attacks

DNN-based video object detection (VOD) powers autonomous driving and video surveillance industries with rising importance and promising opportunities. However, adversarial patch attack yields huge concern in live vision tasks because of its practicality, feasibility, and powerful attack effectiveness. This work proposes Themis, a software/hardware system to defend against adversarial patches for real-time robust video object detection. We observe that adversarial patches exhibit extremely localized superficial feature importance in a small region with non-robust predictions, and thus propose the adversarial region detection algorithm for adversarial effect elimination. Themis also proposes a systematic design to efficiently support the algorithm by eliminating redundant computations and memory traffics. Experimental results show that the proposed methodology can effectively recover the system from the adversarial attack with negligible hardware overhead.

cs.CV↗