Search arXiv⌕ Search

arXiv subjects

Sanghyuk Lee

Publications and source records attributed to Sanghyuk Lee.

At least 37 records · Page 2Linked to original sources
arXiv2023

FP8 versus INT8 for efficient deep learning inference

Recently, the idea of using FP8 as a number format for neural network training has been floating around the deep learning world. Given that most training is currently conducted with entire networks in FP32, or sometimes FP16 with mixed-precision, the step to having some parts of a network run in FP8 with 8-bit weights is an appealing potential speed-up for the generally costly and time-intensive training procedures in deep learning. A natural question arises regarding what this development means for efficient inference on edge devices. In the efficient inference device world, workloads are frequently executed in INT8. Sometimes going even as low as INT4 when efficiency calls for it. In this whitepaper, we compare the performance for both the FP8 and INT formats for efficient on-device inference. We theoretically show the difference between the INT and FP formats for neural networks and present a plethora of post-training quantization and quantization-aware-training results to show how this theory translates to practice. We also provide a hardware analysis showing that the FP formats are somewhere between 50-180% less efficient in terms of compute in dedicated hardware than the INT format. Based on our research and a read of the research field, we conclude that although the proposed FP8 format could be good for training, the results for inference do not warrant a dedicated implementation of FP8 in favor of INT8 for efficient inference. We show that our results are mostly consistent with previous findings but that important comparisons between the formats have thus far been lacking. Finally, we discuss what happens when FP8-trained networks are converted to INT8 and conclude with a brief discussion on the most efficient way for on-device deployment and an extensive suite of INT8 results for many models.

cs.LG↗
arXiv2023

Sharp Sobolev regularity of restricted X-ray transforms

We study $L^p$-Sobolev regularity estimate for the restricted X-ray transforms generated by nondegenerate curves. Making use of the inductive strategy in the recent work by the authors, we establish the sharp $L^p$-regularity estimates for the restricted X-ray transforms in $\mathbb R^{d+1}$, $d\ge 3$. This extends the result due to Pramanik and Seeger in $\mathbb R^3$ to every dimension.

math.CA↗
arXiv2023

A Fusion Model: Towards a Virtual, Physical and Cognitive Integration and its Principles

Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), digital twin, Metaverse and other related digital technologies have attracted much attention in recent years. These new emerging technologies are changing the world significantly. This research introduces a fusion model, i.e. Fusion Universe (FU), where the virtual, physical, and cognitive worlds are merged together. Therefore, it is crucial to establish a set of principles for the fusion model that is compatible with our physical universe laws and principles. This paper investigates several aspects that could affect immersive and interactive experience; and proposes the fundamental principles for Fusion Universe that can integrate physical and virtual world seamlessly.

cs.AI↗
arXiv2023

Uniform stationary phase estimate with limited smoothness

In this paper, we consider the uniform estimate for the oscillatory integral with stationary phase, which was previously studied by Alazard-Burq-Zuily. We significantly reduce the order of required regularity condition on the phase and amplitude functions for the uniform estimate. We also study estimates for the oscillatory integrals of which phase and amplitude functions depend on the oscillation parameter. The novelty of this article lies in the use of the wave packet decomposition, which transforms the decay estimate for the oscillatory integral to the disjointness property of the supports of wave packets. The latter is geometric in its nature and less sensitive to the smoothness of the phase and amplitude functions.

math.CA↗
arXiv2023

Task-Adaptive Pseudo Labeling for Transductive Meta-Learning

Meta-learning performs adaptation through a limited amount of support set, which may cause a sample bias problem. To solve this problem, transductive meta-learning is getting more and more attention, going beyond the conventional inductive learning perspective. This paper proposes so-called task-adaptive pseudo labeling for transductive meta-learning. Specifically, pseudo labels for unlabeled query sets are generated from labeled support sets through label propagation. Pseudo labels enable to adopt the supervised setting as it is and also use the unlabeled query set in the adaptation process. As a result, the proposed method is able to deal with more examples in the adaptation process than inductive ones, which can result in better classification performance of the model. Note that the proposed method is the first approach of applying task adaptation to pseudo labeling. Experiments show that the proposed method outperforms the state-of-the-art (SOTA) technique in 5-way 1-shot few-shot classification.

cs.LG↗
arXiv2023

Almost everywhere convergence of Bochner--Riesz means for the twisted Laplacian

Let $\mathcal L$ denote the twisted Laplacian in $\mathbb C^d$. We study almost everywhere convergence of the Bochner--Riesz mean $S^δ_{t}(\mathcal L) f$ of $f\in L^p(\mathbb C^d)$ as $t\to \infty$, which is an expansion of $f$ in the special Hermite functions. For $2\le p\le \infty$, we obtain the sharp range of the summability indices $δ$ for which the convergence of $S^δ_{t}(\mathcal L) f$ holds for all $f\in L^p(\mathbb C^d)$.

math.CA↗
arXiv2022

Bilinear maximal functions associated with degenerate surfaces

We study $L^{p}\times L^{q}\rightarrow L^{r}$-boundedness of (sub)bilinear maximal functions associated with degenerate hypersurfaces. First, we obtain the maximal bound on the sharp range of exponents $p,q,r$ (except some border line cases) for the bilinear maximal functions given by the model surface $\big\{(y,z)\in\mathbb{R}^{n}\times \mathbb{R}^{n}:|y|^{l_{1}}+|z|^{l_{2}}=1\big\}$, $(l_{1},l_{2})\in [1,\infty)^2$, $n\ge 2$. Our result manifests that nonvanishing Gaussian curvature is not good enough, in contrast with $L^p$-boundedness of the (sub)linear maximal operator associated to hypersurfaces, to characterize the best possible maximal boundedness. Secondly, we consider the bilinear maximal function associated to the finite type curve in $\mathbb R^2$ and obtain a complete characterization of the maximal bound. We also prove multilinear generalizations of the aforementioned results.

math.CA↗
arXiv2022

$L^p$ maximal bound and Sobolev regularity of two-parameter averages over tori

We investigate $L^p$ boundedness of the maximal function defined by the averaging operator $f\to \mathcal{A}_t^s f$ over the two-parameter family of tori $\mathbb{T}_t^{s}:=\{ ( (t+s\cosθ)\cosϕ,\,(t+s\cosθ)\sinϕ,\, s\sinθ): θ, ϕ\in [0,2π) \}$ with $c_0t>s>0$ for some $c_0\in (0,1)$. We prove that the associated (two-parameter) maximal function is bounded on $L^p$ if and only if $p>2$. We also obtain $L^p$--$L^q$ estimates for the local maximal operator on a sharp range of $p,q$. Furthermore, the sharp smoothing estimates are proved including the sharp local smoothing estimates for the operators $f\to \mathcal A_t^s f$ and $f\to \mathcal A_t^{c_0t} f$. For the purpose, we make use of Bourgain--Demeter's decoupling inequality for the cone and Guth--Wang--Zhang's local smoothing estimates for the $2$ dimensional wave operator.

math.CA↗
arXiv2022

Bounds on the Hermite spectral projection operator

We study $L^p$-$L^q$ bounds on the spectral projection operator $Π_λ$ associated to the Hermite operator $H=|x|^2-Δ$ in $\mathbb R^d$. We are mainly concerned with a localized operator $χ_EΠ_λχ_E$ for a subset $E\subset\mathbb R^d$ and undertake the task of characterizing the sharp $L^p$--$L^q$ bounds. We obtain sharp bounds in extended ranges of $p,q$. First, we provide a complete characterization of the sharp $L^p$--$L^q$ bounds when $E$ is away from $\sqrtλ\mathbb S^{d-1}$. Secondly, we obtain the sharp bounds as the set $E$ gets close to $\sqrtλ\mathbb S^{d-1}$. Thirdly, we extend the range of $p,q$ for which the operator $Π_λ$ is uniformly bounded from $L^p(\mathbb R^d)$ to $L^q(\mathbb R^d)$.

math.CA↗
arXiv2022

Strichartz and uniform Sobolev inequalities for the elastic wave equation

We prove dispersive estimate for the elastic wave equation by which we extend the known Strichartz estimates for the classical wave equation to those for the elastic wave equation. In particular, the endpoint Strichartz estimates are deduced. For the purpose we diagonalize the symbols of the Lamé operator and its semigroup, which also gives an alternative and simpler proofs of the previous results on perturbed elastic wave equations. Furthermore, we obtain uniform Sobolev inequalities for the elastic wave operator.

math.AP↗
arXiv2022

Carleman inequalities and unique continuation for the polyharmonic operators

We obtain a complete characterization of $L^p-L^q$ Carleman estimates with weight $e^{v\cdot x}$ for the polyharmonic operators. Our result extends the Carleman inequalities for the Laplacian due to Kenig--Ruiz--Sogge. Consequently, we obtain new unique continuation properties of higher order Schrödinger equations relaxing the integrability assumption on the solution spaces.

math.AP↗
arXiv2022

Remarks on dimension of unions of curves

We study an analogue of Marstrand's circle packing problem for curves in higher dimensions. We consider collections of curves which are generated by translation and dilation of a curve $γ$ in $\mathbb R^d$, i.e., $ x + t γ$, $(x,t) \in \mathbb R^d \times (0,\infty)$. For a Borel set $F \subset \mathbb R^d\times (0,\infty)$, we show the unions of curves $\bigcup_{(x,t) \in F} ( x+tγ)$ has Hausdorff dimension at least $α+1$ whenever $F$ has Hausdorff dimension bigger than $α$, $α\in (0, d-1)$. We also obtain results for unions of curves generated by multi-parameter dilation of $γ$. One of the main ingredients is a local smoothing type estimate (for averages over curves) relative to fractal measures.

math.CA↗
arXiv2022

Circular average relative to fractal measures

We prove new $L^p$- $L^q$ estimates for averages over dilates of the circle with respect to $α$-dimensional fractal measure, which unify different types of maximal estimates for the circular average. Our results are consequences of $L^p$- $L^q$ smoothing estimates for the wave operator relative to fractal measures. We also discuss similar results concerning the spherical averages.

math.CA↗
arXiv2022

Pointwise convergence of sequential Schrödinger means

We study pointwise convergence of the fractional Schrödinger means along sequences $t_n$ which converge to zero. Our main result is that bounds on the maximal function $\sup_{n} |e^{it_n(-Δ)^{α/2}} f| $ can be deduced from those on $\sup_{0<t\le 1} |e^{it(-Δ)^{α/2}} f|$ when $\{t_n\}$ is contained in the Lorentz space $\ell^{r,\infty}$. Consequently, our results provide seemingly optimal results in higher dimensions, which extend the recent work of Dimou-Seeger, and Li-Wang-Yan to higher dimensions. Our approach based on a localization argument also works for other dispersive equations and provides alternative proofs of previous results on sequential convergence.

math.CA↗
arXiv2022

Unique continuation for the heat operator with potentials in weak spaces

We prove strong unique continuation property for the differential inequality $|(\partial_t +Δ)u(x,t)|\le V(x,t)|u(x,t)|$ with $V$ contained in weak spaces. In particular, we establish the strong unique continuation property for $V\in L^\infty_t L^{d/2,\infty}_x$, which has been left open since the works of Escauriaza [6] and Escauriaza-Vega [8]. Our results are consequences of the Carleman estimates for the heat operator in the Lorentz spaces.

math.AP↗
arXiv2022

Sharp smoothing properties of averages over curves

We prove sharp smoothing properties of the averaging operator defined by convolution with a measure on a smooth nondegenerate curve $γ$ in $\mathbb R^d$, $d\ge 3$. Despite the simple geometric structure of such curves, the sharp smoothing estimates have remained largely unknown except for those in low dimensions. Devising a novel inductive strategy, we obtain the optimal $L^p$ Sobolev regularity estimates, which settle the conjecture raised by Beltran-Guo-Hickman-Seeger. Besides, we show the sharp local smoothing estimates for every $d$. As a result, we establish, for the first time, nontrivial $L^p$ boundedness of the maximal average over dilations of $γ$ for $d\ge 4$.

math.CA↗
arXiv2021

Contextual Gradient Scaling for Few-Shot Learning

Model-agnostic meta-learning (MAML) is a well-known optimization-based meta-learning algorithm that works well in various computer vision tasks, e.g., few-shot classification. MAML is to learn an initialization so that a model can adapt to a new task in a few steps. However, since the gradient norm of a classifier (head) is much bigger than those of backbone layers, the model focuses on learning the decision boundary of the classifier with similar representations. Furthermore, gradient norms of high-level layers are small than those of the other layers. So, the backbone of MAML usually learns task-generic features, which results in deteriorated adaptation performance in the inner-loop. To resolve or mitigate this problem, we propose contextual gradient scaling (CxGrad), which scales gradient norms of the backbone to facilitate learning task-specific knowledge in the inner-loop. Since the scaling factors are generated from task-conditioned parameters, gradient norms of the backbone can be scaled in a task-wise fashion. Experimental results show that CxGrad effectively encourages the backbone to learn task-specific knowledge in the inner-loop and improves the performance of MAML up to a significant margin in both same- and cross-domain few-shot classification.

cs.CV↗