arXiv subjects

Dmitry Yudin

Publications and source records attributed to Dmitry Yudin.

At least 37 records · Page 2Linked to original sources

arXiv2025

Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graph

Locating objects described in natural language presents a significant challenge for autonomous agents. Existing CLIP-based open-vocabulary methods successfully perform 3D object grounding with simple (bare) queries, but cannot cope with ambiguous descriptions that demand an understanding of object relations. To tackle this problem, we propose a modular approach called BBQ (Beyond Bare Queries), which constructs 3D scene graph representation with metric and semantic spatial edges and utilizes a large language model as a human-to-agent interface through our deductive scene reasoning algorithm. BBQ employs robust DINO-powered associations to construct 3D object-centric map and an advanced raycasting algorithm with a 2D vision-language model to describe them as graph nodes. On the Replica and ScanNet datasets, we have demonstrated that BBQ takes a leading place in open-vocabulary 3D semantic segmentation compared to other zero-shot methods. Also, we show that leveraging spatial relations is especially effective for scenes containing multiple entities of the same semantic class. On challenging Sr3D+, Nr3D and ScanRefer benchmarks, our deductive approach demonstrates a significant improvement, enabling objects grounding by complex queries compared to other state-of-the-art methods. The combination of our design choices and software implementation has resulted in significant data processing speed in experiments on the robot on-board computer. This promising performance enables the application of our approach in intelligent robotics projects. We made the code publicly available at https://linukc.github.io/BeyondBareQueries/.

cs.CV↗

arXiv2025

DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes

The analysis of events in dynamic environments poses a fundamental challenge in the development of intelligent agents and robots capable of interacting with humans. Current approaches predominantly utilize visual models. However, these methods often capture information implicitly from images, lacking interpretable spatial-temporal object representations. To address this issue we introduce DyGEnc - a novel method for Encoding a Dynamic Graph. This method integrates compressed spatial-temporal structural observation representation with the cognitive capabilities of large language models. The purpose of this integration is to enable advanced question answering based on a sequence of textual scene graphs. Extended evaluations on the STAR and AGQA datasets indicate that DyGEnc outperforms existing visual methods by a large margin of 15-25% in addressing queries regarding the history of human-to-object interactions. Furthermore, the proposed method can be seamlessly extended to process raw input images utilizing foundational models for extracting explicit textual scene graphs, as substantiated by the results of a robotic experiment conducted with a wheeled manipulator platform. We hope that these findings will contribute to the implementation of robust and compressed graph-based robotic memory for long-horizon reasoning. Code is available at github.com/linukc/DyGEnc.

cs.CV↗

arXiv2025

PRISM-TopoMap: Online Topological Mapping with Place Recognition and Scan Matching

Mapping is one of the crucial tasks enabling autonomous navigation of a mobile robot. Conventional mapping methods output a dense geometric map representation, e.g. an occupancy grid, which is not trivial to keep consistent for prolonged runs covering large environments. Meanwhile, capturing the topological structure of the workspace enables fast path planning, is typically less prone to odometry error accumulation, and does not consume much memory. Following this idea, this paper introduces PRISM-TopoMap -- a topological mapping method that maintains a graph of locally aligned locations not relying on global metric coordinates. The proposed method involves original learnable multimodal place recognition paired with the scan matching pipeline for localization and loop closure in the graph of locations. The latter is updated online, and the robot is localized in a proper node at each time step. We conduct a broad experimental evaluation of the suggested approach in a range of photo-realistic environments and on a real robot, and compare it to state of the art. The results of the empirical evaluation confirm that PRISM-Topomap consistently outperforms competitors computationally-wise, achieves high mapping quality and performs well on a real robot. The code of PRISM-Topomap is open-sourced and is available at: https://github.com/kirillMouraviev/prism-topomap.

cs.RO↗

arXiv2024

MSSPlace: Multi-Sensor Place Recognition with Visual and Text Semantics

Place recognition is a challenging task in computer vision, crucial for enabling autonomous vehicles and robots to navigate previously visited environments. While significant progress has been made in learnable multimodal methods that combine onboard camera images and LiDAR point clouds, the full potential of these methods remains largely unexplored in localization applications. In this paper, we study the impact of leveraging a multi-camera setup and integrating diverse data sources for multimodal place recognition, incorporating explicit visual semantics and text descriptions. Our proposed method named MSSPlace utilizes images from multiple cameras, LiDAR point clouds, semantic segmentation masks, and text annotations to generate comprehensive place descriptors. We employ a late fusion approach to integrate these modalities, providing a unified representation. Through extensive experiments on the Oxford RobotCar and NCLT datasets, we systematically analyze the impact of each data source on the overall quality of place descriptors. Our experiments demonstrate that combining data from multiple sensors significantly improves place recognition model performance compared to single modality approaches and leads to state-of-the-art quality. We also show that separate usage of visual or textual semantics (which are more compact representations of sensory data) can achieve promising results in place recognition. The code for our method is publicly available: https://github.com/alexmelekhin/MSSPlace

cs.CV↗

arXiv2024

uSF: Learning Neural Semantic Field with Uncertainty

Recently, there has been an increased interest in NeRF methods which reconstruct differentiable representation of three-dimensional scenes. One of the main limitations of such methods is their inability to assess the confidence of the model in its predictions. In this paper, we propose a new neural network model for the formation of extended vector representations, called uSF, which allows the model to predict not only color and semantic label of each point, but also estimate the corresponding values of uncertainty. We show that with a small number of images available for training, a model quantifying uncertainty performs better than a model without such functionality. Code of the uSF approach is publicly available at https://github.com/sevashasla/usf/.

cs.CV↗

arXiv2024

OFMPNet: Deep End-to-End Model for Occupancy and Flow Prediction in Urban Environment

The task of motion prediction is pivotal for autonomous driving systems, providing crucial data to choose a vehicle behavior strategy within its surroundings. Existing motion prediction techniques primarily focus on predicting the future trajectory of each agent in the scene individually, utilizing its past trajectory data. In this paper, we introduce an end-to-end neural network methodology designed to predict the future behaviors of all dynamic objects in the environment. This approach leverages the occupancy map and the scene's motion flow. We are investigatin various alternatives for constructing a deep encoder-decoder model called OFMPNet. This model uses a sequence of bird's-eye-view road images, occupancy grid, and prior motion flow as input data. The encoder of the model can incorporate transformer, attention-based, or convolutional units. The decoder considers the use of both convolutional modules and recurrent blocks. Additionally, we propose a novel time-weighted motion flow loss, whose application has shown a substantial decrease in end-point error. Our approach has achieved state-of-the-art results on the Waymo Occupancy and Flow Prediction benchmark, with a Soft IoU of 52.1% and an AUC of 76.75% on Flow-Grounded Occupancy.

cs.CV↗

arXiv2024

Fractional Marcus-Hush-Chidsey-Yakopcic current-voltage model for redox-based resistive memory devices

We propose a circuit-level model combining the Marcus-Hush-Chidsey electron current equation and the Yakopcic equation for the state variable for describing resistive switching memory devices of the structure metal-ionic conductor-metal. We extend the dynamics of the state variable originally described by a first-order time derivative by introducing a fractional derivative with an arbitrary order between zero and one. We show that the extended model fits with great fidelity the current-voltage characteristic data obtained on a Si electrochemical metallization memory device with Ag-Cu alloy.

cond-mat.mtrl-sci↗

arXiv2023

Interactive Semantic Map Representation for Skill-based Visual Object Navigation

Visual object navigation using learning methods is one of the key tasks in mobile robotics. This paper introduces a new representation of a scene semantic map formed during the embodied agent interaction with the indoor environment. It is based on a neural network method that adjusts the weights of the segmentation model with backpropagation of the predicted fusion loss values during inference on a regular (backward) or delayed (forward) image sequence. We have implemented this representation into a full-fledged navigation approach called SkillTron, which can select robot skills from end-to-end policies based on reinforcement learning and classic map-based planning methods. The proposed approach makes it possible to form both intermediate goals for robot exploration and the final goal for object navigation. We conducted intensive experiments with the proposed approach in the Habitat environment, which showed a significant superiority in navigation quality metrics compared to state-of-the-art approaches. The developed code and used custom datasets are publicly available at github.com/AIRI-Institute/skill-fusion.

cs.RO↗

arXiv2023

SegmATRon: Embodied Adaptive Semantic Segmentation for Indoor Environment

This paper presents an adaptive transformer model named SegmATRon for embodied image semantic segmentation. Its distinctive feature is the adaptation of model weights during inference on several images using a hybrid multicomponent loss function. We studied this model on datasets collected in the photorealistic Habitat and the synthetic AI2-THOR Simulators. We showed that obtaining additional images using the agent's actions in an indoor environment can improve the quality of semantic segmentation. The code of the proposed approach and datasets are publicly available at https://github.com/wingrune/SegmATRon.

cs.CV↗

arXiv2023

Time-space bi-fractional drift-diffusion equation for anomalous electrochemical transport

The Debye-Falkenhagen differential equation is commonly used as a mean-field macroscopic model for describing electrochemical ionic drift and diffusion in dilute binary electrolytes when subjected to a suddenly applied potential smaller than the thermal voltage. However, the ionic transport in most electrochemical systems, such as electrochemical capacitors, permeation through membranes, biosensors and capacitive desalination, the electrolytic medium is interfaced with porous, disordered, and fractal materials which makes the modeling of electrodiffusive transport with the simple planar electrode theory limited. Here we study a possible generalization of the traditional drift-diffusion equation of Debye and Falkenhagen by incorporating both fractional time and space derivatives for the charge density. The nonlocal (global) fractional time derivative takes into account the past dynamics of the variable such as charge trapping effects and thus subdiffusive transport, while the fractional space derivative allows to simulate superdiffusive transport.

physics.class-ph↗

arXiv2023

Tensor train optimization of parametrized quantum circuits

We examine a particular realization of derivative-free method as implemented on tensor train based optimization to the variational quantum eigensolver. As an example, we consider parametrized quantum circuits composed of a low-depth hardware-efficient ansatz and Hamiltonian variational ansatz for addressing the ground state of the transverse field Ising model. We further make a comparison with gradient-based optimization techniques and discuss on the advantage of using tensor train based optimization, especially in the presence of noise.

quant-ph↗

arXiv2023

A superstatistics approach to the modelling of memristor current-voltage responses

Memristors are expected to form a major cornerstone in the upcoming renaissance of analog computing, owing to their very small spatial footprint and low power consumption. Due to the nature of their structure and operation, the response of a memristor is intrinsically tied to local variabilities in the device. This characteristic is amplified by currently employed semiconductor fabrication processes, which introduce spatial inhomogeneities into the structural fabric that makes up the layers of memristors. In this work, we propose a novel q-deformed current-voltage model for memristors based on the superstatistics framework, which allows the description of system-level responses while taking local variabilities into account. Applied on a Ag-Cu based synaptic memory cell, we demonstrate that our model has a 4-14% lower error than currently used models. Additionally, we show how the resulting q-parameter can be used to make statements about the internal makeup of the memristor, giving insights to spatial inhomogeneities and quality control.

cond-mat.mes-hall↗

arXiv2023

Rethinking Voxelization and Classification for 3D Object Detection

The main challenge in 3D object detection from LiDAR point clouds is achieving real-time performance without affecting the reliability of the network. In other words, the detecting network must be confident enough about its predictions. In this paper, we present a solution to improve network inference speed and precision at the same time by implementing a fast dynamic voxelizer that works on fast pillar-based models in the same way a voxelizer works on slow voxel-based models. In addition, we propose a lightweight detection sub-head model for classifying predicted objects and filter out false detected objects that significantly improves model precision in a negligible time and computing cost. The developed code is publicly available at: https://github.com/YoushaaMurhij/RVCDet.

cs.CV↗

arXiv2022

HPointLoc: Point-based Indoor Place Recognition using Synthetic RGB-D Images

We present a novel dataset named as HPointLoc, specially designed for exploring capabilities of visual place recognition in indoor environment and loop detection in simultaneous localization and mapping. The loop detection sub-task is especially relevant when a robot with an on-board RGB-D camera can drive past the same place (``Point") at different angles. The dataset is based on the popular Habitat simulator, in which it is possible to generate photorealistic indoor scenes using both own sensor data and open datasets, such as Matterport3D. To study the main stages of solving the place recognition problem on the HPointLoc dataset, we proposed a new modular approach named as PNTR. It first performs an image retrieval with the Patch-NetVLAD method, then extracts keypoints and matches them using R2D2, LoFTR or SuperPoint with SuperGlue, and finally performs a camera pose optimization step with TEASER++. Such a solution to the place recognition problem has not been previously studied in existing publications. The PNTR approach has shown the best quality metrics on the HPointLoc dataset and has a high potential for real use in localization systems for unmanned vehicles. The proposed dataset and framework are publicly available: https://github.com/metra4ok/HPointLoc.

cs.CV↗

arXiv2022

Quantum-machine-learning channel discrimination

In the problem of quantum channel discrimination, one distinguishes between a given number of quantum channels, which is done by sending an input state through a channel and measuring the output state. This work studies applications of variational quantum circuits and machine learning techniques for discriminating such channels. In particular, we explore (i) the practical implementation of embedding this task into the framework of variational quantum computing, (ii) training a quantum classifier based on variational quantum circuits, and (iii) applying the quantum kernel estimation technique. For testing these three channel discrimination approaches, we considered a pair of entanglement-breaking channels and the depolarizing channel with two different depolarization factors. For the approach (i), we address solving the quantum channel discrimination problem using widely discussed parallel and sequential strategies. We show the advantage of the latter in terms of better convergence with less quantum resources. Quantum channel discrimination with a variational quantum classifier (ii) allows one to operate even with random and mixed input states and simple variational circuits. The kernel-based classification approach (iii) is also found effective as it allows one to discriminate depolarizing channels associated not with just fixed values of the depolarization factor, but with ranges of it. Additionally, we discovered that a simple modification of one of the commonly used kernels significantly increases the efficiency of this approach. Finally, our numerical findings reveal that the performance of variational methods of channel discrimination depends on the trace of the product of the output states. These findings demonstrate that quantum machine learning can be used to discriminate channels, such as those representing physical noise processes.

quant-ph↗

arXiv2021

Numerical hardware-efficient variational quantum simulation of a soliton solution

Implementing variational quantum algorithms with noisy intermediate-scale quantum machines of up to a hundred qubits is nowadays considered as one of the most promising routes towards achieving a quantum practical advantage. In multiqubit circuits, running advanced quantum algorithms is hampered by the noise inherent to quantum gates which distances us from the idea of universal quantum computing. Based on a one-dimensional quantum spin chain with competing symmetric and asymmetric pairwise exchange interactions, herein we discuss the capabilities of quantum algorithms with special attention paid to a hardware-efficient variational eigensolver. A delicate interplay between magnetic interactions allows one to stabilize a chiral state that destroys the homogeneity of magnetic ordering, thus making this solution highly entangled. Quantifying entanglement in terms of quantum concurrence, we argue that, while being capable of correctly reproducing a uniform magnetic configuration, the hardware-efficient ansatz meets difficulties in providing a detailed description to a noncollinear magnetic structure. The latter naturally limits the application range of variational quantum computing to solve quantum simulation tasks.

quant-ph↗

arXiv2021

Real-time 3D Object Detection using Feature Map Flow

In this paper, we present a real-time 3D detection approach considering time-spatial feature map aggregation from different time steps of deep neural model inference (named feature map flow, FMF). Proposed approach improves the quality of 3D detection center-based baseline and provides real-time performance on the nuScenes and Waymo benchmark. Code is available at https://github.com/YoushaaMurhij/FMFNet

cs.CV↗

arXiv2021

On the origin of electron accumulation layer at clean InAs(111) surfaces

In this paper, we provide a comprehensive theoretical analysis of the electronic structure of InAs(111) surfaces with a special attention paid to the energy region close to the fundamental bandgap. Starting from the bulk electronic structure of InAs as calculated using PBE functional with included Hubbard correction and spin-orbit coupling, we deliver proper values for the bandgap, split-off energy, as well as effective electron, light- and heavy-hole masses in full consistency with available experimental results. On the basis of optimized atomic surfaces we recover scanning tunneling microscopy images, which being supplied with accessible experimental data make it possible to speculate on the formation of electron accumulation layer for both As- and In-terminated InAs(111) surfaces. Moreover, these results are accompanied by band structure simulations of conduction band states.

cond-mat.mtrl-sci↗