Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.32311

GeoLEON-CLIP: Graph-Anchored Multimodal Contrastive Learning for Geographic Region Representation

Radoslaw Malek·Szymon Soltysiak·Aleksandra Kawala-Sterniuk·Piotr Szymanski
View original source ↗Read source document ↗Export metadata ↓

Abstract

Geographic regions are multimodal: satellite imagery captures visual morphology, text conveys human-interpretable semantics, and map structures describe urban context. Existing geospatial contrastive methods typically align image-text or image-location pairs, underusing structured map data. We introduce GeoLEON-CLIP, a graph-anchored contrastive framework that aligns three views of each H3-indexed region: an OpenStreetMap-derived graph embedding, a satellite image crop, and a textual description. Frozen pretrained LEON graph embeddings provide a semantic anchor between vision and language. Frozen graph, image, and text encoders are connected by trainable residual projection heads and trained with a symmetric tri-modal contrastive objective. Experiments show strong graph-image and graph-text alignment; graph-image retrieval achieves the best cross-modal performance in a mixed-resolution pool. To evaluate the anchoring hypothesis, we propose chained cross-modal recall $\mathrm{R}@A@B$, which routes retrieval through the graph modality and compares it with direct retrieval under a matched candidate budget. At $\mathrm{R}@1@1$, image$\rightarrow$graph$\rightarrow$text and text$\rightarrow$graph$\rightarrow$image outperform direct image-text retrieval by 4.8 and 4.4 points, respectively, despite using a single intermediate cell. Downstream probes show task-dependent gains, with the largest improvements on Airbnb and Beijing housing. These results indicate that OpenStreetMap graph structure can anchor multimodal representations combining maps, imagery, and language.

Explore related subjects

cs.CVcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Radoslaw Malek, Szymon Soltysiak, Aleksandra Kawala-Sterniuk, Piotr Szymanski. 2026-10-23. GeoLEON-CLIP: Graph-Anchored Multimodal Contrastive Learning for Geographic Region Representation. https://doi.org/10.1145/3849726.3856091

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Long-Tailed 3D Detection via Multi-Modal Fusion

Contemporary autonomous vehicle (AV) benchmarks have significantly advanced multimodal (LiDAR+RGB) 3D detection. However, despite the naturally long-tailed distribution of object classes, existing benchmarking protocols primarily focus on frequent categories (e.g., pedestrian and car), largely overlooking rare but safety-critical classes such as stroller and emergency vehicle. In practice, reliable detection of both common and rare classes is essential for safe autonomous driving. We formalize this problem as Long-Tailed 3D Detection (LT3D), where evaluation encompasses all annotated classes, including rare ones. To address LT3D, we introduce hierarchical losses that promote feature sharing across classes, diagnostic metrics that assign partial credit to semantically reasonable mistakes with respect to the semantic hierarchy (e.g., confusing a child with an adult), and a multimodal late-fusion (MMLF) framework to fuse detections. In particular, we show that rare-class accuracy benefits substantially from MMLF of independently trained uni-modal LiDAR and RGB detectors. Because of the modular design, unlike prevailing end-to-end trained multi-modal detectors that require paired LiDAR-RGB data, MMLF enables the use of advanced unimodal detectors that are trained on large-scale uni-modal datasets with sufficient data for rare classes. Lastly, we examine three fundamental design choices in MMLF, including the RGB detector representation (2D vs. 3D), cross-modal association (3D vs. image plane), and fusion strategy. We find that 2D RGB detectors recognize rare classes more reliably than 3D RGB detectors, image-plane association is more robust to depth estimation errors, and probabilistic score-calibrated fusion consistently yields the best performance. Extensive experiments on nuScenes and Argoverse2 demonstrate substantial improvements of MMLF, establishing a new state of the art.

cs.CV↗
arXiv2026

STATIC: Surface-normal-guided Two-branch Alignment and Temporal Integration for Consistent Video Depth Estimation

Video depth estimation is essential for applications such as autonomous driving, AR/VR, and robotics. Recent transformer-based single-image monocular depth estimation models perform well on a single image but struggle with depth consistency across video frames. Traditional methods aim to improve temporal consistency using multi-frame temporal modules or explicit priors like optical flow and camera parameters. However, these approaches face issues such as high memory use, reduced performance with dynamic or irregular motion, and a heavy reliance on the accuracy of external motion estimators. We propose STATIC, a novel model that independently learns temporal consistency in geometrically stable and unstable regions using only two adjacent frames. By leveraging intrinsic geometric cues rather than external motion priors, STATIC generates a depth-derived geometry-aware difference mask from surface normals to identify regions with inter-frame geometric variation by measuring directional variance. For geometrically stable regions, the Masked Stable (MS) module enhances temporal consistency by focusing on structurally stable regions. For geometrically unstable regions, the Surface Normal Similarity (SNS) module models inter-frame correspondences and ensures temporal consistency by measuring feature similarities between frames. A final refinement integrates the independently learned stable and unstable features, enabling STATIC to improve consistent depth prediction across consecutive frames. Experiments on KITTI, NYUv2, and Sintel demonstrate strong depth accuracy and temporal consistency, as well as stable performance across the tested frame-rate and camera-motion conditions.

cs.CV↗
arXiv2026

Hierarchical Vision Transformer with Prototypes for Interpretable Medical Image Classification

Explainability is an important requirement for machine learning models in high-risk domains such as medicine. We introduce HierViT, a hierarchical Vision Transformer that combines competitive predictive performance with multiple mechanisms to support the interpretation of its predictions. The model first predicts clinically relevant visual appearance attributes, which are then used to infer the final target prediction through a hierarchical reasoning process. To facilitate understanding of the intermediate representations, attribute-specific prototypes are visualized as representative image examples, while attribute-specific attention maps highlight image regions associated with each visual feature. Together, these attribute-level components provide complementary views of the model's reasoning by linking representative image examples and spatial attention to semantically defined visual features. We evaluate HierViT on two medical benchmark datasets: LIDC-IDRI for lung nodule assessment and derm7pt for skin lesion classification. Across both datasets, HierViT achieves competitive predictive performance while providing attribute-level prototypes and attention visualizations that support the interpretation of its predictions.

cs.CV↗