What You See Is What You Get: Observation-Aligned Supervision for Chart-to-Code Generation

Abstract

Chart-to-code generation is commonly trained through supervised fine-tuning on reference plotting scripts, implicitly treating the gold code as a fully observable target. However, many chart programs contain latent variables that cannot be uniquely recovered from the rendered image. We identify this latent-observation mismatch in four forms across five chart types: aggregation-induced mismatch, where raw samples are reduced to box statistics or histogram bin masses; normalization-induced mismatch, where absolute scale is removed in pie charts; projection-induced mismatch, where 3D information is lost through 2D rendering; and level-set-induced mismatch, where a scalar field is observable only through selected contour lines. These mismatches introduce target ambiguity and require models to generate information unsupported by the image. We propose Observation-Aligned Supervision, which replaces latent variables with visually constrained quantities. We instantiate it using box statistics, bin weights, and wedge proportions, and study 3D scatter and contour charts through controlled experiments. Across multiple VLMs, observation-aligned supervision generally improves observable-value recovery in both-executable evaluations and mostly improves end-to-end recovery, while the contour study reveals a trade off between observation alignment and representational compactness.