Product Costing in Data Pipelines

Abstract

Large data-intensive enterprises spend millions of dollars monthly running data pipelines. Apportioning this cost back to individual products, known as product costing, is a prerequisite for data pricing and cost recovery. However, this process is fundamentally more challenging than physical manufacturing costing due to the unique characteristics of data pipelines, including the non-rivalrous nature of data that complicates the costing formulation, the prevalence of cyclic dependencies that distort the accuracy of results, and the massive scale of pipelines that leads to computational intractability. In this paper, for the first time, we formalize the problem of product costing in data pipelines as a manufacturing cost of data products (MCDP) problem to overcome the non-rivalrous nature of data. To solve MCDP, we propose CostApp, a hybrid algorithm that achieves both exactness and scalability, even for cyclic and massive-scale pipelines. CostApp performs a dimensionality reduction by isolating cyclic dependencies via a feedback arc set (FAS). It aggregates costs onto a compact set of key vertices using a scalable iterative approach, transforming the intractable computational problem into a reduced linear system that can be solved exactly and efficiently by a basic matrix-based method. Experiments on both real-world data and synthetic data show that CostApp scales to process a production pipeline with over 28 million entities within an hour while preserving exactness, supporting pipelines up to 10^2 larger than the direct matrix-based baseline.