Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2611.06281

Building a Method: Challenges and Lessons from Developing an Impact Evaluation Framework for Public Investment in Open Source Software

Laia Domenech Burin
View original source ↗Read source document ↗Export metadata ↓

Abstract

Modern societies and institutions depend on an invisible layer of open digital infrastructure: foundational, openly accessible digital technologies that support a broad range of social and economic activities. Open source software is a core component of this infrastructure, underpinning everyday technologies and critical systems in various critical sectors. As open source software becomes increasingly recognised, funded, and governed as open digital infrastructure, policymakers face growing demands for accountability, transparency, and credible evidence of impact. However, methods for evaluating public investment in open source software remain underdeveloped, with no widely accepted methodological standard. This paper describes the Sovereign Tech Agency's transition from qualitative reporting and funding-recipient self-assessment towards a mixed-methods approach that incorporates quantitative, goal-oriented evaluation to inform funding decisions. Drawing on the development of this framework, it examines its initial design, the practical and structural challenges encountered, and the responses adopted. The work identifies three central challenges: evaluation indicators must reflect heterogeneous project goals and contexts; quantitative assessment relies on fragmented and fragile data infrastructure; and observed changes before, during, and after investment do not establish causal attribution without additional research designs. The paper argues that evaluating investment in open source software is not only a methodological task but also a governance practice, dependent on sustainable open research infrastructure, community knowledge, and ongoing methodological refinement.

Explore related subjects

cs.CY

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Laia Domenech Burin. 2026-11-04. Building a Method: Challenges and Lessons from Developing an Impact Evaluation Framework for Public Investment in Open Source Software. https://arxiv.org/abs/2611.06281

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Digital Security Under Compound Threats: Surveillance, Threat Reasoning, and Habituation Among North Korean Defectors in China

North Korean defectors residing in China face persistent uncertainty regarding digital security and privacy. Through semi-structured interviews with 19 defectors, we examine how individuals understand digital threats and adapt their security behaviors. We identify three interrelated conditions: transnational surveillance pressure from North Korea, identity-linked digital infrastructure and intermediary dependency in China, and threat reasoning shaped by prior surveillance schemas. Participants developed uneven protective practices and followed two trajectories: adaptive habituation driven by knowledge acquisition and erosive habituation driven by the prolonged absence of adverse events. Our findings show how compound threats, attribution uncertainty, and constrained agency shape security reasoning among undocumented users.

cs.CY↗
arXiv2026

The Popular Get More Popular: Auditing Brand Bias in Generative Search

The shift from traditional to generative search has fundamentally transformed online product discovery. Because conversational recommendations synthesize choices into a few direct answers, subtle systemic preferences toward specific providers can exert profound commercial impacts. In this work, we audit this phenomenon through two distinct dimensions: (i) brand popularity bias, the systematic preference for market-dominant entities over long-tail alternatives, and (ii) brand familiarity bias, the ranking advantage an entity encoded in an LLM's parametric memory holds over an unfamiliar competitor. First, evaluating commercial queries across major engines, we observe persistent brand popularity bias: dominant brands in the top quintile are cited 1.4x to 2.8x more frequently than the average brand within a sample of 1,176 top market players, already present in parametric memory that external web retrieval fails to mitigate. Second, through controlled in-context experiments holding product attributes constant, we find that substituting authentic brand signatures with unfamiliar ones significantly degrades recommendation rank. Conversely, fact-preserving content rewrites yield negligible ranking gains. Taken together, our findings demonstrate that generative search engines reinforce a ``rich-get-richer'' dynamic by favoring established market leaders and imposing a parametric barrier to entry that standard content optimization cannot overcome.

cs.CY↗
arXiv2026

Operationalizing NCII and CSAM Law in the Age of AI: Open Problems

Advances in generative AI have driven a surge in synthetic non-consensual intimate imagery (NCII) and child sexual abuse material (CSAM), prompting a series of legal reforms. These reforms rely on legal definitions where technical operationalization is ambiguous, such as whether a depicted person is identifiable, whether a model is capable of or intended for abuse generation, and whether an abusive image can be traced to the model that produced it. Drawing on the law of England and Wales and the United States, we examine the relevant law alongside the technical research needed to operationalize it. From this, we develop a research agenda of open problems at this technical-legal interface, spanning the interpretation of existing law, the prevention of harm before an AI tool is deployed, and enforcement of offenses. These are problems the technical community is well placed to solve, and where progress is vital to real-world protection.

cs.CY↗