Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2608.11891

Benchmark-Based Comparative Assessment of Publicly Benchmarked Indian Foundation Models: A Capability and Evaluation-Maturity Framework

Avinash Agarwal·Vridhi Jain
View original source ↗Read source document ↗Export metadata ↓

Abstract

Purpose: Governments increasingly fund indigenous foundation models to strengthen national AI capability, digital sovereignty, and multilingual computing. This paper assesses India's foundation-model ecosystem and examines whether apparent capability gaps in public benchmark evidence may also reflect gaps in evaluation maturity. Approach: The paper presents a structured, benchmark-based comparative assessment of Indian foundation models against global frontier and comparable-scale models across eight capability domains: general-purpose reasoning, coding and software engineering, agentic AI and computer use, cybersecurity, vision and image understanding, video and multimodal understanding, scientific research, and Indic language capability. Using only publicly reported results, it proposes an exploratory four-dimension Benchmark Maturity Index (BMI), scoring each domain on standardization, participation, independent verification, and national Findings: Indian models achieve strong scores on established benchmarks such as MMLU and MATH-500. However, these are now widely regarded as saturated, and frontier developers no longer report them. Indian models participate far less frequently in newer, agentic, and domain-specialized evaluations, and participation is highly uneven across organizations. Sarvam AI reports the broadest coverage by a substantial margin. The BMI refines, and in some cases revises, the maturity judgments a purely descriptive review would produce. Practical implications: Many apparent capability gaps cannot be distinguished, on available evidence, from evaluation-ecosystem gaps, with direct implications for how national AI programs should design monitoring and funding criteria. Originality: The paper proposes BMI as a reusable instrument for scoring evaluation-ecosystem maturity at the domain level and demonstrates its application to the Indian foundation-model ecosystem.

Explore related subjects

cs.CYcs.AIcs.HC

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Avinash Agarwal, Vridhi Jain. 2026-08-16. Benchmark-Based Comparative Assessment of Publicly Benchmarked Indian Foundation Models: A Capability and Evaluation-Maturity Framework. https://arxiv.org/abs/2608.11891

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Printed but not benchmarkable: most building-decarbonisation disclosure cannot be matched to the pathways that stranding regulation assumes

Cities are beginning to enforce carbon limits on existing buildings. Science-based decarbonisation pathways set those limits one asset type and one jurisdiction at a time. Owners, however, report for the whole firm. We measure what that mismatch costs on two sets of public corporate reports: a census of 502 reports from the 119 listed built-environment firms with a collected report inside a 2,246-firm panel (2003-2023), and 519 real-estate reports from 101 firms (2007-2024). BeDA, a multimodal language-model tool whose reliability we test first, read them. Running the pathway frameworks' own entry tests over published disclosure: 16.5% of census reports (43.7% of real-estate reports) print an operational carbon intensity per square metre; 6.6% (25.0%) can be matched to a pathway for their property type in a covered jurisdiction; and only 5.0% (16.4%) disclose the floor area they divided by. Of the failures at the pathway test, 82-84% follow from reports lumping the portfolio together and 16-18% from a missing curve in the pathway library. The obstacle is the reporting unit, not missing data. The rate is roughly twice as high for European as for US listings (65-71% versus 35% in listed real estate). We also show that a US portfolio's carbon verdict cannot be worked out from disclosure at all. Within one climate zone, the pathway's carbon limit varies by up to 2.79-fold with the electricity subregion, which no report names; its energy limit does not move. Extraction is checked against the source PDFs (97.3% of extracted intensities appear verbatim) and repeats on a second extractor (kappa = 0.97). Recall of the non-disclosing class was 95.1% in a blinded hand audit of 122 reports. The fix follows from the measurement: split intensity by asset type and jurisdiction, and report floor area.

cs.CY↗
arXiv2026

Incipit: Axiom-Grounded Scaffolding for Human-AI Literary Creation

Large language models can produce fluent prose from short prompts, but a direct interaction gives writers little access to the assumptions that shape a long narrative. We present Incipit, an implemented research prototype that inserts an explicit planning layer between writer intent and generated prose. The layer is grounded in literary axioms, which are curated and reusable propositions about human experience and narrative craft. The prototype connects a knowledge base of 1455 axioms and 472 typed relationships to a five round direction dialogue, a retrieval and selection pipeline, and a three level blueprint covering creative premises, story beats, character arcs, and chapter outlines. Writers can inspect and edit the resulting structures before using them as context for scene generation. Additional modules support real event abstraction and five dimensional diagnostic feedback. We describe the design rationale, data flow, implementation boundaries, and a worked design example. Because no controlled user study or independently rated output study has yet been completed, we do not claim that the system improves literary quality. Instead, we outline a future preregistered comparison designed to distinguish the contribution of axiom grounding from that of hierarchical planning. The paper contributes a concrete architecture for making literary knowledge an inspectable coordination object in human AI writing.

cs.CY↗
arXiv2026

Automating Constructive Assessment with Large Language Models: Toward Scalable and Repeated Evaluation of Practical Competence

This study aimed to automate hierarchical diagnostic reasoning (HDR), a constructive method for evaluating practical judgment skills, by developing and testing an evaluation process using a large language model. HDR is a descriptive task that measures higher-order cognitive skills by requiring students to identify and explain errors in case-study-based problems. However, it requires expertise and effort to develop and evaluate. Hence, we proposed and empirically validated the automatic (1) generation of case problems containing errors aligned with educational intentions, (2) scoring of descriptive answers, and (3) generation of structured feedback based on incorrect answers, achieved solely through prompt design without fine-tuning. The internal consistency and construct validity of the generated problems were supported by the experimental score distribution and Cronbach's alpha (0.78). The agreement between automated and human ratings reached 100% under some conditions. The feedback was rated as being as convincing and useful as that from human instructors, demonstrating a practical framework for implementing HDR-based constructive assessment with reproducibility, immediacy, and low cost. The flexibility of large language models will also enable repeated and longitudinal assessments while maintaining structure, showing broad potential for application in educational settings.

cs.CY↗