arXiv subjects

Adam Cook

Publications and source records attributed to Adam Cook.

3 recordsLinked to original sources

arXiv2026

CTI-REALM: Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilities

CTI-REALM (Cyber Threat Real World Evaluation and LLM Benchmarking) is a benchmark designed to evaluate AI agents' ability to interpret cyber threat intelligence (CTI) and develop detection rules. The benchmark provides a realistic environment that replicates the security analyst workflow. This enables agents to examine CTI reports, execute queries, understand schema structures, and construct detection rules. Evaluation involves emulated attacks of varying complexity across Linux systems, cloud platforms, and Azure Kubernetes Service (AKS), with ground truth data for accurate assessment. Agent performance is measured through both final detection results and trajectory-based rewards that capture decision-making effectiveness. This work demonstrates the potential of AI agents to support labor-intensive aspects of detection engineering. Our comprehensive evaluation of 16 frontier models shows that Claude Opus 4.6 (High) achieves the highest overall reward (0.637), followed by Claude Opus 4.5 (0.624) and the GPT-5 family. An ablation study confirms that CTI-specific tools significantly improve agent performance, a variance analysis across repeated runs demonstrates result stability. Finally, a memory augmentation study shows that seeded context can close 33\% of the performance gap between smaller and larger models.

cs.CR↗

arXiv2025

A Meshing Framework for Digital Twins for Extrusion based Additive Manufacturing

Additive manufacturing (AM) allows for manufacturing of complex three-dimensional geometries not typically realizable with standard subtractive manufacturing practices. The internal microstructure of a 3D printed component can have a significant impact on its mechanical, vibrational, and shock properties and allows for a richer design space when this is controllable. Due to the complex interactions of the internal geometry of an extrusion-based AM component, it is common practice to assume a homogeneous behavior or to perform characterization testing on the specific toolpath configurations. To avoid unnecessary testing or material waste, it is necessary to develop an accurate and consistent numerical simulation framework with relevant boundary value problems that can handle the complicated geometry of internal material microstructure present in AM components. Herein, a framework is proposed to directly create computational meshes suitable for finite element analysis (FEA) of the fine-scale features generated from extrusion-based AM tool paths to maintain a strong process-structure-property-performance linkage. This mesh can be manually or automatically analyzed using standard FEA simulations such as quasi-static preloading, modal analysis, or thermal analysis. The framework allows an in-silico assessment of a target AM geometry where fine-scale features may greatly impact quantities of design interest such as in soft elastomeric lattices where toolpath infill can greatly influence the self contact of a structure in compression, which we will use as a motivating exemplar. This approach greatly reduces the waste of both time and resources consumed through traditional build and test design cycles for non-intuitive design spaces. It also further allows for the exploration of toolpath infill to optimize component properties beyond simple linear properties such as density and stiffness.

cs.CE↗

arXiv2024

DocuMint: Docstring Generation for Python using Small Language Models

Effective communication, specifically through documentation, is the beating heart of collaboration among contributors in software development. Recent advancements in language models (LMs) have enabled the introduction of a new type of actor in that ecosystem: LM-powered assistants capable of code generation, optimization, and maintenance. Our study investigates the efficacy of small language models (SLMs) for generating high-quality docstrings by assessing accuracy, conciseness, and clarity, benchmarking performance quantitatively through mathematical formulas and qualitatively through human evaluation using Likert scale. Further, we introduce DocuMint, as a large-scale supervised fine-tuning dataset with 100,000 samples. In quantitative experiments, Llama 3 8B achieved the best performance across all metrics, with conciseness and clarity scores of 0.605 and 64.88, respectively. However, under human evaluation, CodeGemma 7B achieved the highest overall score with an average of 8.3 out of 10 across all metrics. Fine-tuning the CodeGemma 2B model using the DocuMint dataset led to significant improvements in performance across all metrics, with gains of up to 22.5% in conciseness. The fine-tuned model and the dataset can be found in HuggingFace and the code can be found in the repository.

cs.SE↗