Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2212.05777

Design and Evaluation of Crowd-sourcing Platforms Based on Users Confidence Judgments

Samin Nili Ahmadabadi·Maryam Haghifam·Vahid Shah-Mansouri·Sara Ershadmanesh
View original source ↗Read source document ↗Export metadata ↓

Abstract

Crowd-sourcing deals with solving problems by assigning them to a large number of non-experts called crowd using their spare time. In these systems, the final answer to the question is determined by summing up the votes obtained from the community. The popularity of using these systems has increased by facilitation of access to community members through mobile phones and the Internet. One of the issues raised in crowd-sourcing is how to choose people and how to collect answers. Usually, the separation of users is done based on their performance in a pre-test. Designing the pre-test for performance calculation is challenging; The pre-test questions should be chosen in a way that they test the characteristics in people related to the main questions. One of the ways to increase the accuracy of crowd-sourcing systems is to pay attention to people's cognitive characteristics and decision-making model to form a crowd and improve the estimation of the accuracy of their answers to questions. People can estimate the correctness of their responses while making a decision. The accuracy of this estimate is determined by a quantity called metacognition ability. Metacoginition is referred to the case where the confidence level is considered along with the answer to increase the accuracy of the solution. In this paper, by both mathematical and experimental analysis, we would answer the following question: Is it possible to improve the performance of the crowd-sourcing system by knowing the metacognition of individuals and recording and using the users' confidence in their answers?

Explore related subjects

cs.CEcs.HC

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Samin Nili Ahmadabadi, Maryam Haghifam, Vahid Shah-Mansouri, Sara Ershadmanesh. 2024-07-03. Design and Evaluation of Crowd-sourcing Platforms Based on Users Confidence Judgments. https://arxiv.org/abs/2212.05777

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Cosm: Collective switched motion for sparse Ising optimization

We introduce Collective Switched Motion (Cosm), a heuristic optimization framework based on switched collective dynamics. Cosm compiles the objective into subobjectives that are activated sequentially, temporally separating competing local influences. The interplay of switching and local interactions among variables gives rise to collective search behavior, while a new correlated perturbation mechanism encourages coordinated cluster motion. Tests on tuned-hardness spin-glass benchmarks suggest more favorable algorithmic scaling than that reported for leading dynamical solvers. Cosm heuristically attains the certified optima of three of the largest Gset instances (G72, G77, G81), exceeding the previously reported heuristic solutions. On the large random-graph instances G61 and G70, a CPU implementation reliably attains the best-known solutions, reaching cuts of 5799 and 9595 with 99%-confidence times-to-target of 15 s and 2.4 s, respectively. On the heterogeneous-degree G64 instance, Cosm establishes a new best-known cut of 8753. Broadly, the results suggest an alternative approach to heuristic design in which local dynamics and orchestration mechanisms are carefully designed so that effective search emerges.

cs.CE↗
arXiv2026

Decomposing Firm-Level Crisis Responses from Incomplete Market Signals: Evidence from China's IT Sector During COVID-19

Exogenous shocks generate heterogeneous behavioral responses across firms, yet event studies typically report only sector-level averages. This paper develops a multi-method approach combining causal identification (difference-in-differences with cluster-robust inference), unsupervised behavioral discovery (K-means trajectory clustering, Gaussian hidden Markov models), and cross-sectional resilience prediction (logistic regression with nested cross-validation) to decompose firm-level response heterogeneity from noisy market signals. We demonstrate the approach on 246 Chinese A-share IT firms (216 with complete data for all analyses) during the COVID-19 shock (January 2020), using 252 non-IT CSI 300 firms as controls. The return decline was market-wide, not IT-specific (DID p = 0.59); the IT-specific effect was elevated volatility (DID beta = 0.043, cluster-robust p < 0.001), with the effect surviving Benjamini-Hochberg correction in 13 of 30 alternative specifications. Unsupervised clustering produced three trajectory groups: fast recovery (36 companies, +29.7%), resilient/moderate (67 companies), and persistent drag (113 companies, -6.9%). Pre-crisis financial fundamentals showed only modest predictive power for resilience (nested CV AUC = 0.635, 95% CI: 0.558-0.711; permutation p = 0.016), consistent with the limited informativeness of publicly available signals for anticipating crisis outcomes. The combination of causal analysis, unsupervised learning, and prediction represents a reproducible framework which can be applied to crises in other market periods.

cs.CE↗
arXiv2026

An Insurance Broker for Every Small Business: The Economics of Exceptional Care at Scale

Small-business owners need expert guidance on their own terms, across schedules, languages, and channels, but low premiums make exceptional, continuous human service uneconomic for much of the market. Combining public evidence, Kinro operational data, and an illustrative five-year service model, we show why traditional brokerage economics leave 35 million U.S. small businesses underserved. An AI-native brokerage can change those economics by performing and coordinating routine work continuously, while licensed professionals govern consequential exceptions and the brokerage remains accountable.

cs.CE↗