University of Cambridge > Talks.cam > Cambridge Psychometrics Centre Seminars > Crowd IQ: Weighting Votes in Crowdsourcing and Multi-Agent Systems using Item Response Theory

Crowd IQ: Weighting Votes in Crowdsourcing and Multi-Agent Systems using Item Response Theory

Add to your list(s) Download to your calendar using vCal

If you have a question about this talk, please contact Chan Yin Wah Fiona.

Crowd IQ: Weighting Votes in Crowdsourcing and Multi-Agent Systems using Item Response Theory

This work introduces a Weighted Majority Voting (WMV) approach for boosting the performance of crowds and multi-agent systems. Our approach relies on the estimation of performance of individual agents based on their ability modelled using Item Response Theory (IRT), a core theory in modern psychological assessment. We provide a brief introduction to IRT and derive the formula defining the weight of individual agents based on their ability and parameters of the task. Using simulated and empirical samples, we show that WMV (1) offers a significant boost in performance across tasks and crowds of different size, (2) is not substantially affected by not knowing the true task parameters-a common scenario in the real-life setting, (3) significantly increases the chances of the crowd to outperform their smartest member, and (4) boosts the marginal contributions of additional agents. Also, using the example of general intelligence, we show that an agent’s ability can be estimated with appropriate accuracy using a relatively short screening test. We conclude with a discussion of the advantages of WMV and suggest that implementing ability- rather than performance-based metrics can offer great advantages in the context of crowdsourcing and multi-agent platforms.

This talk is part of the Cambridge Psychometrics Centre Seminars series.

Tell a friend about this talk:

This talk is included in these lists:

Note that ex-directory lists are not shown.

 

© 2006-2020 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity