SCIENTIA SINICA Informationis, Volume 49 , Issue 11 : 1412-1427(2019) https://doi.org/10.1360/N112018-00303

An approach for developing a highly trustworthy crowd-sourced workforce

More info
  • ReceivedMay 31, 2019
  • AcceptedSep 11, 2019
  • PublishedNov 4, 2019


In applying crowd-sourcing techniques, one of the most critical challenges is building a crowd workforce that is both capable and trustworthy. Previous studies proposed numerous strategies, methods, and mechanism to motivate individuals; however, although the results improved the effectiveness and efficiency of finishing crowd-sourcing tasks, few studies focused on improving the honesty of crowd-sourced workers and assisting requesters in obtaining the correct quality report. To address this, based on the principal-agent model and signaling game theory, we design a novel mechanism for building a capable and trustworthy crowd-sourced workforce. This mechanism enables information exchange between crowd-sourced workers and requesters, and leverages a random inspection strategy to assign financial incentives/punishments to honest/dishonest behaviors accordingly. To validate our mechanism, we conduct an extensive simulation. The results show this mechanism is effective and efficient to motivate workers to behave in a trustworthily manner and capable of changing the behavior of dishonest workers with minimal extra cost.

Funded by




[1] Bar A J, Maheswaran M. Confidentiality and Integrity in Crowdsourcing Systems. Berlin: Springer, 2013. Google Scholar

[2] Horton J J, Chilton L B. The labor economics of paid crowdsourcing. In: Proceedings of the 11th ACM Conference on Electronic Commerce, 2010. 209--218. Google Scholar

[3] Slivkins A, Vaughan J W. Online decision making in crowdsourcing markets: theoretical challenges. SIGecom Exch, 2014, 12: 4--23. Google Scholar

[4] Becker T E. Integrity in Organizations: Beyond Honesty and Conscientiousness. AMR, 1998, 23: 154-161 CrossRef Google Scholar

[5] Dineen B R, Lewicki R J, Tomlinson E C. Supervisory guidance and behavioral integrity: relationships with employee citizenship and deviant behavior.. J Appl Psychology, 2006, 91: 622-635 CrossRef PubMed Google Scholar

[6] Kidder D L. Is it `who I am', `what I can get away with', or `what you've done to me'? A Multi-theory Examination of Employee Misconduct. J Bus Ethics, 2005, 57: 389-398 CrossRef Google Scholar

[7] Zhang Y, Schaar M. Reputation-based incentive protocols in crowdsourcing applications. In: Proceedings of International Conference on Computer Communications, 2012. 2140--2148. Google Scholar

[8] Ho C J, Slivkins A, Suri S, et al. Incentivizing high quality crowdwork. In: Proceedings of the 24th International Conference on World Wide Web, 2015. 419--429. Google Scholar

[9] Singer Y, Mittal M. Pricing mechanisms for crowdsourcing markets. In: Proceedings of the 22nd International Conference on World Wide Web, 2013. 1157--1166. Google Scholar

[10] Singla A, Krause A. Truthful incentives in crowdsourcing tasks using regret minimization mechanisms. In: Proceedings of the 22nd International Conference on World Wide Web, 2013. 1167--1178. Google Scholar

[11] Ho C J, Slivkins A, Vaughan J W. Adaptive contract design for crowdsourcing markets: bandit algorithms for repeated principal-agent problems. In: Proceedings of the 5th ACM Conference on Economics and Computation, 2014. 359--376. Google Scholar

[12] Laffont J J, Martimort D. The Theory of Incentives: the Principal-agent Model. Princeton: Princeton University Press, 2009. Google Scholar

[13] Grossman S J, Hart O D. An Analysis of the Principal-Agent Problem. Econometrica, 1983, 51: 7 CrossRef Google Scholar

[14] Skyrms B. Signals: Evolution, Learning, and Information. Oxford: Oxford University Press, 2010. Google Scholar

[15] Catteeuw D, Han T A, Manderick B. Evolution of honest signaling by social punishment. In: Proceedings of Annual Conference on Genetic and Evolutionary Computation, 2014. 153--160. Google Scholar

[16] Huttegger S, Skyrms B, Tarres P, et al. Some dynamics of signaling games. In: Proceedings of National Academy of Sciences of the United States of America, 2014. 10873--10880. Google Scholar

[17] Ones D S, Viswesvaran C. Gender, age, and race differences on overt integrity tests: Results across four large-scale job applicant datasets.. J Appl Psychology, 1998, 83: 35-42 CrossRef Google Scholar

[18] Mayer T. The Distribution of Ability and Earnings. Rev Economics Stat, 1960, 42: 189 CrossRef Google Scholar

[19] Ashton M C, Lee K, Son C. Honesty as the sixth factor of personality: correlations with machiavellianism, primary psychopathy, and social adroitness. Eur J Pers, 2000, 14: 359--368. Google Scholar

[20] Train K E. Discrete Choice Methods with Simulation. Cambridge: Cambridge University Press, 2009. Google Scholar

[21] Eppig C, Fincher C L, Thornhill R. Parasite prevalence and the worldwide distribution of cognitive ability.. Proc R Soc B, 2010, 277: 3801-3808 CrossRef PubMed Google Scholar

[22] Cowell F. Measuring Inequality. Oxford: Oxford University Press, 2011. Google Scholar

[23] G?chter S, Schulz J F. Intrinsic honesty and the prevalence of rule violations across societies. Nature, 2016, 531: 496-499 CrossRef PubMed ADS Google Scholar

[24] Mazar N, Amir O, Ariely D. The Dishonesty of Honest People: A Theory of Self-Concept Maintenance. J Marketing Res, 2008, 45: 633-644 CrossRef Google Scholar

[25] Shalvi S, Dana J, Handgraaf M J J. Justified ethicality: Observing desired counterfactuals modifies ethical perceptions and behavior. Organizational Behav Human Decision Processes, 2011, 115: 181-190 CrossRef Google Scholar

[26] Vakharia D, Lease M. Beyond mechanical turk: an analysis of paid crowd work platforms In: Proceedings of the iConference, 2015. Google Scholar

[27] Whiting M E, Gamage D, Gaikwad S S, et al. Crowd guilds: worker-led reputation and feedback on crowdsourcing platforms. In: Proceedings of Conference on Computer Supported Cooperative Work and Social Computing, 2017. 1902--1913. Google Scholar

[28] Bolton P, Dewatripont M. Contract Theory. Cambridge: MIT press, 2005. Google Scholar

[29] Babaioff M, Feldman M, Nisan N. Combinatorial agency. In: Proceedings of the 7th ACM Conference on Electronic Commerce, 2006. 18--28. Google Scholar

[30] Gittins J, Glazebrook K, Weber R. Multi-armed Bandit Allocation Indices. Hoboken: John Wiley and Sons, 2011. Google Scholar

[31] Biswas A, Jain S, Mandal D, et al. A truthful budget feasible multi-armed bandit mechanism for crowdsourcing time critical tasks. In: Proceedings of International Conference on Autonomous Agents and Multiagent Systems, 2015. 1101--1109. Google Scholar

[32] Frazier P, Kempe D, Kleinberg J, et al. Incentivizing exploration. In: Proceedings of the 5th ACM Conference on Economics and Computation, 2014. 5--22. Google Scholar

[33] Jain S, Narayanaswamy B, Narahari Y. A multiarmed bandit incentive mechanism for crowdsourcing demand response in smart grids. In: Proceedings of National Conference on Artificial Intelligence, 2014. 721--727. Google Scholar

[34] Tran-Thanh L, Stein S, Rogers A, et al. Efficient crowdsourcing of unknown experts using multi-armed bandits. In: Proceedings of European Conference on Artificial Intelligence, 2012. 768--773. Google Scholar

[35] den Boer A V. Dynamic pricing and learning: Historical origins, current research, and new directions. Surveys Operations Res Manage Sci, 2015, 20: 1-18 CrossRef Google Scholar

[36] Badanidiyuru A, Kleinberg R, Singer Y. Learning on a budget: posted price mechanisms for online procurement. In: Proceedings of the 13th ACM Conference on Electronic Commerce, 2012. 128--145. Google Scholar

[37] Kleinberg R, Leighton T. The value of knowing a demand curve: bounds on regret for online posted-price auctions. In: Proceedings of the 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. 594--605. Google Scholar

[38] Easley D, Ghosh A. Behavioral mechanism design: optimal crowdsourcing contracts and prospect theory. In: Proceedings of the 6th ACM Conference on Economics and Computation, 2015. 679--696. Google Scholar

[39] Herbert Gintis. The Bounds of Reason: Game Theory and the Unification of the Behavioral Sciences. Princeton: Princeton University Press, 2009. Google Scholar

[40] Horton J J, Zeckhauser R J. Algorithmic wage negotiations: applications to paid crowdsourcing. In: Proceedings of CrowdConf, 2010. Google Scholar

[41] Mason W, Watts D J. Financial incentives and the "performance of crowds". SIGKDD Explor Newsl, 2010, 11: 100-108 CrossRef Google Scholar

[42] Shaw A D, Horton J J, Chen D L. Designing incentives for inexpert human raters. In: Proceedings of Conference on Computer Supported Cooperative Work, 2011. 275--284. Google Scholar

[43] Harris C. You're hired an examination of crowdsourcing incentive models in human resource tasks. In: Proceedings of 2011 Workshop on Crowdsourcing for Search and Data Mining, 2011. 15--18. Google Scholar

[44] Ikeda K, Bernstein M S. Pay it backward: per-task payments on crowdsourcing platforms reduce productivity. In: Proceedings of CHI Conference on Human Factors in Computing Systems, 2016. 4111--4121. Google Scholar

  • Figure 1

    Four types of crowd workers according to their capability and integrity

  • Figure 2

    Signaling game between a crowd worker and a requester

  • Figure 3

    The process of the proposed mechanism. The left part is the overview of all periods, and the right part is the details of interactions in each period

  • Figure 4

    (Color online) Requester's accumulative payoff over 50 periods

  • Figure 5

    (Color online) Individual contribution to requester's payoff in each period

  • Figure 6

    (Color online) Accumulative administration costs

  • Table 1   The composition workforce after 50 periods (averaged over 20 runs), report-sample-and-pay is excluded because it does not kick out any workers
    机制 总余额 高能力工人数 诚信工人数 理想工作者 成功转化工人数
    运行前 运行后 运行前 运行后 运行前 运行后
    Signal-no-tolerance 353.10 102.40 98.50 181.10 137.10 16.75 16.55 336.55
    Report-no-tolerance 17.60 99.65 17.55 183.15 3.30 17.55 3.30 14.30
    Our mechanism 393.15 102.40 102.10 180.40 137.10 18.65 18.60 374.55