logo

SCIENCE CHINA Information Sciences, Volume 64 , Issue 3 : 130104(2021) https://doi.org/10.1007/s11432-020-3118-7

Crowdsourcing aggregation with deep Bayesian learning

More info
  • ReceivedJun 30, 2020
  • AcceptedJul 30, 2020
  • PublishedFeb 7, 2021

Abstract


Acknowledgment

This work was supported by Fundamental Research Funds for the Central Universities (Grant No. NJ20190- 10), National Natural Science Foundation of China (Grant No. 61906089), Jiangsu Province Basic Research Program (Grant No. BK20190408), and China Postdoc Science Foundation (the First Pre-station Special Grant).


References

[1] Horvitz E. Reflections on challenges and promises of mixed-initiative interaction. AI Mag, 2007, 28: 13--22. Google Scholar

[2] Weld D, Lin C, Bragg J. Artificial intelligence and collective intelligence. In: Proceedings of Collective Intelligence Handbook. 2015. Google Scholar

[3] Snow R, O'Connor B, Jurafsky D, et al. Cheap and fast — but is it good? evaluating non-expert annotations for natural language tasks. In: Proceedings of Conference on Empirical Methods in Natural Language Processing, 2008. 254--263. Google Scholar

[4] Raykar V, Yu S, Zhao L, et al. Learning from crowds. J Mach Learn Res, 2010, 11: 1297--1322. Google Scholar

[5] Welinder P, Branson S, Belongie S, et al. The multidimensional wisdom of crowds. In: Proceedings of Advances in Neural Information Processing Systems, 2010. 2024--2432. Google Scholar

[6] Li Q, Li Y, Gao J. A confidence-aware approach for truth discovery on long-tail data. Proc VLDB Endow, 2014, 8: 425-436 CrossRef Google Scholar

[7] Dawid A P, Skene A M. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Appl Stat, 1979, 28: 20 CrossRef Google Scholar

[8] Whitehill J, Ruvolo P, Wu T, et al. Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In: Proceedings of Advances in Neural Information Processing Systems, 2009. 2035--2043. Google Scholar

[9] Zhou D Y, Basu S, Mao Y, et al. Learning from the wisdom of crowds by minimax entropy. In: Proceedings of Advances in Neural Information Processing Systems, 2012. 2195--2203. Google Scholar

[10] Li Y, Rubinstein B I P, Cohn T. Exploiting worker correlation for label aggregation in crowdsourcing. In: Proceedings of the 36th International Conference on Machine Learning, 2019. 3886--3895. Google Scholar

[11] Rodrigues F, Pereira F, Ribeiro B, et al. Gaussian process classification and active learning with multiple annotators. In: Proceedings of the 31th International Conference on Machine Learning, 2014. 433--441. Google Scholar

[12] Albarqouni S, Baur C, Achilles F. AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images. IEEE Trans Med Imag, 2016, 35: 1313-1321 CrossRef Google Scholar

[13] Atarashi K, Oyama S, Kurihara M. Semi-supervised learning from crowds using deep generative models. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence, 2018. 1555--1562. Google Scholar

[14] Rodrigues F, Pereira P C. Deep learning from crowds. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence, 2018. 1611--1618. Google Scholar

[15] Tanno R, Saeedi A, Sankaranarayanan S, et al. Learning from noisy labels by regularized estimation of annotator confusion. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019. 11244--11253. Google Scholar

[16] Imamura H, Sato I, Sugiyama M. Analysis of minimax error rate for crowdsourcing and its application to worker clustering model. In: Proceedings of the 35th International Conference on Machine Learning, 2018. 2152--2161. Google Scholar

[17] Sheng V S, Zhang J, Gu B. Majority Voting and Pairing with Multiple Noisy Labeling. IEEE Trans Knowl Data Eng, 2019, 31: 1355-1368 CrossRef Google Scholar

[18] Tao F, Jiang L, Li C. Label similarity-based weighted soft majority voting and pairing for crowdsourcing. Knowl Inf Syst, 2020, 62: 2521-2538 CrossRef Google Scholar

[19] Liu Q, Peng J, Ihler A. Variational inference for crowdsourcing. In: Proceedings of Advances in Neural Information Processing Systems, 2012. 692--700. Google Scholar

[20] Kim H C, Ghahramani Z. Bayesian classifier combination. In: Proceedings of the 15th International Conference on Artificial Intelligence and Statistics, 2012. 619--627. Google Scholar

[21] Simpson E, Roberts S, Psorakis I, et al. Dynamic Bayesian combination of multiple imperfect classifiers. In: Decision Making and Imperfection, 2013. Google Scholar

[22] Venanzi M, Guiver J, Kazai P, et al. Community-based bayesian aggregation models for crowdsourcing. In: Proceedings of the 23rd International Conference on World Wide Web, 2014. 155--164. Google Scholar

[23] Moreno P G, Artes-Rodrguez A, Teh Y W, et al. Bayesian nonparametric crowdsourcing. J Mach Learn Res, 2015, 16: 1607--1627. Google Scholar

[24] Rodrigues F, Lourenco M, Ribeiro B. Learning Supervised Topic Models for Classification and Regression from Crowds. IEEE Trans Pattern Anal Mach Intell, 2017, 39: 2409-2422 CrossRef Google Scholar

[25] Zhang H, Jiang L X, Xu W Q. Multiple noisy label distribution propagation for crowdsourcing. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence, 2019. 1473--1479. Google Scholar

[26] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521: 436-444 CrossRef ADS Google Scholar

[27] Luo Y C, Tian T, Shi J X, et al. Semi-crowdsourced clustering with deep generative models. In: Proceedings of Advances in Neural Information Processing Systems, 2018. 3216--3226. Google Scholar

[28] Kingma D P, Mohamed S, Rezende D J, et al. Semi-supervised learning with deep generative models. In: Proceedings of Advances in Neural Information Processing Systems, 2014. 3581--3589. Google Scholar

[29] Kingma D P, Welling M. Auto-encoding variational bayes. In: Proceedings of the 2nd International Conference on Learning Representations, 2014. Google Scholar

[30] Johnson M J, Duvenaud D, Wiltschko A B, et al. Composing graphical models with neural networks for structured representations and fast inference. In: Proceedings of Advances in Neural Information Processing Systems, 2016. 2946--2954. Google Scholar

[31] Lin X, Zou Y, Li S. Finite-time stability and finite-time weighted L 2-gain analysis for switched systems with time-varying delay. 8 CrossRef Google Scholar

[32] Li S Y, Jiang Y, Chawla N V. Multi-Label Learning from Crowds. IEEE Trans Knowl Data Eng, 2019, 31: 1369-1382 CrossRef Google Scholar

  • Figure 1

    The plate notation for our proposed BayesDGC.

  • Figure 2

    (Color online) The positive instance fraction of each label for (a) dataset1 and (b) dataset2.

  • Figure 3

    (Color online) Accuracy results of all methods on $22$ real-word datasets.

  • Figure 4

    (Color online) F1 score results of all methods on $22$ real-word datasets.

  •   

    Algorithm 1 Bayesian deep generative crowdsourcing (BayesDGC)

    Input: example features $\X=\{~\x_1,\ldots,\x_N\}$, crowd annotations $\L~\in~\{0,1,\ldots,K\}^{N\times~W}$, initial value for the global variational parameters $\bet_{\bTheta}$ and neural network parameters $\bgamma$.

    Repeat:

    $\;\;$ Given $\bet_{\bTheta},\bgamma$, update the true label's natural parameter $\bet^{\ast}_{\y_i}$ for each example using Eq. (14);

    $\;\;$ Given estimated $\bet^{\ast}_{\y_i}$, compute the gradient of $\bet_{\bTheta}$ using Eqs. (17) and (18) and $\bgamma$ via DNN back propagation, then conduct SGD updating for them;

    Until the lower bound $\mathcal{J}(\bet_{\bTheta},~\bgamma)$ converges or the maximum number of epochs is reached.