logo

SCIENTIA SINICA Informationis, Volume 50 , Issue 9 : 1281(2020) https://doi.org/10.1360/SSI-2020-0204

Toward the third generation of artificial intelligence

More info
  • ReceivedJul 6, 2020
  • AcceptedAug 12, 2020
  • PublishedSep 22, 2020

Abstract

There have been two competing paradigms of artificial intelligence (AI) development since 1956, i.e., symbolism and connectionism (or subsymbolism). Both started at the same time, but symbolism had dominated AI development until the end of the 1980s. Connectionism began to develop in the 1990s and reached its climax at the beginning of this century, and it is likely to displace symbolism. Today, it seems that the two paradigms only simulate the human mind (or brain) in different ways and have their own advantages. True human intelligence cannot be achieved by relying on only one paradigm. Both are necessary to establish a new, explainable, and robust AI theory and method and develop safe, trustworthy, reliable, and extensible AI technology. To this end, it is imperative to combine the two paradigms, and the present article will illustrate this idea. For the sake of description, symbolism, connectionism, and the newly developed paradigm are termed as first-, second-, and third-generation AIs.


Funded by

国家自然科学基金重点国际合作项目(61620106010)


References

[1] Simon H A. Models of Man. New York: Wiley & Sons, 1957. Google Scholar

[2] Newell A, Simon H A. Computer science as empirical inquiry: symbols and search. Commun ACM, 1976, 19: 113-126 CrossRef Google Scholar

[3] Newell A. Physical Symbol Systems*. Cognitive Sci, 1980, 4: 135-183 CrossRef Google Scholar

[4] Fodor J A. Methodological solipsism considered as a research strategy in cognitive psychology. Behav Brain Sci, 1980, 3: 63-73 CrossRef Google Scholar

[5] McCarthy J, Minsky M L, Rochester N, et al. A proposal for the Dartmouth summer research project on artificial intelligence. 1955, 27: 12 https://doi.org/10.1609/aimag.v27i4.1904. Google Scholar

[6] Lindsay R K, Buchanan B G, Feigenbaum E A, et al. Applications of Artificial Intelligence for Organic Chemistry: The Dendral Project. New York: McGraw-Hill Book Company, 1980. Google Scholar

[7] Buchanan B G, Shortliffe E H. Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project. Boston: Addison Wesley, 1984. Google Scholar

[8] Muggleton S, de Raedt L. Inductive Logic Programming: Theory and methods. J Logic Programming, 1994, 19-20: 629-679 CrossRef Google Scholar

[9] Fabrizio R, Elena B, Riccardo Z, et al. A history of probabilistic inductive logic programming. Frontiers in Robotics and AI, 2014, 1: 1-6. Google Scholar

[10] Yang, Qiang, et al. Transfer Learning. Cambridge: Cambridge Press, 2020. Google Scholar

[11] Ehrlinger L, Wöß W. Towards a definition of knowledge graphs. In: Proceedings of Joint Proceedings of the Posters and Demos Track of 12th International Conference on Semantic Systems — SEMANTiCS2016 and 1st International Workshop on Semantic Change & Evolving Semantics (SuCCESS16), At Leipzig, 2016. 48: 1--4. Google Scholar

[12] Russel S J, Norvig P. Artificial Intelligence-A modern Approach. 2nd ed. Upper Saddle River: Prentice Hall, Inc., 2003. Google Scholar

[13] Rosenblatt F. The perceptron: A probabilistic model for information storage and organization in the brain.. Psychological Rev, 1958, 65: 386-408 CrossRef Google Scholar

[14] Rosenblatt F. Principles of Neurodynamics. Arch Gen Psychiatry. 1962, 7(3): 218-219. Google Scholar

[15] McCulloch W S, Pitts W. A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys, 1943, 5: 115-133 CrossRef Google Scholar

[16] Hebb D O. The Organization of Behavior-A Neuropsychological Theory. Abingdon: Psychology Press, 1949. Google Scholar

[17] Minsky M, Papert S A. Perceptrons: An Introduction to Computational Geometry. Cambridge: The MIT Press, 1969. Google Scholar

[18] Cauchy A. Methode generale pour la resolution des systemes d'equations simultanees. Comptes Rendus Hebd Seances Acad Sci, 1847, 25: 536--538. Google Scholar

[19] Nesterov Y E. A Method for Solving the Convex Programming Problem with Convergence Rate O(1/k2). Soviet Mathematics Doklady, 1983, 27: 372-376. Google Scholar

[20] Rumelhart D E, Hinton G E, Williams R J. Learning representations by back-propagating errors. Nature, 1986, 323: 533-536 CrossRef ADS Google Scholar

[21] Janocha K, Czarnecki W M. On Loss Functions for Deep Neural Networks in Classification. Schedae Informaticae, 2016, 25: 49-59. Google Scholar

[22] Wan L, Zeiler M, Zhang S X, et al. Regularization of neural networks using dropconnect. In: Proceedings of the 30th International Conference on Machine Learning (ICML), Atlanta, 2013. 1058--1066. Google Scholar

[23] Fukushima K. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybernetics, 1980, 36: 193-202 CrossRef Google Scholar

[24] Deep convolutional networks for quality assessment of protein folds. Bioinformatics, 2018, 34: 4046-4053 CrossRef Google Scholar

[25] Lipton J C, Berkowitz J. A Critical Review of Recurrent Neural Networks for Sequence Learning. 2015,. arXiv Google Scholar

[26] Hochreiter S, Schmidhuber J. Long Short-Term Memory. Neural Computation, 1997, 9: 1735-1780 CrossRef Google Scholar

[27] Hinton G E, Osindero S, Teh Y W. A Fast Learning Algorithm for Deep Belief Nets. Neural Computation, 2006, 18: 1527-1554 CrossRef Google Scholar

[28] Lecun Y, Bengio Y, and Hinton G E. Deep Learning. Nature, 2015, 521: 436-444. Google Scholar

[29] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, 2016. 770--778. Google Scholar

[30] Coulom R. Efficient selectivity and backup operators in Monte-Carlo tree search. In: Proceedings of the International Conference on Computers and Games, Berlin, 2006. 72--83. Google Scholar

[31] Sutton R S, Barto A G. Reinforcement Learning: An Introduction. Cambridge: The MIT Press, 2015. Google Scholar

[32] Dong Y P, Liao F Z, Pang T Y, et al. Boosting adversarial attacks with momentum. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, 2018. 9185--9193. Google Scholar

[33] Dong Y P, Su H, Zhu J, et al. Towards interpretable deep neural networks by leveraging adversarial examples. In: Proceedings of the IJCAI workshop on AISC, Sydney, 2019. 1--6. Google Scholar

[34] Ferrucci D, Brown E, Chu-Carroll J. Building Watson: An Overview of the DeepQA Project. AIMag, 2010, 31: 59-79 CrossRef Google Scholar

[35] Harnad S. The symbol grounding problem. Physica D-NOnlinear Phenomena, 1990, 42: 335-346 CrossRef Google Scholar

[36] Chen X, Duan Y, Houthooft R, et al. InfoGAN, interpretable representation learning by information maximizing generative adversarial nets. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Barcelona SPAIN, 2016. 2172--2180. Google Scholar

[37] Liu Y, Wei F Y, Shao J, et al. Exploring disentangled feature representation beyond face identificatio. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, 2018. 2080--2089. Google Scholar

[38] Higgins I, Matthey L, Pal A, et al. Beta-VAE: learning basic visual concepts with a constrained variational framework. In: Proceedings of the 32nd International Conference on Logic Programming (ICLP), New York City, 2016. 1--12. Google Scholar

[39] Siddharth N, Brooks P, Alban D, et al. Inducing interpretable representations with variational autoencoders. 2016,. arXiv Google Scholar

[40] Li C, Xu K, Zhu J, et al. Triple generative adversarial nets. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, 2017. 4088--4098. Google Scholar

[41] Sparsity-Regularized HMAX for Visual Recognition. PLoS ONE, 2014, 9: e81813 CrossRef ADS Google Scholar

[42] Mnih V, Kavukcuoglu K, Silver D. Human-level control through deep reinforcement learning. Nature, 2015, 518: 529-533 CrossRef ADS Google Scholar

[43] Vinyals O, Babuschkin I, Czarnecki W M. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 2019, 575: 350-354 CrossRef ADS Google Scholar

[44] Silver D, Huang A, Maddison C J. Mastering the game of Go with deep neural networks and tree search. Nature, 2016, 529: 484-489 CrossRef ADS Google Scholar

[45] Silver D, Schrittwieser J, Simonyan K. Mastering the game of Go without human knowledge. Nature, 2017, 550: 354-359 CrossRef ADS Google Scholar

[46] Duan Y, Chen X, Houthooft R, et al. Benchmarking deep reinforcement learning for continuous control. In: Proceedings of the International Conference on Machine Learning (ICML), New York, 2016. 1329--1338. Google Scholar

[47] Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms. 2017,. arXiv Google Scholar

[48] Wiering, Marco, and Martijn Van Otterlo. Reinforcement learning. Springer, 2012, 12. Google Scholar

[49] Fran?ois-Lavet V, et al. An Introduction to Deep Reinforcement Learning. Foundations and Trends in Machine Learning, 2018, 11: 3-4. Google Scholar

[50] Huang S Y, Su H, Zhu J, et al. Combo-action: training agent for FPS game with auxiliary tasks. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI), Honolulu, 2019. 954--961. Google Scholar

[51] Song S H, Weng J Y, Su H, et al. Playing FPS games with environment-aware hierarchical reinforcement learning. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), Macau, 2019. 3475--3482. Google Scholar

[52] Zhou Y C, Li J L, Zhu J. Posterior sampling for multi-agent reinforcement learning: solving extensive games with imperfect information. In: Proceedings of the International Conference on Learning Representations (ICLR), Addis Ababa, 2020. Google Scholar

[53] Mikolov T, Chen K, Corrado G, et al. Efficient estimation of word representations in vector space. In: Proceedings of the International Conference on Learning Representations (ICLR), Scottsdale, 2013. 1--6. Google Scholar

[54] Pennington J, Socher R, Manning C D. Glove, global vectors for word representation. In: Proceedings of the Conference On Empirical Methods in Natural Language Processing (EMNLP), Doha, 2014. 1532--1543. Google Scholar

[55] Mikolov T, Sutskever I, Chen K, et al. Distributed representations of words and phrases and their compositionality. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Lake Tahoe, 2013. 3111--3119. Google Scholar

[56] Kusner M J, Sun Y, Kolkin N I, et al. From word embeddings to document distances. In: Proceedings of the 32nd International Conference on Machine Learning (ICML), Lille, 2015. 957--966. Google Scholar

[57] Zhang J, Liu Y, Luan H B, et al. Prior knowledge integration for neural machine translation using posterior regularization. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2017. 1514--1523. Google Scholar

[58] Ding Y Z, Liu Y, Luan H B, et al. Visualizing and understanding neural machine translation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2017. 1150--1159. Google Scholar

[59] 张钹. 后深度学习时代的人工智能. 中国人工智能通讯学会通讯, 2017, 7: 3--5. Google Scholar

[60] Liu M, Shi J, Li Z, et al. Towards Better Analysis of Deep Convolutional Neural Networks. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 2016. 91--100. Google Scholar

[61] Wang Y L, Su H, Hu X L. Interpret neural networks by identifying critical data routing paths. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, 2018. 8906--8914. Google Scholar

[62] Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Montreal, 2014. 1--6. Google Scholar

[63] Dong Y P, Su H, Zhu J, et al. Improving interpretability of deep neural networks with semantic information. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 2017. 4306--4314. Google Scholar

[64] Dong Y P, Fu Q-A, Yang X, et al. Benchmarking adversarial robustness on image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, 2020. 321--331. Google Scholar

[65] Liao F Z, Liang M, Dong Y P, et al. Defense against adversarial attacks using high-level representation guided denoiser. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, 2018. 1778--1787. Google Scholar

[66] Pang T Y, Xu K, Du C, et al. Improving adversarial robustness via promoting ensemble diversity. In: Proceedings of the International Conference on Machine Learning (ICML), Long Beach, 2019. 4970--4979. Google Scholar

[67] Castelvecchi D. Can we open the black box of AI? Nature News, 2016, 7623: 20. Google Scholar

[68] Ghahramani Z. Probabilistic machine learning and artificial intelligence. Nature, 2015, 521: 452-459 CrossRef ADS Google Scholar

[69] Zhu J, Chen J, Hu W, et al. Big Learning with Bayesian Methods, National Science Review, 2017, 4: 627-651. Google Scholar

[70] Dempster A, Laird N, Rubin D. Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society, Series B (Methodological), 1977, 39: 1-38. Google Scholar

[71] Zhu J, Chen N, Xing E P. Bayesian Inference with Posterior Regularization and applications to Infinite Latent SVMs. Journal of Machine Learning Research (JMLR), 2014, 15: 1799-1847. Google Scholar

[72] Williams P M. Bayesian Conditionalisation and the Principle of Minimum Information. Br J Philos Sci, 1980, 31: 131-144 CrossRef Google Scholar

[73] Mei S, Zhu J, Zhu X. Robust RegBayes: Selectively Incorporating First-Order Logic Domain Knowledge into Bayesian Models. In: Proceedings of the International Conference on Machine Learning (ICML), Beijing, 2014. 253--261. Google Scholar

[74] Zhu J, Ahmed A, Xing E P. MedLDA: maximum margin supervised topic models. Journal of Machine Learning Research (JMLR), 2012, 13: 2237--2278. Google Scholar

[75] Mackay D, Bridle J, Cheeseman P C. Bayesian methods for adaptive models. Dissertation for Ph.D. Degree. Pasadena: California Institute of Technology, 1992. Google Scholar

[76] Neal R M, Hinton G. Bayesian learning for neural networksm. Dissertation for Ph.D. Degree. Toronto: University of Toronto, 1995. Google Scholar

[77] Shi J, Sun S, Zhu J. A spectral approach to gradient estimation for implicit distributions. In: Proceedings of the 35th International Conference on Machine Learning (ICML), Stockholm, 2018. 4644--4653. Google Scholar

[78] Zhou Y, Shi J, Zhu J. Nonparametric score estimators. In: Proceedings of the International Conference on Machine Learning (ICML), Vienna, 2020. Google Scholar

[79] Wang Z, Ren T, Zhu J, et al. Function Space Particle Optimization for Bayesian Neural Networks. In: Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, 2019. 1--12. Google Scholar

[80] Xu K, Li C, Zhu J, et al. Understanding and stabilizing GANs' training dynamics using control theory. In: Proceedings of the International Conference on Machine Learning (ICML), Vienna, 2020. Google Scholar

[81] Chen J, Lu C, Chenli B, et al. VFlow: more expressive generative flows with variational data augmentation. In: Proceedings of the International Conference on Machine Learning (ICML), Vienna, 2020. Google Scholar

[82] Shi J, Chen J, Zhu J, et al. ZhuSuan: a library for bayesian deep learning. 2017,. arXiv Google Scholar

[83] Ren M, Kiros R, Zemel R S. Exploring models and data for image question answering. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Montréal, 2015. 2953--2961. Google Scholar

[84] Zhu Y, Groth O, Bernstein M S, et al. Visual7w: grounded question answering in images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, 2016. 4995--5004. Google Scholar

[85] Li J Z, Su H, Zhu J, et al. Essay-anchor attentive multi-modal bilinear pooling for textbook question answering. In: Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), San Diego, 2018. 1--6. Google Scholar

Copyright 2020  CHINA SCIENCE PUBLISHING & MEDIA LTD.  中国科技出版传媒股份有限公司  版权所有

京ICP备14028887号-23       京公网安备11010102003388号