logo

SCIENTIA SINICA Informationis, Volume 49, Issue 10: 1234-1266(2019) https://doi.org/10.1360/N112019-00003

Automatic test-input generation for Android applications

More info
  • ReceivedJan 14, 2019
  • AcceptedMar 27, 2019
  • PublishedOct 16, 2019

Abstract

Automatic test-input generation is an important and expensive testing activity that greatly impacts the effectiveness of automatic testing. There are unique challenges to automatically generating test inputs for Android applications (apps) due to the unique mechanism of the Android platform. Therefore, numerous automatic test-input generation methods for Android apps have been proposed. This study proposes a description framework to demonstrate the key issues in automatic test-input generation and includes three dimensions to describe the technique (representation of search space and the generation and evaluation of candidate test inputs) and two performance metrics for the dimensions (thoroughness and efficiency). Furthermore, existing techniques, as well as potential future work, are discussed.


Funded by

国家重点研发计划(2017YFB1001801)

国家自然科学基金(61690204,61802165)


Acknowledgment

感谢江苏省软件新技术与产业化协同创新中心.


References

[1] Xu J F. Software automation. J Comput Res Develop, 1994, 11: 9--15. Google Scholar

[2] Yao L, Shu Y A. Research on key techniques for software automatic testing. J Anhui Univ Natl Sci Edit, 2003, 27: 14. Google Scholar

[3] Number of Android apps on Google play. https://www.appbrain.com/stats/number-of-android-apps. Google Scholar

[4] Rubinov K, Baresi L. What Are We Missing When Testing Our Android Apps?. Computer, 2018, 51: 60-68 CrossRef Google Scholar

[5] Choudhary S R, Gorla A, Orso A. Automated test input generation for Android: are we there yet? In: Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), Lincoln, 2015. 429--440. Google Scholar

[6] Sahinoglu M, Incki K, Aktas M S. Mobile application verification: a systematic mapping study. In: Proceedings of International Conference on Computational Science and its Applications, Banff, 2015. 147--163. Google Scholar

[7] Zein S, Salleh N, Grundy J. A systematic mapping study of mobile application testing techniques. J Syst Software, 2016, 117: 334-356 CrossRef Google Scholar

[8] Kong P, Li L, Gao J. Automated Testing of Android Apps: A Systematic Literature Review. IEEE Trans Rel, 2019, 68: 45-66 CrossRef Google Scholar

[9] Su T, Meng G Z, Chen Y T, et al. Guided, stochastic model-based GUI testing of Android apps. In: Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017), New York, 2017. 245--256. Google Scholar

[10] Li Q W, Jiang Y Y, Gu T X, et al. Effectively manifesting concurrency bugs in Android apps. In: Proceedings of the 23rd Asia-Pacific Software Engineering Conference (APSEC), Hamilton, 2016. 209--216. Google Scholar

[11] Wang J, Jiang Y, Xu C. AATT+: Effectively manifesting concurrency bugs in Android apps. Sci Comput Programming, 2018, 163: 1-18 CrossRef Google Scholar

[12] Bai G, Ye Q, Wu Y. Towards Model Checking Android Applications. IIEEE Trans Software Eng, 2018, 44: 595-612 CrossRef Google Scholar

[13] Gao X, Tan X H, Dong Z, et al. Android testing via synthetic symbolic execution. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE 2018), New York, 2018. 419--429. Google Scholar

[14] Yang G L, Huang J, Gu G F. Automated generation of event-oriented exploits in Android hybrid apps. In: Proceedings of Network and Distributed System Security Symposium, San Diego, 2018. Google Scholar

[15] Mirzaei N, Bagheri H, Mahmood R, et al. SIG-Droid: automated system input generation for Android applications. In: Proceedings of the 26th International Symposium on Software Reliability Engineering (ISSRE), Gaithersbury, 2015. 461--471. Google Scholar

[16] Fan L L, Su T, Chen S, et al. Efficiently manifesting asynchronous programming errors in Android apps. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE 2018), New York, 2018. 486--497. Google Scholar

[17] Jensen C S, Prasad M R, Møller A. Automated testing with targeted event sequence generation. In: Proceedings of International Symposium on Software Testing and Analysis (ISSTA 2013), New York, 2013. 67--77. Google Scholar

[18] Anand S, Naik M, Harrold M J, et al. Automated concolic testing of smartphone apps. In: Proceedings of the 20th International Symposium on the Foundations of Software Engineering (FSE'12), New York, 2012. Google Scholar

[19] Yepang Liu , Chang Xu , Cheung S C. GreenDroid: Automated Diagnosis of Energy Inefficiency for Smartphone Applications. IEEE Trans Software Eng, 2014, 40: 911-940 CrossRef Google Scholar

[20] Wang J, Liu Y P, Xu C, et al. E-greenDroid: effective energy inefficiency analysis for android applications. In: Proceedings of the 8th Asia-Pacific Symposium on Internetware (Internetware'16), Beijing, 2016. 71--80. Google Scholar

[21] Mirzaei N, Malek S, Pasareanu C S. Testing android apps through symbolic execution. SIGSOFT Softw Eng Notes, 2012, 37: 1 CrossRef Google Scholar

[22] Liu Y, Wang J, Xu C, et al. NavyDroid: detecting energy inefficiency problems for smartphone applications. In: Proceedings of the 9th Asia-Pacific Symposium on Internetware (Internetware'17), Shanghai, 2017. Google Scholar

[23] Zhang L L, Liang C M, Liu U X, et al. Systematically testing background services of mobile apps. In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2017), Piscataway, 2017. 4--15. Google Scholar

[24] Baek Y M, Bae D H. Automated model-based Android GUI testing using multi-level GUI comparison criteria. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2016), New York, 2016. 238--249. Google Scholar

[25] Choi W, Necula G, Sen K. Guided GUI testing of android apps with minimal restart and approximate learning. In: Proceedings of ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications (OOPSLA'13), New York, 2013. 623--640. Google Scholar

[26] Wu X Y, Jiang Y Y, Xu C, et al. Testing Android apps via guided gesture event generation. In: Proceedings of the 23rd Asia-Pacific Software Engineering Conference (APSEC), Hamilton, 2016. 201--208. Google Scholar

[27] Mao K, Harman M, Jia Y. Sapienz: multi-objective automated testing for Android applications. In: Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA 2016), New York, 2016. 94--105. Google Scholar

[28] Mahmood R, Mirzaei N, Malek S. EvoDroid: segmented evolutionary testing of Android apps. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014), New York, 2014. 599--609. Google Scholar

[29] Mao K, Harman M, Jia Y. Crowd intelligence enhances automated mobile testing. In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2017), Piscataway, 2017. 16--26. Google Scholar

[30] Sadeghi A, Jabbarvand R, Malek S. PATDroid: permission-aware GUI testing of Android. In: Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017), New York, 2017. 220--232. Google Scholar

[31] Koroglu Y, Sen A. TCM: test case mutation to improve crash detection in Android. In: Proceedings of International Conference on Fundamental Approaches to Software Engineering, Thessaloniki, 2018. 264--280. Google Scholar

[32] Adamsen C Q, Mezzetti G, Møller A. Systematic execution of Android test suites in adverse conditions. In: Proceedings of International Symposium on Software Testing and Analysis (ISSTA 2015), New York, 2015. 83--93. Google Scholar

[33] Wegener J. Evolutionary testing techniques. In: Proceedings of Stochastic Algorithms: Foundations and Applications, Berlin, 2005. 82--94. Google Scholar

[34] Fudenberg D, Tirole J. Game Theory. Cambridge: MIT Press, 1991. Google Scholar

[35] Stormo G D. DNA binding sites: representation and discovery.. Bioinformatics, 2000, 16: 16-23 CrossRef PubMed Google Scholar

[36] Mahmood R, Esfahani N, Kacem T, et al. A whitebox approach for automated security testing of Android applications on the cloud. In: Proceedings of the 7th International Workshop on Automation of Software Test (AST), Zurich, 2017. 22--28. Google Scholar

[37] Liu P, Zhang X Y, Pistoia M, et al. Automatic text input generation for mobile testing. In: Proceedings of the 39th International Conference on Software Engineering (ICSE'17), Piscataway, 2017. 643--653. Google Scholar

[38] Schmidhuber J. Deep learning in neural networks: an overview.. Neural Networks, 2015, 61: 85-117 CrossRef PubMed Google Scholar

[39] Goldberg Y, Levy O. word2vec explained: deriving Mikolov et al.'s negative-sampling word-embedding method. 2014,. arXiv Google Scholar

[40] Linares-Vásquez M, White M, Bernal-Cárdenas C, et al. Mining Android app usages for generating actionable GUI-based execution scenarios. In: Proceedings of the 12th Working Conference on Mining Software Repositories (MSR'15), Piscataway, 2015. 111--122. Google Scholar

[41] Amalfitano D, Fasolino A R, Tramontana P, et al. Considering context events in event-based testing of mobile applications. In: Proceedings of the 6th International Conference on Software Testing, Verification and Validation Workshops, Luxembourg, 2013. 126--133. Google Scholar

[42] UI/application exerciser monkey. https://developer.android.com/studio/test/monkey. Google Scholar

[43] Cao C, Meng C L, Ge H J, et al. Xdroid: testing Android apps with dependency injection. In: Proceedings of the 41st Annual Computer Software and Applications Conference (COMPSAC), Turin, 2017. 214--223. Google Scholar

[44] Shahriar H, North S, Mawangi E. Testing of memory leak in Android applications. In: Proceedings of the 15th International Symposium on High-Assurance Systems Engineering, Miami Beach, 2014. 176--183. Google Scholar

[45] Ye H, Cheng S Y, Zhang L B, et al. DroidFuzzer: fuzzing the Android apps with intent-filter tag. In: Proceedings of International Conference on Advances in Mobile Computing & Multimedia (MoMM'13), New York, 2013. Google Scholar

[46] Clarke L A. A program testing system. In: Proceedings of Annual Conference, Houston, 1976. 488--491. Google Scholar

[47] Lam W K. Hardware Design Verification: Simulation and Formal Method-Based Approaches. Upper Saddle River: Prentice Hall, 2008. Google Scholar

[48] Liu Y, Wang J, Xu C. NavyDroid: an efficient tool of energy inefficiency problem diagnosis for Android applications. Sci China Inf Sci, 2018, 61: 050103 CrossRef Google Scholar

[49] Anand S, Naik M, Harrold M J, et al. Automated concolic testing of smartphone apps. In: Proceedings of the 20th International Symposium on the Foundations of Software Engineering (FSE'12), New York, 2012. Google Scholar

[50] Yeh C C, Lu H L, Chen C Y, et al. CRAXDroid: automatic Android system testing by selective symbolic execution. In: Proceedings of the 8th International Conference on Software Security and Reliability-Companion, San Francisco, 2014. 140--148. Google Scholar

[51] Zhang L L, Liang C M, Liu Y X, et al. Systematically testing background services of mobile apps. In: Proceedingsof the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2017), Piscataway, 2017.4?C15. Google Scholar

[52] Zhang L L, Liang C M, Liu Y X, et al. Systematically testing background services of mobile apps. In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2017), Piscataway, 2017. 4--15. Google Scholar

[53] Keng J C J, Jiang L X, Wee Y K, et al. Graph-aided directed testing of Android applications for checking runtime privacy behaviours. In: Proceedings of the 11th International Workshop on Automation of Software Test (AST'16), New York, 2016. 57--63. Google Scholar

[54] Griebe T, Gruhn V. A model-based approach to test automation for context-aware mobile applications. In: Proceedings of the 29th Annual ACM Symposium on Applied Computing (SAC'14), New York, 2014. 420--427. Google Scholar

[55] James L. Peterson. Petri Nets. ACM Computing Surveys, 1997, 9: 223-252. Google Scholar

[56] Yang K, Zhuge J W, Wang Y K, et al. IntentFuzzer: detecting capability leaks of Android applications. In: Proceedings of the 9th ACM Symposium on Information, Computer and Communications Security (ASIA CCS'14), New York, 2014. 531--536. Google Scholar

[57] Ma J, Liu S C, Jiang Y Y, et al. LESdroid: a tool for detecting exported service leaks of Android applications. In: Proceedings of the 26th Conference on Program Comprehension (ICPC'18), New York, 2018. 244--254. Google Scholar

[58] Amalfitano D, Fasolino A R, Tramontana P. A GUI crawling-based technique for Android mobile application testing. In: Proceedings of the 4th International Conference on Software Testing, Verification and Validation Workshops, Berlin, 2011. 252--261. Google Scholar

[59] Azim T, Neamtiu I. Targeted and depth-first exploration for systematic testing of android apps. SIGPLAN Not, 2013, 48: 641-660 CrossRef Google Scholar

[60] Song W, Qian X X, Huang J. EHBDroid: Beyond GUI testing for Android applications. In: Proceedings of IEEE/ACM International Conference on Automated Software Engineering, 2017. 27--37. Google Scholar

[61] Tang H Y, Wu G Q, Wei J, et al. Generating test cases to expose concurrency bugs in Android applications. In: Proceedings of IEEE/ACM International Conference on Automated Software Engineering, Singapore, 2016. 648--653. Google Scholar

[62] Amalfitano D, Fasolino A R, Tramontana P. MobiGUITAR -- A Tool for Automated Model-Based Testing of Mobile Apps. IEEE Softw, 2015, 32: 53-59 CrossRef Google Scholar

[63] Moran K, Linares-Vásquez M, Bernal-Cárdenas C, et al. Automatically discovering, reporting and reproducing Android application crashes. In: Proceedings of IEEE International Conference on Software Testing, Verification and Validation, Kyoto, 2016. 33--44. Google Scholar

[64] Dong F, Wang H Y, Li L, et al. FraudDroid: automated ad fraud detection for Android apps. In: Proceedings of the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Lake Buena Vista, 2018. 257--268. Google Scholar

[65] Arnatovich Y L, Ngo M N, Tan H B K, et al. Achieving high code coverage in Android UI testing via automated widget exercising. In: Proceedings of the 23rd Asia-Pacific Software Engineering Conference (APSEC), Hamilton, 2017. 193--200. Google Scholar

[66] Adamo D, Nurmuradov D, Piparia S. Combinatorial-based event sequence testing of Android applications. Inf Software Tech, 2018, 99: 98-117 CrossRef Google Scholar

[67] Koroglu Y, Sen A, Muslu O, et al. QBE: Qlearning-based exploration of Android applications. In: Proceedings of the 11th International Conference on Software Testing, Verification and Validation (ICST), Vasteras, 2018. 105--115. Google Scholar

[68] Watkins C J C H. Learning from delayed rewards. Dissertation for Ph.D. Degree. Cambridge: Cambridge University, 1989. Google Scholar

[69] Gu T X, Cao C, Liu T C, et al. AimDroid: activity-insulated multi-level automated testing for Android applications. In: Proceedings of IEEE International Conference on Software Maintenance and Evolution, Shanghai, 2017. 103--114. Google Scholar

[70] Machiry A, Tahiliani R, Naik M. Dynodroid: an input generation system for Android apps. In: Proceedings of the 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2013), New York, 2013. 224--234. Google Scholar

[71] Hao S, Liu B, Nath S, et al. PUMA: programmable UI-automation for large-scale dynamic analysis of mobile apps. In: Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys'14), New York, 2014. 204--217. Google Scholar

[72] Ma J, Liu S, Yue S T, et al. LeakDAF: an automated tool for detecting leaked activities and fragments of Android applications. In: Proceedings of the 41st Annual Computer Software and Applications Conference (COMPSAC), Turin, 2017. 23--32. Google Scholar

[73] Liang C M, Lane N D, Brouwers N, et al. Caiipa: automated large-scale mobile app testing through contextual fuzzing. In: Proceedings of the 20th Annual International Conference on Mobile Computing and Networking (MobiCom'14), New York, 2014. 519--530. Google Scholar

[74] Zhu H W, Ye X J, Zhang X J, et al. A context-aware approach for dynamic GUI testing of Android applications. In: Proceedings of the 39th Annual Computer Software and Applications Conference, Taichung, 2015. 248--253. Google Scholar

[75] Rosenfeld A, Kardashov O, Zang O. Automation of Android applications testing using machine learning activities classification. In: Proceedings of the 5th International Conference on Mobile Software Engineering and Systems, Montréal 2018. 122-0132. Google Scholar

[76] Yang W, Prasad M R, Xie T. A grey-box approach for automated GUI-model generation of mobile applications. In: Proceedings of the 16th International Conference on Fundamental Approaches to Software Engineering (FASE'13), Berlin, 2013. 250--265. Google Scholar

[77] Chen J, Han G, Guo S Q, et al. FragDroid: automated user interface interaction with activity and fragment analysis in Android applications. In: Proceedings of the 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), Luxembourg City, 2018. 398--409. Google Scholar

[78] Haowei Wu, Yan Wang, Atanas Rountev. Sentinel: generating GUI tests for Android sensor leaks. In: Proceedings of the 13th International Workshop on Automation of Software Test (AST'18). ACM, New York, NY, USA, 2018. 27-33. Google Scholar

[79] Pradel M, Schuh P, Necula G, et al. EventBreak: analyzing the responsiveness of user interfaces through performance-guided test generation. In: Proceedings of ACM International Conference on Object Oriented Programming Systems Languages & Applications (OOPSLA'14), New York, 2014. 33--47. Google Scholar

[80] Banerjee A, Chong L K, Chattopadhyay S, et al. Detecting energy bugs and hotspots in mobile apps. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014), New York, 2014. 588--598. Google Scholar

[81] Amalfitano D, Amatucci N, Fasolino A R, et al. AGRippin: a novel search based testing technique for Android applications. In: Proceedings of the 3rd International Workshop on Software Development Lifecycle for Mobile (DeMobile 2015), New York, 2015. 5--12. Google Scholar

[82] Mirzaei N, Garcia J, Bagheri H, et al. Reducing combinatorics in GUI testing of android applications. In: Proceedings of the 38th International Conference on Software Engineering (ICSE'16), New York, 2016. 559--570. Google Scholar

[83] Jespersen N S. An Introduction to Markov Chain Monte Carlo. Soc Sci Electron, 2010, 11: 395-402 CrossRef Google Scholar

[84] Li X J, Jiang Y Y, Liu Y P, et al. User guided automation for testing mobile apps. In: Proceedings of the 21st Asia-Pacific Software Engineering Conference, Jeju, 2014. 27--34. Google Scholar

Copyright 2020 Science China Press Co., Ltd. 《中国科学》杂志社有限责任公司 版权所有

京ICP备18024590号-1