logo

SCIENCE CHINA Information Sciences, Volume 62, Issue 9: 191101(2019) https://doi.org/10.1007/s11432-018-9608-3

Software quality assessment model: a systematic mapping study

More info
  • ReceivedApr 8, 2018
  • AcceptedSep 7, 2018
  • PublishedJul 26, 2019

Abstract

Quality model is regarded as a well-accepted approach for assessing, managing and improving software product quality. There are three categories of quality models for software products, i.e., definition model, assessment model, and prediction model. Quality assessment model (QAM) is a metric-based approach to assess the software quality. It is typically regarded as of high importance for its clear method on how to assess a system. However, the current state-of-the-art in QAM research is under limited investigation. To address this gap, the paper provides an organized and synthesized summary of the current QAMs. In detail, we conduct a systematic mapping study (SMS) for structuring the relevant articles. We obtain a total of 716 papers from the five databases, and 31 papers are selected as relevant studies at last. In summary, our work focuses on QAMs from the following aspects: software metrics, quality factors, aggregation methods, evaluation methods and tool support. According to the analysis results, our work discovers five needs that researchers in this area should continue to address: (1) new method and criteria to tailor a quality framework (i.e., structure of software metrics and quality factors) according to different specifics, (2) systematic investigations on the effectiveness, strength and weakness of different aggregation methods to guide the method selection in different context, (3) more investigations on evaluating QAMs in the context of industrial cases, (4) further investigations or real-world case studies on the QAMs related tools, and (5) building a public and diverse software benchmark which can be adopted in different application context.


Acknowledgment

This work was partially supported by National Key Research and Development Program of China (Grant No. 2018YFB1003904), National Natural Science Foundation of China (Grant No. 61602403), and China Postdoctoral Science Foundation (Grant No. 2017M621931).


References

[1] ISO. Information technology, software product evaluation. ISO/IEC 14598-1. 1999. https://www.iso.org/standard/24902.html. Google Scholar

[2] Bakota T, Hegedus P, Kortvelyesi P, et al. A probabilistic software quality model. In: Proceedings of the 27th IEEE International Conference on Software Maintenance, 2011. 243--252. Google Scholar

[3] Boehm B W, Brown J R, Kaspar H. Characteristics of software quality. North-Holland 1978. Google Scholar

[4] ISO. Software engineering-product quality. ISO/IEC 9126. 2001. https://www.iso.org/standard/22749.html. Google Scholar

[5] ISO. Systems and software engineering-systems and software quality requirements and evaluation (SQuaRE)-system and software quality models. ISO/IEC 25010. 2011. https://www.iso.org/standard/35733.html. Google Scholar

[6] Catal C, Diri B. A systematic review of software fault prediction studies. Expert Syst Appl, 2009, 36: 7346-7354 CrossRef Google Scholar

[7] ?ztürk M M, Cavusoglu U, Zengin A. A novel defect prediction method for web pages using k-means+. Expert Syst Appl, 2015, 42: 6496-6506 CrossRef Google Scholar

[8] Deissenboeck F, Juergens E, Lochmann K, et al. Software quality models: purposes, usage scenarios and requirements. In: Proceedings of the ICSE Workshop on Software Quality, 2009. 9--14. Google Scholar

[9] Dromey R G. A model for software product quality. IEEE Trans Software Eng, 1995, 21: 146-162 CrossRef Google Scholar

[10] ISO. Systems and software engineering -- vocabulary. ISO/IEC/IEEE 24765. 2010. https://www.iso.org/standard/50518.html. Google Scholar

[11] Wagner S. Software Product Quality Control. Berlin: Springer, 2013. Google Scholar

[12] Chotjaratwanich U, Arpnikanondt C. A visualization technique for metrics-based hierarchical quality models. In: Proceedings of the 19th Asia-Pacific Software Engineering Conference (APSEC), 2012. 733--736. Google Scholar

[13] Mordal-Manet K, Balmas F, Denier S, et al. The squale model-a practice-based industrial quality model. In: Proceedings of the IEEE International Conference on Software Maintenance, 2009. 531--534. Google Scholar

[14] Klas M, Heidrich J, Munch J, et al. CQML scheme: a classification scheme for comprehensive quality model landscapes. In: Proceedings of the EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), 2009. 243--250. Google Scholar

[15] Montagud S, Abrah?o S, Insfran E. A systematic review of quality attributes and measures for software product lines. Software Qual J, 2012, 20: 425-486 CrossRef Google Scholar

[16] Riaz M, Mendes E, Tempero E. A systematic review of software maintainability prediction and metrics. In: Proceedings of the 3rd International Symposium on Empirical Software Engineering and Measurement, 2009. 367--377. Google Scholar

[17] Febrero F, Calero C, Moraga M. A Systematic Mapping Study of Software Reliability Modeling. Inf Software Tech, 2014, 56: 839-849 CrossRef Google Scholar

[18] Kitchenham B. What's up with software metrics? - A preliminary mapping study. J Syst Software, 2010, 83: 37-51 CrossRef Google Scholar

[19] Tomas P, Escalona M J, Mejias M. Open source tools for measuring the Internal Quality of Java software products. A survey. Comput Standards Interfaces, 2013, 36: 244-255 CrossRef Google Scholar

[20] Petersen K, Feldt R, Mujtaba S. Systematic mapping studies in software engineering. In: Proceedings of the International Conference on Evaluation and Assessment in Software Engineering (EASE), 2008. 68--77. Google Scholar

[21] Yan M, Xia X, Zhang X H, et al. A systematic mapping study of quality assessment models for software products. In: Proceedings of the International Conference on Software Analysis, Testing and Evolution (SATE), 2017. 63--71. Google Scholar

[22] Kitchenham B. Guidelines for Performing Systematic Literature Reviews in Software Engineering. EBSE Technical Report EBSE-2007-01. 2007. Google Scholar

[23] Brereton P, Kitchenham B A, Budgen D. Lessons from applying the systematic literature review process within the software engineering domain. J Syst Software, 2007, 80: 571-583 CrossRef Google Scholar

[24] Chernyi A I. The ISI Web of Knowledge, a modern system for the information support of scientific research: a review. Sci TechInf Proc, 2009, 36: 351-358 CrossRef Google Scholar

[25] Webster J, Watson R T. Analyzing the past to prepare for the future: writing a literature review. Manag Inf Syst Quart, 2002, 26: 3. Google Scholar

[26] Kvam K, Lie R, Bakkelund D. Legacy system exorcism by pareto's principle. In: Proceedings of the Companion to the 20th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, 2005. 250--256. Google Scholar

[27] Morisio M, Stamelos I, Tsoukias A. SOFTWARE PRODUCT AND PROCESS ASSESSMENT THROUGH PROFILE-BASED EVALUATION. Int J Soft Eng Knowl Eng, 2003, 13: 495-512 CrossRef Google Scholar

[28] Satrijandi N, Widyani Y. Efficiency measurement of java android code. In: Proceedings of the International Conference on Data and Software Engineering, 2014. Google Scholar

[29] Wagner S, Goeb A, Heinemann L. Operationalised product quality models and assessment: The Quamoco approach. Inf Software Tech, 2015, 62: 101-123 CrossRef Google Scholar

[30] Xianrong Zheng , Martin P, Brohman K. CLOUDQUAL: A Quality Model for Cloud Services. IEEE Trans Ind Inf, 2014, 10: 1527-1536 CrossRef Google Scholar

[31] Mayr A, Plosch R, Saft M. Objective safety compliance checks for source code. In: Proceedings of the 36th International Conference on Software Engineering, 2014. 115--124. Google Scholar

[32] Gupta S, Singh H K, Venkatasubramanyam R D, et al. SCQAM: a scalable structured code quality assessment method for industrial software. In: Proceedings of the 22nd International Conference on Program Comprehension, 2014. 244--252. Google Scholar

[33] Athanasiou D, Nugroho A, Visser J. Test Code Quality and Its Relation to Issue Handling Performance. IEEE Trans Software Eng, 2014, 40: 1100-1125 CrossRef Google Scholar

[34] Srivastava S, Kumar R. Indirect method to measure software quality using ck-oo suite. In: Proceedings of the International Conference on Intelligent Systems and Signal Processing, 2013. 47--51. Google Scholar

[35] Samarthyam G, Suryanarayana G, Sharma T. Midas: a design quality assessment method for industrial software. In: Proceedings of the International Conference on Software Engineering, 2013. 911--920. Google Scholar

[36] Mayr A, Plosch R, Saft M. Objective measurement of safety in the context of IEC 61508-3. In: Proceedings of the EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), 2013. 45--52. Google Scholar

[37] Mayr A, Plosch R, Klas M, et al. A comprehensive code-based quality model for embedded systems: systematic development and validation by industrial projects. In: Proceedings of the 23rd International Symposium on Software Reliability Engineering (ISSRE), 2012. 281--290. Google Scholar

[38] Baggen R, Correia J P, Schill K. Standardized code quality benchmarking for improving software maintainability. Software Qual J, 2012, 20: 287-307 CrossRef Google Scholar

[39] Wagner S, Lochmann K, Heinemann L, et al. The quamoco product quality modelling and assessment approach. In: Proceedings of the International Conference on Software Engineering, 2012. 1133--1142. Google Scholar

[40] Challa J S, Paul A, Dada Y. Integrated Software Quality Evaluation: A Fuzzy Multi-Criteria Approach. J Inf Processing Syst, 2011, 7: 473-518 CrossRef Google Scholar

[41] Lan Y Q, Liu Y F, Kuang M X. Evaluate the quality of foundational software platform by Bayesian network. In: Proceedings of International Conference on Artificial Intelligence and Computational Intelligence, 2010. Google Scholar

[42] Glott R, Groven A K, Haaland K, et al. Quality models for free/libre open source software towards the “silver bullet"? In: Proceedings of the 36th EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), 2010. 439--446. Google Scholar

[43] Soley R M, Curtis B. The consortium for IT software quality. In: Proceedings of International Conference on Software Quality, 2013. Google Scholar

[44] Letouzey J L, Coq T. The sqale analysis model: an analysis model compliant with the representation condition for assessing the quality of software source code. In: Proceedings of the International Conference on Advances in System Testing and Validation Lifecycle, 2010. 43--48. Google Scholar

[45] Marchetto A. OQMw: an OO quality model for web applications. Tamkang J Sci Eng, 2009, 12: 459--470. Google Scholar

[46] Khomh F, Gueheneuc Y G. Dequalite: building design-based software quality models. In: Proceedings of the 15th Conference on Pattern Languages of Programs, 2008. Google Scholar

[47] Li Z, Lin L, Hui G. 2-d software quality model and case study in software flexibility research. In: Proceedings of the International Conference on Intelligence for Modelling Control and Automation, 2008. 1147--1152. Google Scholar

[48] Pl?sch R, Gruber H, Hentschel A. The EMISQ method and its tool support-expert-based evaluation of internal software quality. Innovations Syst Softw Eng, 2008, 4: 3-15 CrossRef Google Scholar

[49] Samoladas I, Gousios G, Spinellis D. The SQO-OSS Quality Model: Measurement Based Open Source Software Evaluation. In: Proceedings of IFIP International Conference on Open Source Systems, 2008. 237--248. Google Scholar

[50] Ortega M, Pérez M, Rojas T. Construction of a systemic quality model for evaluating a software product. Software Qual J, 2003, 11: 219-242 CrossRef Google Scholar

[51] Franch X, Carvallo J P. Using quality models in software package selection. IEEE Softw, 2003, 20: 34--41. Google Scholar

[52] Bansiya J, Davis C G. A hierarchical model for object-oriented design quality assessment. IEEE Trans Software Eng, 2002, 28: 4-17 CrossRef Google Scholar

[53] Blin M J, Tsoukiàs A. Multi-criteria methodology contribution to the software quality evaluation. Software Qual J, 2001, 9: 113-132 CrossRef Google Scholar

[54] Pedrycz W, Peters J F, Ramanna S. Software quality measurement: concepts and fuzzy neural relational model. In: Proceedings of the IEEE International Conference on Fuzzy Systems Proceedings, 1998, 1026--1031. Google Scholar

[55] Gomez O, Oktaba H, Piattini M, et al. A systematic review measurement in software engineering: state-of-the-art in measures. In: Poroceedings of International Conference on Software and Data Technologies, 2008. 165--176. Google Scholar

[56] McCabe T J. A complexity measure. IEEE Trans Softw Eng, 1976, 2: 308--320. Google Scholar

[57] Halstead M H. Elements of Software Science. New York: Elsevier Science Inc., 1977. Google Scholar

[58] Chidamber S R, Kemerer C F. A metrics suite for object oriented design. IEEE Trans Software Eng, 1994, 20: 476-493 CrossRef Google Scholar

[59] Martin R C. Agile Software Development: Principles, Patterns, and Practices. Upper Saddle River: Prentice Hall PTR, 2003. Google Scholar

[60] Fowler M. Refactoring: Improving the Design of Existing Code. Boston: Addison-Wesley Longman Publishing, 1999. Google Scholar

[61] Deissenboeck F, Wagner S, Pizka M, et al. An activity-based quality model for maintainability. In: Proceedings of the IEEE International Conference on Software Maintenance, 2007. 184--193. Google Scholar

[62] Malhotra R. A systematic review of machine learning techniques for software fault prediction. Appl Soft Computing, 2015, 27: 504-518 CrossRef Google Scholar

[63] Ploesch R, Schuerz S, Koerner C. On the validity of the it-cisq quality model for automatic measurement of maintainability. In: Proceedings of the 39th Annual Computer Software and Applications Conference (COMPSAC), 2015. 326--334. Google Scholar

[64] IEEE. IEEE recommended practice for software requirements specifications. In: Institute of Electrical and Electronics Engineers, 1998. Google Scholar

[65] Barron F H, Barrett B E. Decision Quality Using Ranked Attribute Weights. Manage Sci, 1996, 42: 1515-1523 CrossRef Google Scholar

[66] Correia J P, Kanellopoulos Y, Visser J. A survey-based study of the mapping of system properties to ISO/IEC 9126 maintainability characteristics. In: Proceedings of the IEEE International Conference on Software Maintenance, 2009. 61--70. Google Scholar

[67] Ross T J. Fuzzy Logic with Engineering Applications. Hoboken: John Wiley and Sons, 2009. Google Scholar

[68] Heckman S, Williams L. A systematic literature review of actionable alert identification techniques for automated static code analysis. Inf Software Tech, 2011, 53: 363-387 CrossRef Google Scholar

[69] K?ksalan M, Ulu C. An interactive approach for placing alternatives in preference classes. Eur J Operational Res, 2003, 144: 429-439 CrossRef Google Scholar

[70] Larichev O. An approach to ordinal classification problems. Int Trans Operational Res, 1994, 1: 375-385 CrossRef Google Scholar

[71] Mordal K, Anquetil N, Laval J. Software quality metrics aggregation in industry. J Softw Evol Proc, 2013, 25: 1117-1135 CrossRef Google Scholar

[72] Jorgensen M, Shepperd M. A Systematic Review of Software Development Cost Estimation Studies. IEEE Trans Software Eng, 2007, 33: 33-53 CrossRef Google Scholar

  • Figure 1

    The general concept of a QAM. A QAM depends on an aggregation step to aggregate metrics to quality factors. Similar to ISO 9126, the factors may be decomposed into sub-factors. In addition, there may be several number of middle layers between metrics and quality factors to make the model more comprehensive, such as the practice layer and criteria layer in Squale model [13].

  • Figure 2

    Distribution of papers by different features. (a) Metric category; (b) factors; (c) aggregation methods;protect łinebreak (d) evaluation methods.

  • Table 1   Research questions and motivations
    Research question Motivationtabularnewline
    RQ1. What metrics and factors are commonly used by QAMs? Identify the categories and trends of metrics and quality factorsused in QAMs.tabularnewline
    RQ2. What are the current aggregation methods used by QAMs? Summarize current aggregation methods in QAMs.tabularnewline
    RQ3. What are the current validation methods used for evaluating QAMs? Summarize the current validation methods in QAMs.tabularnewline
    RQ4. What are the current usage of tools based on QAMs? Identify the current usage of the related tools.tabularnewline
    RQ5. What are the challenges for the improvement of QAMs? Identify the challenges for the further improvement.tabularnewline
  • Table 2   Selected databases
    Database Location
    ISI web of knowledge isiknowledge.com
    Scopus www.scopus.com
    IEEE Xplore www.ieeexplore.ieee.org
    ACM digital library www.portal.acm.org
    Springer link.springer.com
  • Table 3   Search strings in different databases
    Database Search string and settingstabularnewline
    ISI web of knowledge TS=(((“software quality model*”) OR(“software quality” AND “qualitymodel*”)) AND (metric* OR measure*)tabularnewline
    Scopus TITLE-ABS((“software quality model*”)OR (“software quality” AND “qualitymodel*”) AND (metric* OR measure*) tabularnewline
    IEEE Xplore ((((“Document title”: “softwarequality model*” OR (“software quality”AND “quality model*”)) OR (Abstract: “softwarequality model*” OR (“software quality”AND “quality model*”))))) AND ((“Documenttitle”: “metric*” OR “measure*”)OR (“Abstract”: “metric*”OR “measure*”)) tabularnewline
    ACM digital library (((Title: “software quality model*”)OR ((Title: “software quality”) and (Title:“quality model*”))) OR ((Abstract: “softwarequality model*”) OR ((Abstract: “softwarequality”) and (Abstract: “quality model*”))))AND ((Title: “metric*”) OR (Title: “measure*”)OR (Abstract: “metric*”) OR (Abstract:“measure*”))tabularnewline
    Springer ` “software quality model” or (“softwarequality” and “quality model”)and (“measure*” or “metric*”)'tabularnewline
  • Table 4   Overview of search result
    Stage Papers Added papers Total paperstabularnewline
    Stage 1: by search strings 716 0 716tabularnewlineStage 2: by title and abstract 128 0 128tabularnewlineStage 3: by content 28 3 31tabularnewline
  • Table 5   Inclusion criteria (IC) and exclusion criteria (EC)
    IC Description tabularnewline
    1 The paper proposes a software quality assessment model.tabularnewline2 The paper is based on software product metrics.tabularnewline3 The paper focuses on software product quality rather than processquality.tabularnewline4 The paper presents a hierarchical mapping model which aggregates metricsto factors.
    EC Descriptiontabularnewline
    1 The paper focuses on software process quality.tabularnewline2 The paper focuses on a prediction model without an assessment model.tabularnewline3 The paper only provides a definition quality model without an assessmentmethod. tabularnewline4 The paper is not accessible.tabularnewline5 The document is not a paper, such as a conference cover, poster, etc.tabularnewline6 The paper is not written in English.tabularnewline
  • Table 6   Detailed information of selected studies
    Study ID Title Yeartabularnewline
    S1 [29] Operationalised product quality models and assessment: the Quamocoapproach 2015tabularnewlineS2 [30] CLOUDQUAL: a quality model for cloud services 2014tabularnewlineS3 [28] Efficiency measurement of Java Android code 2014tabularnewlineS4 [31] Objective safety compliance checks for source code 2014tabularnewlineS5 [32] SCQAM: a scalable structured code quality assessment method for industrialsoftware 2014tabularnewlineS6 [33] Test code quality and its relation to issue handling performance 2014tabularnewlineS7 [34] Indirect method to measure software quality using CK-OO suite 2013tabularnewlineS8 [35] MIDAS: a design quality assessment method for industrial software 2013tabularnewlineS9 [36] Objective measurement of safety in the context of IEC 61508-3 2013tabularnewlineS10 [37] A comprehensive code-based quality model for embedded systems 2012tabularnewlineS11 [38] Standardized code quality benchmarking for improving software maintainability 2012tabularnewlineS12 [39] The Quamoco product quality modelling and assessment approach 2012tabularnewlineS13 [2] A probabilistic software quality model 2011tabularnewlineS14 [40] Integrated software quality evaluation: a fuzzy multi-criteria approach 2011tabularnewlineS15 [41] Evaluate the quality of foundational software platform by Bayesiannetwork 2010tabularnewlineS16 [42] Quality models for free/libre open source software – towards the “silverbullet” ? 2010tabularnewlineS17 [43] The consortium for IT software quality 2010tabularnewlineS18 [44] The SQALE analysis model: an analysis model compliant with the representationcondition for 2010
    assessing the quality of software source code tabularnewlineS19 [45] OQMw: An OO quality model for web applications 2009tabularnewlineS20 [13] The Squale model – a practice-based industrial quality model 2009tabularnewlineS21 [46] DEQUALITE: building design-based software quality models 2008tabularnewlineS22 [47] 2-D software quality model and case study in software flexibilityresearch 2008tabularnewlineS23 [48] The EMISQ method and its tool support-expert-based evaluation of internalsoftware quality 2008tabularnewlineS24 [49] The SQO-OSS quality model: measurement based open source softwareevaluation 2008tabularnewlineS25 [26] Legacy system exorcism by Pareto's principle 2005tabularnewlineS26 [50] Construction of a systemic quality model for evaluating a softwareproduct 2003tabularnewlineS27 [27] Software product and process assessment through profile-based evaluation 2003tabularnewlineS28 [51] Using quality models in software package selection 2003tabularnewlineS29 [52] A hierarchical model for object-oriented design quality assessment 2002tabularnewlineS30 [53] Multi-criteria methodology contribution to the software quality evaluation 2001tabularnewlineS31 [54] Software quality measurement: concepts and fuzzy neural relationalmodel 1998tabularnewline
  • Table 7   Publication venues of selected studies
    Publication venue Type # of paperstabularnewline
    ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages,and Applications Conference 1tabularnewlineInternational Conference on Computational Intelligence for ModellingControl & Automation Conference 1tabularnewlineIEEE International Conference on Fuzzy Systems at the IEEE World Congresson Computational Intelligence Conference 1tabularnewlineIEEE International Conference on Software Maintenance and Evolution (ICSME) Conference 2tabularnewlineInternational Conference on Data and Software Engineering (ICODSE) Conference 1tabularnewlineInternational Conference on Intelligent Systems and Signal Processing(ISSP) Conference 1tabularnewlineInternational Conference on Software Engineering (ICSE) Conference 3tabularnewlineInternational Conference on Program Comprehension (ICPC) Conference 1tabularnewlineInternational Symposium on Software Reliability Engineering (ISSRE) Conference 1tabularnewlineInternational Conference on Advances in System Testing and ValidationLifecycle Conference 1tabularnewlineEuromicro Conference on Software Engineering and Advanced Applications(SEAA) Conference 2tabularnewline Innovations in Systems and Software Engineering Journal 1tabularnewline International Journal of Software Engineering and Knowledge Engineering Journal 1tabularnewline Information and Software Technology (IST) Journal 1tabularnewline IEEE Transactions on Software Engineering (TSE) Journal 2tabularnewline IEEE Transactions on Industrial Informatics Journal 1tabularnewline IEEE Software Journal 1tabularnewline Journal of Information Processing Systems Journal 1tabularnewline Lecture Notes in Computer Science Journal 1tabularnewline Open Source Development, Communities and Quality Journal 1tabularnewline Pattern Languages of Programs Journal 1tabularnewline Software Quality Journal (SQJ) Journal 3tabularnewline Tamkang Journal of Science and Engineering Journal 1tabularnewlineSoftware Engineering Approaches for Offshore and Outsourced Development Book 1tabularnewline
  • Table 8   Overview of the eight tools
    Tool Related study Current usage Publish year Open sourcetabularnewline
    SIG quality model S6, S11 Industrial 2007 NotabularnewlineQuamoco S1, S12 Industrial 2008 YestabularnewlineFSQQT S14 Academic 2011 NotabularnewlineSQALE S18 Industrial 2010 NotabularnewlineSquale S20 Industrial 2008 YestabularnewlineSPQR S23 Academic 2008 NotabularnewlineAlitheia S24 Academic 2008 YestabularnewlineXradar S25 Academic 2004 Yestabularnewline

Copyright 2020 Science China Press Co., Ltd. 《中国科学》杂志社有限责任公司 版权所有

京ICP备18024590号-1