lnu.sePublications
Change search
Link to record
Permanent link

Direct link
BETA
Ulan, Maria
Publications (4 of 4) Show all publications
Ulan, M. (2019). Foundation of Multi-Criteria Quality Scoring. (Licentiate dissertation). Växjö: Linnaeus University Press
Open this publication in new window or tab >>Foundation of Multi-Criteria Quality Scoring
2019 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Software quality becomes more critical as our dependence on software increases. We need better quality assessment than ever. Comparison and ranking of software artifacts, detection of bad or good quality are important tasks for quality assessment.

Software quality models are widely used to support quality assessment. In general, they have a hierarchical structure and defines quality in terms of sub-qualities and metrics in a tree-like structure. Different metrics evaluate different quality criteria, and several metrics often needs to be assessed and aggregated to obtain a total quality score. The quality models standards of today do not enable numerical metrics aggregation. They leave aggregation to decision makers, and different methods of aggregation lead to different assessment results and interpretations. Hence, there is a need to define metrics aggregation formally based on well-known theories.

We propose to consider the probabilistic nature of quality as a solution. We consider metrics as random variables and define quality scores based on joint probabilities. The aggregation, and the quality model in extension, express quality as the probability of detecting something with equal or worse quality, based on all software projects observed; good and bad quality is expressed in terms of lower and higher probabilities. We analyze metrics dependencies using Bayesian networks and define quality models as directed acyclic graphs. Nodes correspond to metrics, and edges indicate dependencies. We propose an implementation using multi-threading to improve the efficiency of joint probabilities computations.

We validate our approach theoretically and in an empirical study on software quality assessment of approximately 100\,000 real-world software artifacts with approximately 4\,000\,000 measurements in total. The results show that our approach gives likely results and scales in performance to large projects.

We also applied our approach to a multi-criteria decision-making task to propose a ranking method to aid evaluation processes. We use a real-world funding allocation problem for a call that attracted approximately 600 applications to evaluate our approach. We compared our approach with the traditional weighted sum aggregation model and found that ranks are similar between the two methods, but our approach provides a more sound basis for a fair assessment.

Further, we implemented an exploratory multivariate data visualization tool, which visualizes the similarities between software artifacts based on joint distributions. We illustrate the usability of our tool with two case studies of real-world examples: a set of technical documents and an open source project written in Java.

Our overall results show that our approach for multi-criteria quality scoring is well-defined, has a clear interpretation, and is applicable under realistic conditions, generalizable, and transferable to other domains.

Place, publisher, year, edition, pages
Växjö: Linnaeus University Press, 2019
National Category
Computer and Information Sciences
Research subject
Computer and Information Sciences Computer Science
Identifiers
urn:nbn:se:lnu:diva-82619 (URN)
Presentation
2019-06-07, B2018, Hus B, Växjö, 10:15 (English)
Opponent
Supervisors
Available from: 2019-05-29 Created: 2019-05-20 Last updated: 2019-05-29Bibliographically approved
Ulan, M., Ericsson, M., Löwe, W. & Wingkvist, A. (2019). Multi-criteria Ranking Based on Joint Distributions: A Tool to Support Decision Making. In: Pańkowska M., Sandkuhl K (Ed.), Perspectives in Business Informatics Research.BIR 2019: 18th International Conference on Business Informatics Research. Paper presented at 18th International Conference, BIR 2019, Katowice, Poland, September 23-25, 2019 (pp. 74-88). Springer
Open this publication in new window or tab >>Multi-criteria Ranking Based on Joint Distributions: A Tool to Support Decision Making
2019 (English)In: Perspectives in Business Informatics Research.BIR 2019: 18th International Conference on Business Informatics Research / [ed] Pańkowska M., Sandkuhl K, Springer, 2019, p. 74-88Conference paper, Published paper (Refereed)
Abstract [en]

Sound assessment and ranking of alternatives are fundamental to effective decision making. Creating an overall ranking is not trivial if there are multiple criteria, and none of the alternatives is the best according to all criteria. To address this challenge, we propose an approach that aggregates criteria scores based on their joint (probability) distribution and obtains the ranking as a weighted product of these scores. We evaluate our approach in a real-world use case based on a funding allocation problem and compare it with the traditional weighted sum aggregation model. The results show that the approaches assign similar ranks, while our approach is more interpretable and sensitive.

Place, publisher, year, edition, pages
Springer, 2019
Series
Lecture Notes in Business Information Processing, ISSN 1865-1348, E-ISSN 1865-1356 ; 365
Keywords
Aggregation, Management by objectives, Ranking
National Category
Software Engineering
Research subject
Computer and Information Sciences Computer Science, Computer Science; Computer Science, Software Technology
Identifiers
urn:nbn:se:lnu:diva-89171 (URN)10.1007/978-3-030-31143-8_6 (DOI)978-3-030-31142-1 (ISBN)978-3-030-31143-8 (ISBN)
Conference
18th International Conference, BIR 2019, Katowice, Poland, September 23-25, 2019
Funder
Knowledge Foundation, 20150088
Available from: 2019-09-17 Created: 2019-09-17 Last updated: 2019-09-18Bibliographically approved
Ulan, M., Löwe, W., Ericsson, M. & Wingkvist, A. (2018). Introducing Quality Models Based On Joint Probabilities: Introducing Quality Models Based On Joint Probabilities. In: ICSE '18 Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings: . Paper presented at 40th ACM/IEEE International Conference on Software Engineering (ICSE), MAY 27-JUN 03, 2018, Gothenburg, SWEDEN (pp. 216-217). IEEE
Open this publication in new window or tab >>Introducing Quality Models Based On Joint Probabilities: Introducing Quality Models Based On Joint Probabilities
2018 (English)In: ICSE '18 Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, IEEE, 2018, p. 216-217Conference paper, Poster (with or without abstract) (Refereed)
Abstract [en]

Multi-dimensional goals can be formalized in so-called quality models. Often, each dimension is assessed with a set of metrics that are not comparable; they come with different units, scale types, and distributions of values. Aggregating the metrics to a single quality score in an ad-hoc manner cannot be expected to provide a reliable basis for decision making. Therefore, aggregation needs to be mathematically well-defined and interpretable. We present such a way of defining quality models based on joint probabilities. We exemplify our approach using a quality model with 30 standard metrics assessing technical documentation quality and study ca. 20,000 real-world files. We study the effect of several tests on the independence and results show that metrics are, in general, not independent. Finally, we exemplify our suggested definition of quality models in this domain.

Place, publisher, year, edition, pages
IEEE, 2018
Series
Proceedings of the IEEE-ACM International Conference on Software Engineering Companion, ISSN 2574-1926
Keywords
Quality assessment, Software metrics, Bayesian networks
National Category
Software Engineering
Research subject
Computer Science, Software Technology
Identifiers
urn:nbn:se:lnu:diva-79015 (URN)10.1145/3183440.3195103 (DOI)000450109000078 ()2-s2.0-85049686901 (Scopus ID)978-1-4503-5663-3 (ISBN)
Conference
40th ACM/IEEE International Conference on Software Engineering (ICSE), MAY 27-JUN 03, 2018, Gothenburg, SWEDEN
Available from: 2018-12-06 Created: 2018-12-06 Last updated: 2019-08-29Bibliographically approved
Ulan, M., Hönel, S., Martins, R. M., Ericsson, M., Löwe, W., Wingkvist, A. & Kerren, A. (2018). Quality Models Inside Out: Interactive Visualization of Software Metrics by Means of Joint Probabilities. In: J. Ángel Velázquez Iturbide, Jaime Urquiza Fuentes, Andreas Kerren, and Mircea F. Lungu (Ed.), Proceedings of the 2018 Sixth IEEE Working Conference on Software Visualization, (VISSOFT), Madrid, Spain, 2018: . Paper presented at IEEE Working Conference on Software Visualization (VISSOFT), Madrid, Spain, 24-25 September, 2018 (pp. 65-75). IEEE
Open this publication in new window or tab >>Quality Models Inside Out: Interactive Visualization of Software Metrics by Means of Joint Probabilities
Show others...
2018 (English)In: Proceedings of the 2018 Sixth IEEE Working Conference on Software Visualization, (VISSOFT), Madrid, Spain, 2018 / [ed] J. Ángel Velázquez Iturbide, Jaime Urquiza Fuentes, Andreas Kerren, and Mircea F. Lungu, IEEE, 2018, p. 65-75Conference paper, Published paper (Refereed)
Abstract [en]

Assessing software quality, in general, is hard; each metric has a different interpretation, scale, range of values, or measurement method. Combining these metrics automatically is especially difficult, because they measure different aspects of software quality, and creating a single global final quality score limits the evaluation of the specific quality aspects and trade-offs that exist when looking at different metrics. We present a way to visualize multiple aspects of software quality. In general, software quality can be decomposed hierarchically into characteristics, which can be assessed by various direct and indirect metrics. These characteristics are then combined and aggregated to assess the quality of the software system as a whole. We introduce an approach for quality assessment based on joint distributions of metrics values. Visualizations of these distributions allow users to explore and compare the quality metrics of software systems and their artifacts, and to detect patterns, correlations, and anomalies. Furthermore, it is possible to identify common properties and flaws, as our visualization approach provides rich interactions for visual queries to the quality models’ multivariate data. We evaluate our approach in two use cases based on: 30 real-world technical documentation projects with 20,000 XML documents, and an open source project written in Java with 1000 classes. Our results show that the proposed approach allows an analyst to detect possible causes of bad or good quality.

Place, publisher, year, edition, pages
IEEE, 2018
Keywords
hierarchical data exploration, multivariate data visualization, joint probabilities, t-SNE, data abstraction
National Category
Human Computer Interaction Software Engineering
Research subject
Computer Science, Information and software visualization; Computer Science, Software Technology
Identifiers
urn:nbn:se:lnu:diva-78093 (URN)10.1109/VISSOFT.2018.00015 (DOI)2-s2.0-85058463111 (Scopus ID)978-1-5386-8292-0 (ISBN)978-1-5386-8293-7 (ISBN)
Conference
IEEE Working Conference on Software Visualization (VISSOFT), Madrid, Spain, 24-25 September, 2018
Projects
Software technology for self-adaptive systems
Funder
Knowledge Foundation, 20150088
Available from: 2018-10-01 Created: 2018-10-01 Last updated: 2019-08-29Bibliographically approved
Organisations

Search in DiVA

Show all publications