Software Engineering
See recent articles
- [1] arXiv:2407.13839 [pdf, html, other]
-
Title: AROhI: An Interactive Tool for Estimating ROI of Data AnalyticsComments: Submitted to a conferenceSubjects: Software Engineering (cs.SE)
The cost of adopting new technology is rarely analyzed and discussed, while it is vital for many software companies worldwide. Thus, it is crucial to consider Return On Investment (ROI) when performing data analytics. Decisions on "How much analytics is needed"? are hard to answer. ROI could guide decision support on the What?, How?, and How Much? Analytics for a given problem. This work details a comprehensive tool that provides conventional and advanced ML approaches for demonstration using requirements dependency extraction and their ROI analysis as use case. Utilizing advanced ML techniques such as Active Learning, Transfer Learning and primitive Large language model: BERT (Bidirectional Encoder Representations from Transformers) as its various components for automating dependency extraction, the tool outcomes demonstrate a mechanism to compute the ROI of ML algorithms to present a clear picture of trade-offs between the cost and benefits of a technology investment.
- [2] arXiv:2407.13900 [pdf, html, other]
-
Title: Exploring the Evidence-Based Beliefs and Behaviors of LLM-Based Programming AssistantsSubjects: Software Engineering (cs.SE)
Recent innovations in artificial intelligence (AI), primarily powered by large language models (LLMs), have transformed how programmers develop and maintain software -- leading to new frontiers in software engineering (SE). The advanced capabilities of LLM-based programming assistants to support software development tasks have led to a rise in the adoption of LLMs in SE. However, little is known about the evidenced-based practices, tools and processes verified by research findings, supported and adopted by AI programming assistants. To this end, our work conducts a preliminary evaluation exploring the beliefs and behaviors of LLM used to support software development tasks. We investigate 17 evidence-based claims posited by empirical SE research across five LLM-based programming assistants. Our findings show that LLM-based programming assistants have ambiguous beliefs regarding research claims, lack credible evidence to support responses, and are incapable of adopting practices demonstrated by empirical SE research to support development tasks. Based on our results, we provide implications for practitioners adopting LLM-based programming assistants in development contexts and shed light on future research directions to enhance the reliability and trustworthiness of LLMs -- aiming to increase awareness and adoption of evidence-based SE research findings in practice.
- [3] arXiv:2407.13902 [pdf, html, other]
-
Title: EvaluateXAI: A Framework to Evaluate the Reliability and Consistency of Rule-based XAI Techniques for Software Analytics TasksComments: This manuscript was accepted in the Journal of Systems and Software (JSS)Subjects: Software Engineering (cs.SE)
The advancement of machine learning (ML) models has led to the development of ML-based approaches to improve numerous software engineering tasks in software maintenance and evolution. Nevertheless, research indicates that despite their potential successes, ML models may not be employed in real-world scenarios because they often remain a black box to practitioners, lacking explainability in their reasoning. Recently, various rule-based model-agnostic Explainable AI (XAI) techniques, such as PyExplainer and LIME, have been employed to explain the predictions of ML models in software analytics tasks. This paper assesses the ability of these techniques (e.g., PyExplainer and LIME) to generate reliable and consistent explanations for ML models across various software analytics tasks, including Just-in-Time (JIT) defect prediction, clone detection, and the classification of useful code review comments. Our manual investigations find inconsistencies and anomalies in the explanations generated by these techniques. Therefore, we design a novel framework: Evaluation of Explainable AI (EvaluateXAI), along with granular-level evaluation metrics, to automatically assess the effectiveness of rule-based XAI techniques in generating reliable and consistent explanations for ML models in software analytics tasks. After conducting in-depth experiments involving seven state-of-the-art ML models trained on five datasets and six evaluation metrics, we find that none of the evaluation metrics reached 100\%, indicating the unreliability of the explanations generated by XAI techniques. Additionally, PyExplainer and LIME failed to provide consistent explanations for 86.11% and 77.78% of the experimental combinations, respectively. Therefore, our experimental findings emphasize the necessity for further research in XAI to produce reliable and consistent explanations for ML models in software analytics tasks.
- [4] arXiv:2407.13915 [pdf, html, other]
-
Title: Microservices-based Software Systems Reengineering: State-of-the-Art and Future DirectionsThakshila Imiya Mohottige (1), Artem Polyvyanyy (1), Rajkumar Buyya (1), Colin Fidge (2), Alistair Barros (2) ((1) University of Melbourne, (2) Queensland University of Technology)Comments: 40 pages, 4 figures, 23 tablesSubjects: Software Engineering (cs.SE); Distributed, Parallel, and Cluster Computing (cs.DC)
Designing software compatible with cloud-based Microservice Architectures (MSAs) is vital due to the performance, scalability, and availability limitations. As the complexity of a system increases, it is subject to deprecation, difficulties in making updates, and risks in introducing defects when making changes. Microservices are small, loosely coupled, highly cohesive units that interact to provide system functionalities. We provide a comprehensive survey of current research into ways of identifying services in systems that can be redeployed as microservices. Static, dynamic, and hybrid approaches have been explored. While code analysis techniques dominate the area, dynamic and hybrid approaches remain open research topics.
- [5] arXiv:2407.14023 [pdf, html, other]
-
Title: Towards Extracting Ethical Concerns-related Software Requirements from App ReviewsSubjects: Software Engineering (cs.SE)
As mobile applications become increasingly integral to our daily lives, concerns about ethics have grown drastically. Users share their experiences, report bugs, and request new features in application reviews, often highlighting safety, privacy, and accountability concerns. Approaches using machine learning techniques have been used in the past to identify these ethical concerns. However, understanding the underlying reasons behind them and extracting requirements that could address these concerns is crucial for safer software solution development. Thus, we propose a novel approach that leverages a knowledge graph (KG) model to extract software requirements from app reviews, capturing contextual data related to ethical concerns. Our framework consists of three main components: developing an ontology with relevant entities and relations, extracting key entities from app reviews, and creating connections between them. This study analyzes app reviews of the Uber mobile application (a popular taxi/ride app) and presents the preliminary results from the proposed solution. Initial results show that KG can effectively capture contextual data related to software ethical concerns, the underlying reasons behind these concerns, and the corresponding potential requirements.
- [6] arXiv:2407.14114 [pdf, other]
-
Title: A3Rank: Augmentation Alignment Analysis for Prioritizing Overconfident Failing Samples for Deep Learning ModelsSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
Sharpening deep learning models by training them with examples close to the decision boundary is a well-known best practice. Nonetheless, these models are still error-prone in producing predictions. In practice, the inference of the deep learning models in many application systems is guarded by a rejector, such as a confidence-based rejector, to filter out samples with insufficient prediction confidence. Such confidence-based rejectors cannot effectively guard against failing samples with high confidence. Existing test case prioritization techniques effectively distinguish confusing samples from confident samples to identify failing samples among the confusing ones, yet prioritizing the failing ones high among many confident ones is challenging. In this paper, we propose $A^3$Rank, a novel test case prioritization technique with augmentation alignment analysis, to address this problem. $A^3$Rank generates augmented versions of each test case and assesses the extent of the prediction result for the test case misaligned with these of the augmented versions and vice versa. Our experiment shows that $A^3$Rank can effectively rank failing samples escaping from the checking of confidence-based rejectors, which significantly outperforms the peer techniques by 163.63\% in the detection ratio of top-ranked samples. We also provide a framework to construct a detector devoted to augmenting these rejectors to defend these failing samples, and our detector can achieve a significantly higher defense success rate.
- [7] arXiv:2407.14118 [pdf, html, other]
-
Title: Beyond Code Generation: Assessing Code LLM Maturity with PostconditionsSubjects: Software Engineering (cs.SE)
Most existing code Large Language Model (LLM) benchmarks, e.g., EvalPlus, focus on the code generation tasks. Namely, they contain a natural language description of a problem and ask the LLM to write code to solve the problem. We argue that they do not capture all capabilities needed to assess the quality of a code LLM. In this paper, we propose a code LLM maturity model, based on the postcondition generation problem, to access a more complete set of code LLM capabilities. We choose the postcondition generation problem as it requires the code LLM to understand the code including semantics, natural language, and also have the capability to generate unambiguous postconditions in programming languages (i.e., the generation capablity). Moreover, postconditions have various types, requiring different levels of these capabilities, making it suitable to evaluate the maturity of the code LLM. Based on our designed maturity model, we augment the EvalPlus dataset to a postcondition testing benchmark, and evaluated several open-sourced models. Our results highlight the necessary improvements needed for better LLMs for code. Code: this https URL
- [8] arXiv:2407.14361 [pdf, html, other]
-
Title: FuzzTheREST: An Intelligent Automated Black-box RESTful API FuzzerComments: 10 pages, 4 figures, published in DCAI 2024 conference, 2 tablesSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Software's pervasive impact and increasing reliance in the era of digital transformation raise concerns about vulnerabilities, emphasizing the need for software security. Fuzzy testing is a dynamic analysis software testing technique that consists of feeding faulty input data to a System Under Test (SUT) and observing its behavior. Specifically regarding black-box RESTful API testing, recent literature has attempted to automate this technique using heuristics to perform the input search and using the HTTP response status codes for classification. However, most approaches do not keep track of code coverage, which is important to validate the solution. This work introduces a black-box RESTful API fuzzy testing tool that employs Reinforcement Learning (RL) for vulnerability detection. The fuzzer operates via the OpenAPI Specification (OAS) file and a scenarios file, which includes information to communicate with the SUT and the sequences of functionalities to test, respectively. To evaluate its effectiveness, the tool was tested on the Petstore API. The tool found a total of six unique vulnerabilities and achieved 55\% code coverage.
- [9] arXiv:2407.14372 [pdf, other]
-
Title: SCoPE: Evaluating LLMs for Software Vulnerability DetectionComments: 10 pages, 3 figures, 1 table, published in DCAI 24 conferenceSubjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Machine Learning (cs.LG)
In recent years, code security has become increasingly important, especially with the rise of interconnected technologies. Detecting vulnerabilities early in the software development process has demonstrated numerous benefits. Consequently, the scientific community started using machine learning for automated detection of source code vulnerabilities. This work explores and refines the CVEFixes dataset, which is commonly used to train models for code-related tasks, specifically the C/C++ subset. To this purpose, the Source Code Processing Engine (SCoPE), a framework composed of strategized techniques that can be used to reduce the size and normalize C/C++ functions is presented. The output generated by SCoPE was used to create a new version of CVEFixes. This refined dataset was then employed in a feature representation analysis to assess the effectiveness of the tool's code processing techniques, consisting of fine-tuning three pre-trained LLMs for software vulnerability detection. The results show that SCoPE successfully helped to identify 905 duplicates within the evaluated subset. The LLM results corroborate with the literature regarding their suitability for software vulnerability detection, with the best model achieving 53% F1-score.
New submissions for Monday, 22 July 2024 (showing 9 of 9 entries )
- [10] arXiv:2407.14402 (cross-list from cs.AI) [pdf, html, other]
-
Title: The Vision of Autonomic Computing: Can LLMs Make It a Reality?Zhiyang Zhang, Fangkai Yang, Xiaoting Qin, Jue Zhang, Qingwei Lin, Gong Cheng, Dongmei Zhang, Saravan Rajmohan, Qi ZhangSubjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC); Multiagent Systems (cs.MA); Software Engineering (cs.SE)
The Vision of Autonomic Computing (ACV), proposed over two decades ago, envisions computing systems that self-manage akin to biological organisms, adapting seamlessly to changing environments. Despite decades of research, achieving ACV remains challenging due to the dynamic and complex nature of modern computing systems. Recent advancements in Large Language Models (LLMs) offer promising solutions to these challenges by leveraging their extensive knowledge, language understanding, and task automation capabilities. This paper explores the feasibility of realizing ACV through an LLM-based multi-agent framework for microservice management. We introduce a five-level taxonomy for autonomous service maintenance and present an online evaluation benchmark based on the Sock Shop microservice demo project to assess our framework's performance. Our findings demonstrate significant progress towards achieving Level 3 autonomy, highlighting the effectiveness of LLMs in detecting and resolving issues within microservice architectures. This study contributes to advancing autonomic computing by pioneering the integration of LLMs into microservice management frameworks, paving the way for more adaptive and self-managing computing systems. The code will be made available at this https URL.
Cross submissions for Monday, 22 July 2024 (showing 1 of 1 entries )
- [11] arXiv:2405.01176 (replaced) [pdf, html, other]
-
Title: SOPA: A Framework for Sustainability-Oriented Process Analysis and Re-design in Business Process ManagementSubjects: Software Engineering (cs.SE)
Given the continuous global degradation of the Earth's ecosystem due to unsustainable human activity, it is increasingly important for enterprises to evaluate the effects they have on the environment. Consequently, assessing the impact of business processes on sustainability is becoming an important consideration in the discipline of Business Process Management (BPM). However, existing practical approaches that aim at a sustainability-oriented analysis of business processes provide only a limited perspective on the environmental impact caused. Further, they provide no clear and practically applicable mechanism for sustainability-driven process analysis and re-design. Following a design science methodology, we here propose and study SOPA, a framework for sustainability-oriented process analysis and re-design. SOPA extends the BPM life cycle by use of Life Cycle Assessment (LCA) for sustainability analysis in combination with Activity-based Costing (ABC). We evaluate SOPA and its usefulness with a case study, by means of an implementation to support the approach, thereby also illustrating the practical applicability of this work.
- [12] arXiv:2406.05940 (replaced) [pdf, html, other]
-
Title: M2CVD: Enhancing Vulnerability Semantic through Multi-Model Collaboration for Code Vulnerability DetectionSubjects: Software Engineering (cs.SE)
Large Language Models (LLMs) have strong capabilities in code comprehension, but fine-tuning costs and semantic alignment issues limit their project-specific optimization; conversely, code models such CodeBERT are easy to fine-tune, but it is often difficult to learn vulnerability semantics from complex code languages. To address these challenges, this paper introduces the Multi-Model Collaborative Vulnerability Detection approach (M2CVD) that leverages the strong capability of analyzing vulnerability semantics from LLMs to improve the detection accuracy of code models. M2CVD employs a novel collaborative process: first enhancing the quality of vulnerability semantic description produced by LLMs through the understanding of project code by code models, and then using these improved vulnerability semantic description to boost the detection accuracy of code models. We demonstrated M2CVD's effectiveness on two real-world datasets, where M2CVD significantly outperformed the baseline. In addition, we demonstrate that the M2CVD collaborative method can extend to other different LLMs and code models to improve their accuracy in vulnerability detection tasks.