Test Case Prioritization For Black Box Testing Essay Example

greatly aided me in gaining a comprehensive understanding of the research area.

In addition, I would also like to thank Associate Professor Ben Soh for his encouragement and assistance in this thesis. His guidance in thesis writing has empowered me to enhance my critical reading and writing skills.

Component-based software engineering (CBSE) is a newly recognized discipline in software engineering that offers the convenience of fast design and implementation.

Component-based software engineering (CBSE) focuses on composing software systems instead of programming software. Unlike object-oriented (OO) techniques, COTS components are black box in nature due to the lack of source code, making testing, verification, and validation more complex. Without source code, many OO testing techniques cannot be used. The main challenges in component-based software development are the difficulty of finding suitable ready-made components and the uncertainty of their quality. Additionally, components have different dependencies and lack source code, while their target context is unknown. In unit testing, the component developer assesses the component's correct functionality and implementation logic.

When the system integrator combines multiple components to create a system, they primarily focus on testing the functionality that each component is expected to provide. The integrator specifically tests the functional aspect of each component. To identify any errors, the system integration tester utilizes integration testing and system testing methods. In component-based systems, the main emphasis of testing is on interfaces, which are the external parts of the components. These interfaces are prone to problems related to component synchronization, coupling complexity, and runtime events that occur during the integration of the components. In previous studies, several techniques were suggested for testing component-based systems, with most of them being either code-based or model-based

approaches.

Testing is a time and resource-consuming process in software development. Today's entrepreneurs face the critical management problem of reducing testing costs without compromising effectiveness. The solution is to prioritize test cases so that the most important ones are run earlier in the testing process. Test case prioritization can have different goals, such as increasing system reliability or performance, or reducing cost per test case. Based on the testing objectives, specific prioritization factors are selected to sort and organize the test cases. Regression testing, although time and cost consuming, is an important aspect of testing.

In the domain of object-oriented (OO) development, various techniques have been suggested for prioritizing test cases during regression testing. These techniques primarily focus on analyzing code and are not easily applicable in component-based development environments. In OO, factors that are directly associated with complexity and severity of faults are utilized as measures, whereas in component-based systems (CBS), such measures are rarely available. Thus, we must rely on indirect measures like the proneness of components to faults or the complexity of integration. As the number of interfaces used for integrating components increases in a component-based system, it becomes evident that the complexity of system integration also rises.

This paper proposes a test case prioritization technique in a component environment, taking into consideration the dependency and fault severity of components. The higher the dependency of a component, the more prone it is to faults. The criticality and fault proneness of components are the factors used in this technique.

The proposed metrics/equation of criticality of component is used to determine the importance of a component for testing purposes. The paper is structured as follows: section 1 and

2 introduce and provide background information on object-oriented techniques and their limitations compared to component-based system development techniques. Section 3 presents a survey of previous techniques for test case prioritization based on prioritization factors. Section 4 describes different complexity metrics for component-based systems and compares them. Section 5 compares various techniques and metrics used to measure fault proneness.

Section 6 introduces a proposed technique for prioritizing test cases in component-based black box testing. This technique is based on a new matrix called Dependency density of Component (DDcomp) and involves two prioritization factors: Criticality of Component and Fault Proneness of Component interactions.

Object-oriented technology and software design principles have become widely used in software development to mimic the behavior of complex real-world systems. The fundamental principles of object-oriented techniques focus on incorporating the states and behaviors of the subject system. This primarily relies on how objects are connected to each other, rather than why. Examples of these relationships include associations, inheritance, polymorphism, and uses, which often overshadow the understanding of why objects are related to each other. In response to this phenomenon, object technology abstracts the continuous nature of real-world systems by incorporating time and space as discrete quantities. Component-based technology, a subset and special case of object-oriented technology, views each logical particle of the software system as a unique component.

A component, whether it is a class, object, module, or subsystem, operates independently within a system. This makes the development of complex software easier as components can be obtained from different vendors and combined to create the system at a reduced cost and time. Component-based systems treat the component's functionality as a black box since the

source code is not available. Therefore, white box testing on commercially off-the-shelf (COTS) components is nearly impossible. Developers must rely on functional black box testing instead.

There are various methods, techniques, and frameworks for testing both object-oriented (OO) and component-based (CB) systems at different stages of software development including requirement capturing, design, and coding. However, when dealing with special cases like embedded real-time systems or component-based systems with unique problem spaces and development requirements, the limitations of object-oriented techniques become apparent. Developers need to be aware of these limitations and work within their constraints.

Real-world systems consist of time and space, which are essential factors forming a continuum. However, the OO technique fails to adequately depict the continuous behavior of objects as time progresses. This restriction is evident in several papers [1], exploring topics like the interaction between electric and magnetic fields, thermal conductivity in thermodynamics, and gyroscopic effects on rotating objects.

The best approach for tackling the problem mentioned above is through component based techniques. These techniques prioritize the completeness of integration and offer some flexibility in defining relative time measures during component interactions. However, this aspect is not the main focus of this paper. When modeling an object-oriented system, both the static and behavioral aspects are emphasized. Research and surveys have shown that development processes are just as important as difficulties encountered during modeling. When testing object-oriented systems, attention is primarily directed towards statements, functions, classes, relations, and implementation logic. In contrast, commercial-off-the-shelf (COTS) components are characterized as black boxes.

Due to the system integrator's lack of knowledge regarding the logic and implementation of the component, white-box testing cannot be conducted on said component. Instead, the integrator

can only assess its functionality through accessing its interface or port. Consequently, black box testing becomes the most effective method for testing component-based systems in this scenario. These systems are constructed by assembling pre-built software components that have been developed independently and are referred to as "plug and play" parts. This approach provides flexibility by separating stable portions of the system.

Components (or the components of their composition) can be described as black-box entities that encapsulate services behind well-defined interfaces. These interfaces are typically highly restricted, reflecting a specific plug compatibility model supported by a component framework. Figure 1 above presents a comparison between traditional development and CB development.

The new model utilizes the capability of real-time software to function as a multitasking application. This allows the driver to have its own thread of control. Instead of following the traditional object-oriented process-driven approach depicted in (Figure 1a), the driver employs a data-driven approach, as illustrated in (Figure 1b), to interact with other parts of the application software. The reusability of software components is where CB methods truly excel, offering various advantages such as enhanced productivity, maintainability, portability, and software system quality.

A reusable component can be compared to a box that includes the code and documentation. These components can be classified as Black Box, Glass-box, or white box. Many COTS components are either black boxes or glass-boxes. In black-box reuse, the user only sees the interface of the component, not its implementation. The interface includes public methods, user documentation, requirements, and restrictions. In glass box reuse, the user can also see inside the box but cannot interact with its contents.

This solution has an advantage over black-box reuse,

as the reuse can better understand the box and its use. The importance of component reusability depends on certain quality factors, including the presence of Meta-Information, the observability rate of the component, the customizability rate of the component, and the self-completeness of the component's return value. Developing CB systems has become increasingly challenging due to growing complexity, shorter development time, and smaller budgets. Therefore, ensuring correctness and reliability are crucial aspects in this engineering field.

State-based testing techniques, such as state coverage and transition coverage, are widely used. Suman has proposed a method to generate state models of components without relying on the source code or other related models. These state models are crucial for conducting state-based regression testing on component-based applications. Traditional testing methods often struggle to identify state-based bugs.

Bringmann proposed a test method using "time partition" to define executable test cases for testing the continuous behavior of automotive embedded systems. This method employs a graphical notation for test cases, which is easily understandable and capable of expressing complex, fully automated tests.

Lalchandani also presented a model-based component regression testing method that utilizes dynamic slicing algorithms to identify bugs in architecture models created during design time. These models are converted into Architecture Component Dependence Graph (ACDG).

Various empirical studies have been conducted to highlight component integration issues and techniques in the component integration testing process. In Rahman's survey, different component integration testing techniques were explored, including built-in testing (BIT), testable architecture, metadata-based testing, certification strategy, and user's specification-based testing.

Based on these findings, it is possible to develop new component-based integration testing techniques that have not yet been invented. Moreover, it is feasible to compare the differences and

similarities among these methods.

The categorization mentioned may have different uses depending on the user. The method chosen for testing the component will vary based on several factors and assumptions. However, there is currently no existing research on the testing process. Common OO-metrics used for calculating system complexity include code base measurements like lines of code, function point analysis, bugs or faults per line of code, code coverage, number of lines of customer requirements, and number of classes/interfaces. These metrics cannot be used to assess or measure the complexity of system or component interactions until the coding stage.

The traditional metrics used to calculate the functional importance of a system's module are not suitable for CBSD. These metrics are designed for non-component based software systems and do not work well with CBSD because of the nature of the COTS-components. COTS-components are black box in nature and their size or implementation details are not known in advance. Most traditional size metrics are based on lines of code, which is not applicable with COTS-components. The lack of source code for some components prevents thorough white box testing. However, in some cases, it is possible to glass-box-test CB-systems by utilizing design or functionality models.

Component metrics are used to measure various aspects of CB-system Complexity. They provide valuable information about the system's external quality aspects, including maintainability, reusability, and reliability. These metrics are employed to assess properties such as quality, complexity, or force-effort of the entity being measured. Component complexity is a combination of intrinsic complexities resulting from methods within the component and extrinsic complexities arising from interactions with other components.