Real Time Software Testing -Chapter-5

                    Test design techniques

5.1 The TEST DEVELOPMENT PROCESS

The process described in this section can be done in different ways, from very informal with little or no documentation, to very formal (as it is described below). The level of formality depends on the context of the testing, including the organization, the maturity of testing and development processes, time constraints and the people involved.

During test analysis, the test basis documentation is analyzed in order to determine what to test, i.e. to identify the test conditions. A test condition is defined as an item or event that could be verified by one or more test cases (e.g. a function, transaction, quality characteristic or structural element).
Establishing traceability from test conditions back to the specifications and requirements enables both impact analysis, when requirements change, and requirements coverage to be determined for a set of tests. During test analysis the detailed test approach is implemented to select the test design techniques to use, based on, among other considerations, the risks identified
During test design the test cases and test data are created and specified. A test case consists of a set of input values, execution preconditions, expected results and execution post-conditions, developed to cover certain test condition(s). The ‘Standard for Software Test Documentation’ (IEEE 829) describes the content of test design specifications (containing test conditions) and test case specifications.

Expected results should be produced as part of the specification of a test case and include outputs, changes to data and states, and any other consequences of the test. If expected results have not been defined then a plausible, but erroneous, result may be interpreted as the correct one. Expected results should ideally be defined prior to test execution.

During test implementation the test cases are developed, implemented, prioritized and organized in the test procedure specification. The test procedure (or manual test script) specifies the sequence of action for the execution of a test. If tests are run using a test execution tool, the sequence of actions is specified in a test script (which is an automated test procedure).


The various test procedures and automated test scripts are subsequently formed into a test execution schedule that defines the order in which the various test procedures, and possibly automated test scripts, are executed, when they are to be carried out and by whom. The test execution schedule will take into account such factors as regression tests, prioritization, and technical and logical dependencies

5.2 Categories of test design techniques

The purpose of a test design technique is to identify test conditions and test cases.


It is a classic distinction to denote test techniques as black box or white box. Black-box techniques (which include specification-based and experienced-based techniques) are a way to derive and select test conditions or test cases based on an analysis of the test basis documentation and the experience of developers, testers and users, whether functional or non-functional, for a component or system without reference to its internal structure. White-box techniques (also called structural or structure-based techniques) are based on an analysis of the structure of the component or system.
Some techniques fall clearly into a single category; others have elements of more than one category.
This syllabus refers to specification-based or experience-based approaches as black-box techniques and structure-based as white-box techniques.
Common features of specification-based techniques:
•
Models, either formal or informal, are used for the specification of the problem to be solved, the software or its components.
•
From these models test cases can be derived systematically.
Common features of structure-based techniques:
•
Information about how the software is constructed is used to derive the test cases, for example, code and design.
•
The extent of coverage of the software can be measured for existing test cases, and further test cases can be derived systematically to increase coverage.
Common features of experience-based techniques:
•
The knowledge and experience of people are used to derive the test cases.
•
Knowledge of testers, developers, users and other stakeholders about the software, its usage and its environment.
•
Knowledge about likely defects and their distribution


5.3 Specification-based or black-box techniques

5.3.1 Equivalence partitioning

Inputs to the software or system are divided into groups that are expected to exhibit similar behavior, so they are likely to be processed in the same way. Equivalence partitions (or classes) can be found for both valid data and invalid data, i.e. values that should be rejected. Partitions can also be identified for outputs, internal values, time-related values (e.g. before or after an event) and for interface parameters (e.g. during integration testing). Tests can be designed to cover partitions. Equivalence partitioning is applicable at all levels of testing.

Equivalence partitioning as a technique can be used to achieve input and output coverage. It can be applied to human input, input via interfaces to a system, or interface parameters in integration testing.

Example:

If you are testing for an input box accepting numbers from 1 to 1000 then, there is no use in writing thousand test cases for all 1000 valid input numbers plus other test cases for invalid data.
Using equivalence partitioning method above test cases can be divided into three sets of input data called classes. Each test case is a representative of respective class.
So, from the above example, we can divide our test cases into three equivalence classes of some valid and invalid inputs.

1. One input data class with all valid inputs. Pick a single value from range 1 to 1000 as a valid test case. If you select other values between 1 and 1000 then result is going to be same. So one test case for valid input data should be sufficient.
2. Another input data class with all values below lower limit (for example any value below 1, as an invalid input data test case).
3. Last input data with any value greater than 1000 to represent third invalid input class.

5.3.2 Boundary value analysis

Behavior at the edge of each equivalence partition is more likely to be incorrect, so boundaries are an area where testing is likely to yield defects. The maximum and minimum values of a partition are its boundary values. A boundary value for a valid partition is a valid boundary value; the boundary of an invalid partition is an invalid boundary value. Tests can be designed to cover both valid and invalid boundary values. When designing test cases, a test for each boundary value is chosen.
Boundary value analysis can be applied at all test levels. It is relatively easy to apply and its defect finding capability is high; detailed specifications are helpful.
This technique is often considered as an extension of equivalence partitioning. It can be used on equivalence classes for user input on screen as well as, for example, on time ranges (e.g. time out, transactional speed requirements) or table ranges (e.g. table size is 256*256). Boundary values may also be used for test data selection

Referring to our example mentioned above we will have the following test cases:
1. Test cases with test data exactly as the input boundaries of input domain (values 1 and 1000 in our case)
2. Test data with values just below the extreme edges of input domains (values 0 and 999 in our case)
3. Test data with values just above the extreme edges of input domain (values 2 and 1001 in our case

5.3.3 Cause Effect Analysis

The main drawback of the previous two techniques is that they do not explore the combination of input conditions.
Cause effect analysis is an approach for studying the specifications carefully and identifying the combinations of input conditions (causes) and their effect in the form of a table and designing test cases
It is suitable for applications in which combinations of input conditions are few and readily visible.

5.3.4 Cause Effect Graphing

This is a rigorous approach, recommended for complex systems only. In such systems the number of inputs and number of equivalent classes for each input could be many and hence the number of input combinations usually is astronomical. Hence we need a systematic approach to select a subset of these input conditions.
Guidelines for graphing:
-
Divide specifications into workable pieces as it may be practically difficult to work on large specifications.
-
Identify the causes and their effects. A cause is an input condition or an equivalence class of input conditions. An effect is an output condition or a system transformation.
-
Link causes and effects in a Boolean graph which is the cause-effect graph.
-
Make decision tables based on the graph. This is done by having one row each for a node in the graph. The number of columns will depend on the number of different combinations of input conditions which can be made.
-

Convert the columns in the decision table into test cases.

Example: 

A program accepts Transaction Code - 3 characters as input. For a valid input the following must be true.
1st character (denoting issue or receipt) + for issue- for receipt
2nd character - a digit
3rd character - a digit

To carry out cause effect graphing, the control flow graph is constructed as below.

In the graph:

(1) or (2) must be true (V in the graph to be interpreted as OR)
(3) and (4) must be true (? in the graph to be interpreted as AND)
The Boolean graph has to be interpreted as follows:
-
node (1) turns true if the 1st character is ‘+’
-
node (2) turns true if the 1st character is ‘-’ (both node (1) and node (2) cannot be true simultaneously)
-
node(3) becomes true if the 2nd character is a digit
-
node(4) becomes true if the 3rd character is a digit the intermediate
-
node (5) turns true if (1) or (2) is true (i.e., if the 1st character is ‘+’ or ‘-‘) the intermediate
-
node (6) turns true if (3) and (4) are true (i.e., if the 2nd and 3rd characters are digits) The final
-
node (7) turns true if (5) and (6) are true. (i.e., if the 1st character is ‘+’ or ‘-‘, 2nd and 3rd characters are digits)
-

The final node will be true for any valid input and false for any invalid input

A partial decision table corresponding to the above graph: Node	Some possible combination of node states
(1)	0	1	1	1	0	1
(2)	0	0	0	0	0	0
(3)	0	0	0	1	1	1
(4)	0	0	1	0	1	1
(5)	0	1	1	0	0	1
(6)	0	0	0	1	1	1
(7)	0	0	0	0	0	1
Sample Test Case for the Column	$xy	+ab	+a4	+2y	@45	+67

The sample test cases can be derived by giving values to the input characters such that the nodes turn true/false as given in the columns of the decision table

5.3.5 Decision table testing

Decision tables are a good way to capture system requirements that contain logical conditions, and to document internal system design. They may be used to record complex business rules that a system is to implement. The specification is analyzed, and conditions and actions of the system are identified. The input conditions and actions are most often stated in such a way that they can either be true or false (Boolean). The decision table contains the triggering conditions, often combinations of true and false for all input conditions, and the resulting actions for each combination of conditions. Each column of the table corresponds to a business rule that defines a unique combination of conditions, which result in the execution of the actions associated with that rule. The coverage standard commonly used with decision table testing is to have at least one test per column, which typically involves covering all combinations of triggering conditions.

The strength of decision table testing is that it creates combinations of conditions that might not otherwise have been exercised during testing. It may be applied to all situations when the action of the software depends on several logical decisions.

5.3.6 State transition testing

A system may exhibit a different response depending on current conditions or previous history (its state). In this case, that aspect of the system can be shown as a state transition diagram. It allows the tester to view the software in terms of its states, transitions between states, the inputs or events that trigger state changes (transitions) and the actions which may result from those transitions. The states of the system or object under test are separate, identifiable and finite in number. A state table shows the relationship between the states and inputs, and can highlight possible transitions that are invalid. Tests can be designed to cover a typical sequence of states, to cover every state, to exercise every transition, to exercise specific sequences of transitions or to test invalid transitions.

State transition testing is much used within the embedded software industry and technical automation in general. However, the technique is also suitable for modeling a business object having specific states or testing screen-dialogue flows

5.3.7 Use case testing

Tests can be specified from use cases or business scenarios. A use case describes interactions between actors, including users and the system, which produce a result of value to a system user. Each use case has preconditions, which need to be met for a use case to work successfully. Each use case terminates with post-conditions, which are the observable results and final state of the system after the use case has been completed. A use case usually has a mainstream (i.e. most likely) scenario, and sometimes alternative branches.

Use cases describe the “process flows” through a system based on its actual likely use, so the test cases derived from use cases are most useful in uncovering defects in the process flows during real-world use of the system. Use cases, often referred to as scenarios, are very useful for designing acceptance tests with customer/user participation. They also help uncover integration defects caused by the interaction and interference of different components, which individual component testing would not see.

5.3.8  Structure-based or white-box techniques

Structure-based testing/white-box testing is based on an identified structure of the software or system, as seen in the following examples:

o

Component level: the structure is that of the code itself, i.e. statements, decisions or branches.

o

Integration level: the structure may be a call tree (a diagram in which modules call other modules).

o

System level: the structure may be a menu structure, business process or web page structure.

In this section, two code-related structural techniques for code coverage, based on statements and decisions, are discussed. For decision testing, a control flow diagram may be used to visualize the alternatives for each decision

5.3.9 Use case testing

Tests can be specified from use cases or business scenarios. A use case describes interactions between actors, including users and the system, which produce a result of value to a system user. Each use case has preconditions, which need to be met for a use case to work successfully. Each use case terminates with post-conditions, which are the observable results and final state of the system after the use case has been completed. A use case usually has a mainstream (i.e. most likely) scenario, and sometimes alternative branches.

Use cases describe the “process flows” through a system based on its actual likely use, so the test cases derived from use cases are most useful in uncovering defects in the process flows during real-world use of the system. Use cases, often referred to as scenarios, are very useful for designing acceptance tests with customer/user participation. They also help uncover integration defects caused by the interaction and interference of different components, which individual component testing would not see.

5.4 Structure-based or white-box techniques

Structure-based testing/white-box testing is based on an identified structure of the software or system, as seen in the following examples:

o

Component level: the structure is that of the code itself, i.e. statements, decisions or branches.

o

Integration level: the structure may be a call tree (a diagram in which modules call other modules).

o

System level: the structure may be a menu structure, business process or web page structure.

In this section, two code-related structural techniques for code coverage, based on statements and decisions, are discussed. For decision testing, a control flow diagram may be used to visualize the alternatives for each decision.

5.4.1 Statement testing and coverage or Basic path testing

In component testing, statement coverage is the assessment of the percentage of executable statements that have been exercised by a test case suite. Statement testing derives test cases to execute specific statements, normally to increase statement coverage.

Basis Path Testing is white box testing method where we design test cases to cover every statement, every branch and every predicate (condition) in the code which has been written. Thus the method attempts statement coverage, decision coverage and condition coverage

To perform Basis Path Testing

•

Derive a logical complexity measure of procedural design

o

Break the module into blocks delimited by statements that affect the control flow (eg.: statement like return, exit, jump etc. and conditions)

o

Mark out these as nodes in a control flow graph

o

Draw connectors (arcs) with arrow heads to mark the flow of logic

o

Identify the number of regions (Cyclomatic Number) which is equivalent to the McCabe’s number

•

Define a basis set of execution paths

o

Determine independent paths

•

Derive test case to exercise (cover) the basis set

McCabe’s Number (Cyclomatic Complexity)

•

Gives a quantitative measure of the logical complexity of the module

•

Defines the number of independent paths

•

Provides an upper bound to the number of tests that must be conducted to ensure that all the statements are executed at least once.

•

Complexity of a flow graph ‘g’, v(g), is computed in one of three ways:

o

V(G) = No. of regions of G

o

V(G) = E - N +2 (E: No. of edges & N: No. of nodes)

o

V(G) = P + 1 (P: No. of predicate nodes in G or No. of conditions in the code)

McCabe’s Number = No. of Regions (Count the mutually exclusive closed regions and also the whole outer space as one region) = 2 (in the above graph)

Two other formulae as given below also define the above measure: McCabe’s Number = E - N +2(= 6 – 6 +2 = 2 for the above graph) McCabe’s Number = P + 1(=1 + 1= 2 for the above graph)

Please note that if the number of conditions is more than one in a single control structure, each condition needs to be separately marked as a node.

When the McCabe’s number is 2, it indicates that there two linearly independent paths in the code. i.e., two different ways in which the graph can be traversed from the 1st node to the last node. The independents in the above graph are:

i)

1-2-3-5-6

ii)

ii) 1-2-4-5-6

The last step is to write test cases corresponding to the listed paths. This would mean giving the input conditions in such a way that the above paths are traced by the control of execution. The test cases for the paths listed here are show in the following table.

Path	Input Condition		Expected Result	Actual Result		Remarks
i)		value of ‘a’ > value of ‘b’			Increment ‘a’ by 1
ii)		value of ‘a’ <= value of ‘b’			Increment ‘b’ by 1

5.4.2 Decision testing and coverage

Decision coverage, related to branch testing, is the assessment of the percentage of decision outcomes (e.g. the True and False options of an IF statement) that have been exercised by a test case suite. Decision testing derives test cases to execute specific decision outcomes, normally to increase decision coverage.

Decision testing is a form of control flow testing as it generates a specific flow of control through the decision points. Decision coverage is stronger than statement coverage: 100% decision coverage guarantees 100% statement coverage, but not vice versa.

5.4.3 Other structure-based techniques

There are stronger levels of structural coverage beyond decision coverage, for example, condition coverage and multiple condition coverage.

The concept of coverage can also be applied at other test levels (e.g. at integration level) where the percentage of modules, components or classes that have been exercised by a test case suite could be expressed as module, component or class coverage.

Tool support is useful for the structural testing of code.

5.5 Experience-based techniques

Experienced-based testing is where tests are derived from the tester’s skill and intuition and their experience with similar applications and technologies. When used to augment systematic techniques, these techniques can be useful in identifying special tests not easily captured by formal techniques, especially when applied after more formal approaches. However, this technique may yield widely varying degrees of effectiveness, depending on the testers’ experience. A commonly used experienced-based technique is error guessing. Generally testers anticipate defects based on experience. A structured approach to the error guessing technique is to enumerate a list of possible errors and to design tests that attack these errors. This systematic approach is called fault attack. These defect and failure lists can be built based on experience, available defect and failure data, and from common knowledge about why software fails.

Exploratory testing is concurrent test design, test execution, test logging and learning, based on a test charter containing test objectives, and carried out within time-boxes. It is an approach that is most useful where there are few or inadequate specifications and severe time pressure, or in order to augment or complement other, more formal testing. It can serve as a check on the test process, to help ensure that the most serious defects are found.

Error Guessing

Error guessing is a supplementary technique where test case design is based on the tester's intuition and experience. There is no formal procedure. However, a checklist of common errors could be helpful here.

5.6 Choosing test techniques

The choice of which test techniques to use depends on a number of factors, including the type of system, regulatory standards, customer or contractual requirements, level of risk, type of risk, test objective, documentation available, knowledge of the testers, time and budget, development life cycle, use case models and previous experience of types of defects found.

Some techniques are more applicable to certain situations and test levels; others are applicable to all test levels.

Real Time Software Testing -Chapter-4

                          Static Testing

Why Static Testing necessary?

The benefit is clear once you think about it. If you can find a problem in the requirements before it turns into a problem in the system that will save time and money. The following statistics would be mind boggling.

When to start the Static Testing?


To get value from static testing, we have to start at the right time. For example, reviewing the requirements after the programmers have finished coding the entire system may help testers design test cases. However, the significant return on the static testing investment is no longer available, as testers can't prevent bugs in code that's already written. For optimal returns, a static testing should happen as soon as possible after the item to be tested has been created, while the assumptions and inspirations remain fresh in the creator's mind and none of the errors in the item have caused negative consequences in downstream processes. Effective reviews involve the right people. Business domain experts must attend requirements reviews, system architects must attend design reviews, and expert programmers must attend code reviews. As testers, we can also be valuable participants, because we're good at spotting inconsistencies, vagueness, missing details, and the like. However, testers who attend review meetings do need to bring sufficient knowledge of the business domain, system architecture, and programming to each review. And everyone who attends a review, walkthrough or inspection should understand the basic ground rules of such events.

Below is the diagram of static testing

4.1 Static techniques and the test process

Unlike dynamic testing, which requires the execution of software; static testing techniques rely on the manual examination (reviews) and automated analysis (static analysis) of the code or other project documentation.
Reviews are a way of testing software work products (including code) and can be performed well before dynamic test execution. Defects detected during reviews early in the life cycle are often much cheaper to remove than those detected while running tests (e.g. defects found in requirements).
A review could be done entirely as a manual activity, but there is also tool support. The main manual activity is to examine a work product and make comments about it. Any software work product can be reviewed, including requirements specifications, design specifications, code, test plans, test specifications, test cases, test scripts, user guides or web pages.
Benefits of reviews include early defect detection and correction, development productivity improvements, reduced development timescales, reduced testing cost and time, lifetime cost reductions, fewer defects and improved communication. Reviews can find omissions, for example, in requirements, which are unlikely to be found in dynamic testing.
Reviews, static analysis and dynamic testing have the same objective – identifying defects. They are complementary: the different techniques can find different types of defects effectively and efficiently. Compared to dynamic testing, static techniques find causes of failures (defects) rather than the failures themselves.

Typical defects that are easier to find in reviews than in dynamic testing are: deviations from standards, requirement defects, design defects, insufficient maintainability and incorrect interface specifications.


IEEE classifies Static testing under three broad categories:

• Reviews
• Walkthroughs
• Inspections

Reviews

What is “Reviews?”

A meeting at which the software element is presented to project personnel, managers, users, customers or other interested parties for comment or approval. The software element can be Project Plans, URS, SRS, Design Documents, code, Test Plans, User Manual.

What are objectives of “Reviews”?

To ensure that:

•

The software element conforms to its specifications.

•

The development of the software element is being done as per plans, standards, and guidelines applicable for the project.

•

Changes to the software element are properly implemented and affect only those system areas identified by the change specification

Reviews - Input

•

A statement of objectives for the technical reviews

•

The software element being examined

•

Software project management plan

•

Current anomalies or issues list for the software product

•

Documented review procedures

•

Earlier review report - when applicable

•

Review team members should receive the review materials in advance and they come prepared for the meeting

•

Check list for defects

Reviews – Meeting

•

Examine the software element for adherence to specifications and standards

•

Changes to software element are properly implemented and affect only the specified areas

•

Record all deviations

•

Assign responsibility for getting the issues resolved

•

Review sessions are not expected to find solutions to the deviations.

•

The areas of major concerns, status on previous feedback and review days utilized are also recorded.

•

The review leader shall verify, later, that the action items assigned in the meeting are closed

Reviews - Outputs

•

List of review findings

•

List of resolved and unresolved issues found during the later re-verification

4.2 Review process

The different types of reviews vary from very informal (e.g. no written instructions for reviewers) to very formal (i.e. well structured and regulated). The formality of a review process is related to factors such as the maturity of the development process, any legal or regulatory requirements or the need for an audit trail.

The way a review is carried out depends on the agreed objective of the review.

Phases of a formal review

A typical formal review has the following main phases:

1. Planning: selecting the personnel, allocating roles; defining the entry and exit criteria for more formal review types (e.g. inspection); and selecting which parts of documents to look at.

2. Kick-off: distributing documents; explaining the objectives, process and documents to the participants; and checking entry criteria (for more formal review types).

3. Individual preparation: work done by each of the participants on their own before the review meeting, noting potential defects, questions and comments.

4. Review meeting: discussion or logging, with documented results or minutes (for more formal review types). The meeting participants may simply note defects, make recommendations for handling the defects, or make decisions about the defects.

5. Rework: fixing defects found, typically done by the author.

6. Follow-up: checking that defects have been addressed, gathering metrics and checking on exit criteria (for more formal review types).

Roles and responsibilities

A typical formal review will include the roles below:

•

Manager: decides on the execution of reviews, allocates time in project schedules and determines if the review objectives have been met.

•

Moderator: the person who leads the review of the document or set of documents, including planning the review, running the meeting, and follow-up after the meeting. If necessary, the moderator may mediate between the various points of view and is often the person upon whom the success of the review rests.

•

Author: the writer or person with chief responsibility for the document(s) to be reviewed.

•

Reviewers: individuals with a specific technical or business background (also called checkers or inspectors) who, after the necessary preparation, identify and describe findings (e.g. defects) in the product under review. Reviewers should be chosen to represent different perspectives and roles in the review process, and should take part in any review meetings.

•

Scribe (or recorder): documents all the issues, problems and open points that were identified during the meeting.

Looking at documents from different perspectives and using checklists can make reviews more effective and efficient, for example, a checklist based on perspectives such as user, maintainer, tester or operations, or a checklist of typical requirements problems.

4.2.1 Types of review

A single document may be the subject of more than one review. If more than one type of review is used, the order may vary. For example, an informal review may be carried out before a technical review, or an inspection may be carried out on a requirements specification before a walkthrough with customers. The main characteristics, options and purposes of common review types are:

Informal review

Key characteristics:

•

no formal process;

•

there may be pair programming or a technical lead reviewing designs and code;

•

optionally may be documented;

•

may vary in usefulness depending on the reviewer;

•

main purpose: inexpensive way to get some benefit

Real Time Software Testing -Chapter-3

Testing throughout the software life cycle

3.1 Software development models

Testing does not exist in isolation; test activities are related to software development activities.

Different development life cycle models need different approaches to testing.

3.2 V-model (sequential development model)

Although variants of the V-model exist, a common type of V-model uses four test levels, corresponding to the four development levels.

The four levels used in this syllabus are:

                Component (unit) testing;

                Integration testing;

                System testing;

                Acceptance testing.

 The aim is to work together rather than be confrontational. Keep the focus on delivering a quality product.

• Results should be presented in a non-personal way. The work-product may be wrong, so say this in a non-personal way.

• Attempt to understand how others feel; it is possible to discuss problems and still leave all parties feeling positive.

In practice, a V-model may have more, fewer or different levels of development and testing, depending on the project and the software product. For example, there may be component integration testing after component testing, and system integration testing after system testing.

Software work products (such as business scenarios or use cases, requirements specifications, design documents and code) produced during development is often the basis of testing in one or more test levels. References for generic work products include Capability Maturity Model Integration (CMMI) or ‘Software life cycle processes’ (IEEE/IEC 12207). Verification and validation (and early test design) can be carried out during the development of the software work products

3.3 Iterative-incremental development models

Iterative-incremental development is the process of establishing requirements, designing, building and testing a system, done as a series of shorter development cycles. Examples are: prototyping, rapid application development (RAD), Rational Unified Process (RUP) and agile development models. The resulting system produced by iteration may be tested at several levels as part of its development. An increment, added to others developed previously, forms a growing partial system, which should also be tested. Regression testing is increasingly important on all iterations after the first one. Verification and validation can be carried out on each increment.

3.4 Testing within a life cycle model

In any life cycle model, there are several characteristics of good testing:

                For every development activity there is a corresponding testing activity.

                Each test level has test objectives specific to that level.

                The analysis and design of tests for a given test level should begin during the corresponding development activity.

                Testers should be involved in reviewing documents as soon as drafts are available in the development life cycle.

Test levels can be combined or reorganized depending on the nature of the project or the system architecture. For example, for the integration of a commercial off-the-shelf (COTS) software product into a system, the purchaser may perform integration testing at the system level (e.g. integration to the infrastructure and other systems, or system deployment) and acceptance testing (functional and/or non-functional, and user and/or operational testing).

3.4 Test levels

For each of the test levels, the following can be identified: their generic objectives, the work product(s) being referenced for deriving test cases (i.e. the test basis), and the test object (i.e. what is being tested), typical defects and failures to be found, test harness requirements and tool support, and specific approaches and responsibilities.

3.5 Component testing

Component testing searches for defects in, and verifies the functioning of, software (e.g. modules, programs, objects, classes, etc.) that are separately testable. It may be done in isolation from the rest of the system, depending on the context of the development life cycle and the system. Stubs, drivers and simulators may be used.

Component testing may include testing of functionality and specific non-functional characteristics, such as resource-behavior (e.g. memory leaks) or robustness testing, as well as structural testing (e.g. branch coverage). Test cases are derived from work products such as a specification of the component, the software design or the data mode

Typically, component testing occurs with access to the code being tested and with the support of the development environment, such as a unit test framework or debugging tool, and, in practice, usually involves the programmer who wrote the code. Defects are typically fixed as soon as they are found, without formally recording incidents.

One approach to component testing is to prepare and automate test cases before coding. This is called a test-first approach or test-driven development. This approach is highly iterative and is based on cycles of developing test cases, then building and integrating small pieces of code, and executing the component tests until they pass.

3.6 Integration testing

Integration testing tests interfaces between components, interactions with different parts of a system, such as the operating system, file system, hardware, or interfaces between systems.

There may be more than one level of integration testing and it may be carried out on test objects of varying size. For example:

1. Component integration testing tests the interactions between software components and is done after component testing;

2. System integration testing tests the interactions between different systems and may be done after system testing. In this case, the developing organization may control only one side of the interface, so changes may be destabilizing. Business processes implemented as workflows may involve a series of systems. Cross-platform issues may be significant.

The greater the scope of integration, the more difficult it becomes to isolate failures to a specific component or system, which may lead to increased risk.

Systematic integration strategies may be based on the system architecture (such as top-down and bottom-up), functional tasks, transaction processing sequences, or some other aspect of the system or component. In order to reduce the risk of late defect discovery, integration should normally be incremental rather than “big bang”.

Testing of specific non-functional characteristics (e.g. performance) may be included in integration testing.

At each stage of integration, testers concentrate solely on the integration itself. For example, if they are integrating module A with module B they are interested in testing the communication between the modules, not the functionality of either module. Both functional and structural approaches may be used.

Ideally, testers should understand the architecture and influence integration planning. If integration tests are planned before components or systems are built, they can be built in the order required for most efficient testing

3.7 System testing

System testing is concerned with the behavior of a whole system/product as defined by the scope of a development project or programme.

In system testing, the test environment should correspond to the final target or production environment as much as possible in order to minimize the risk of environment-specific failures not being found in testing.

System testing may include tests based on risks and/or on requirements specifications, business processes, use cases, or other high level descriptions of system behavior, interactions with the operating system, and system resources.

System testing should investigate both functional and non-functional requirements of the system. Requirements may exist as text and/or models. Testers also need to deal with incomplete or undocumented requirements. System testing of functional requirements starts by using the most appropriate specification-based (black-box) techniques for the aspect of the system to be tested. For example, a decision table may be created for combinations of effects described in business rules. Structure-based techniques (white-box) may then be used to assess the thoroughness of the testing with respect to a structural element, such as menu structure or web page navigation.

An independent test team often carries out system testing.

3.8 Acceptance testing

Acceptance testing is often the responsibility of the customers or users of a system; other stakeholders may be involved as well.

The goal in acceptance testing is to establish confidence in the system, parts of the system or specific non-functional characteristics of the system. Finding defects is not the main focus in acceptance testing. Acceptance testing may assess the system’s readiness for deployment and use, although it is not necessarily the final level of testing. For example, a large-scale system integration test may come after the acceptance test for a system.

Acceptance testing may occur as more than just a single test level, for example:

                A COTS software product may be acceptance tested when it is installed or integrated.

                Acceptance testing of the usability of a component may be done during component testing.

Acceptance testing of a new functional enhancement may come before system testing.

Typical forms of acceptance testing include the following:

->User acceptance testing

Typically verifies the fitness for use of the system by business users

->Operational (acceptance) testing

The acceptance of the system by the system administrators, including:

                testing of backup/restore;

                disaster recovery;

                user management;

                maintenance tasks;

                Periodic checks of security vulnerabilities.

->Contract and regulation acceptance testing

Contract acceptance testing is performed against a contract’s acceptance criteria for producing custom-developed software. Acceptance criteria should be defined when the contract is agreed. Regulation acceptance testing is performed against any regulations that must be adhered to, such as governmental, legal or safety regulations.

->Alpha and beta (or field) testing

Developers of market, or COTS, software often want to get feedback from potential or existing customers in their market before the software product is put up for sale commercially. Alpha testing is performed at the developing organization’s site. Beta testing, or field testing, is performed by people at their own locations. Both are performed by potential customers, not the developers of the product.

Organizations may use other terms as well, such as factory acceptance testing and site acceptance testing for systems that are tested before and after being moved to a customer’s site

4. Levels of Testing in detail

Overview

In developing a large system, testing usually involves several stages (Refer the following figure.

                Unit Testing

                Integration Testing

                System Testing

               Acceptance Testing 

Initially, each program component (module) is tested independently verifying the component functions with the types of input identified by studying component’s design. Such a testing is called Unit Testing (or component or module testing). Unit testing is done in a controlled environment with a predetermined set of data fed into the component to observe what output actions and data are produced.

When collections of components have been unit-tested, the next step is ensuring that the interfaces among the components are defined and handled properly. This process of verifying the synergy of system components against the program Design Specification is called Integration Testing.

Once the system is integrated, the overall functionality is tested against the Software Requirements Specification (SRS). Then, the other non-functional requirements like performance testing are done to ensure readiness of the system to work successfully in a customer’s actual working environment. This step is called System Testing.

The next step is customer’s validation of the system against User Requirements Specification (URS). Customer in their working environment does this exercise of Acceptance Testing usually with assistance from the developers. Once the system is accepted, it will be installed and will be put to use.

Unit Testing

Unit Testing is done mostly by developers it advocates to address the goal of finding faults in modules (components):

Examining the code :Typically the static testing methods like Reviews, Walkthroughs and Inspections are used

Proving code correct :After coding and review exercise if we want to ascertain the correctness of the code we can use formal methods. A program is correct if it implements the functions and data properly as indicated in the design, and if it interfaces properly with all other components. One way to investigate program correctness is to view the code as a statement of logical flow. Using mathematical logic, if we can formulate the program as a set of assertions and theorems, we can show that the truth of the theorems implies the correctness of the code.

                 

• Use of this approach forces us to be more rigorous and precise in specification. Much work is involved in setting up and carrying out the proof. For example, the code for performing bubble sort is much smaller than its logical description and proof.

Testing program components (modules)

                In the absence of simpler methods and automated tools, “Proving code correctness” will be an elusive goal for software engineers. Proving views programs in terms of classes of data and conditions and the proof may not involve execution of the code. On the contrary, testing is a series of experiments to observe the behavior of the program for various input conditions. While proof tells us how a program will work in a hypothetical environment described by the design and requirements, testing gives us information about how a program works in its actual operating environment.

To test a component (module), input data and conditions are chosen to demonstrate an observable behavior of the code. A test case is a particular choice of input data to be used in testing a program. Test case are generated by using either black-box or white-box approaches

Integration Testing

Integration is the process of assembling unit-tested modules. We need to test the following aspects that are not previously addressed while independently testing the modules:

                Interfaces: To ensure “interface integrity,” the transfer of data between modules is tested. When data is passed to another module, by way of a call, there should not be any loss or corruption of data. The loss or corruption of data can happen due to mismatch or differences in the number or order of calling and receiving parameters.

                Module combinations may produce a different behavior due to combinations of data that are not exercised during unit testing.

                Global data structures, if used, may reveal errors due to unintended usage in some module.

Integration Strategies

Depending on design approach, one of the following integration strategies can be adopted:

· Big Bang approach

· Incremental approach

                Top-down testing

                Bottom-up testing

                Sandwich testing 

Big Bang approach consists of testing each module individually and linking all these modules together only when every module in the system has been tested

Though Big Bang approach seems to be advantageous when we construct independent module concurrently, this approach is quite challenging and risky as we integrate all modules in a single step and test the resulting system. Locating interface errors, if any, becomes difficult here.

The alternative strategy is an incremental approach, wherein modules of a system are consolidated with already tested components of the system. In this way, the software is gradually built up, spreading the integration testing load more evenly through the construction phase. Incremental approach can be implemented in two distinct ways: Top-down and Bottom-up.

In Top-down testing, testing begins with the topmost module. A module will be integrated into the system only when the module which calls it has been already integrated successfully. An example order of Top-down testing for the above illustration will be:

The testing starts with M1. To test M1 in isolation, communications to modules M2, M3 and M4 have to be somehow simulated by the tester somehow, as these modules may not be ready yet. To simulate responses of M2, M3 and M4 whenever they are to be invoked from M1, “stubs” are created. Simple applications may require stubs which simply return control to their superior modules. More complex situation demand stubs to simulate a full range of responses, including parameter passing. Stubs may be individually created by the tester (as programs in their own right) or they may be provided by a software testing harness, which is a piece of software specifically designed to provide a testing environment.

In the above illustration, M1 would require stubs to simulate the activities of M2, M3 and M4. The integration of M3 would require a stub or stubs (?!) for M5 and M4 would require stubs for M6 and M7. Elementary modules (those which call no subordinates) require no stubs

Bottom-up testing begins with elementary modules. If M5 is ready, we need to simulate the activities of its superior, M3. Such a “driver” for M5 would simulate the invocation activities of M3. As with the stub, the complexity of a driver would depend upon the application under test. The driver would be responsible for invoking the module under test; it could be responsible for passing test data (as parameters) and it might be responsible for receiving output data. Again, the driving function can be provided through a testing harness or may be created by the tester as a program. The following diagram shows the bottom-up testing approach for the above illustration

Bottom up Approach

driver must be provided for modules M2, M5, M6, M7, M3 and M4. There is no need for a driver for the topmost node, M1.

System Testing

The objective of unit and integration testing was to ensure that the code implemented the design properly. In system testing, we need to ensure that the system does what the customer wants it to do. Initially the functions (functional requirements) performed by the system are tested. A function test checks whether the integrated system performs its functions as specified in the requirements.

After ensuring that the system performs the intended functions, the performance test is done. This non-functional requirement includes security, accuracy, speed, and reliability.

System testing begins with function testing. Since the focus here is on functionality, a black-box approach is taken (Refer Test Techniques). Function testing is performed in a controlled situation. Since function testing compares the system’s actual performance with its requirements, test cases are developed from requirements document (SRS). For example a word processing system can be tested by examining the following functions: document creation, document modification and document deletion. To test document modification, adding a character, adding a word, adding a paragraph, deleting a character, deleting a word, deleting a formatting, etc. are to be tested.

Performance testing addresses the non-functional requirements. System performance is measured against the performance objectives set by the customer. For example, function testing may have demonstrated how the system handles deposit or withdraw transactions in a bank account package. Performance testing evaluates the speed with which calculations are made, the precision of the computation, the security precautions required, and the response time to user inquiry.

Acceptance Testing

Acceptance testing is the customer (and user) evaluation of the system, primarily to determine whether the system meets their needs and expectations. Usually acceptance test is done by customer with assistance from developers. Customers can evaluate the system either by conducting a benchmark test or by a pilot test. In benchmark test, the system performance is evaluated against test cases that represent typical conditions under which the system will operate when actually installed. A pilot test installs the system on an experimental basis, and the system is evaluated against everyday working.

Sometimes the system is piloted in-house before the customer runs the real pilot test. The in-house test, in such case, is called an alpha test, and the customer’s pilot is a beta test. This approach is common in the case of commercial software where the system has to be released to a wide variety of customers.

A third approach, parallel testing, is done when a new system is replacing an existing one or is part of a phased development. The new system is put to use in parallel with previous version and will facilitate gradual transition of users, and to compare and contrast the new system with the old.

Test types

A group of test activities can be aimed at verifying the software system (or a part of a system) based on a specific reason or target for testing.

A test type is focused on a particular test objective, which could be the testing of a function to be performed by the software; a non-functional quality characteristic, such as reliability or usability, the structure or architecture of the software or system; or related to changes, i.e. confirming that defects have been fixed (confirmation testing) and looking for unintended changes (regression testing).

A model of the software may be developed and/or used in structural and functional testing, for example, in functional testing a process flow model, a state transition model or a plain language specification; and for structural testing a control flow model or menu structure model. 

4.1 Testing of function (functional testing)

The functions that a system, subsystem or component are to perform may be described in work products such as a requirements specification, use cases, or a

functional specification, or they may be undocumented. The functions are “what” the system does.

Functional tests are based on functions and features (described in documents or understood by the testers) and their interoperability with specific systems, and may be performed at all test levels (e.g. tests for components may be based on a component specification).

Specification-based techniques may be used to derive test conditions and test cases from the functionality of the software or system. Functional testing considers the external behavior of the software (black-box testing).

A type of functional testing, security testing, investigates the functions (e.g. a firewall) relating to detection of threats, such as viruses, from malicious outsiders. Another type of functional testing, interoperability testing, evaluates the capability of the software product to interact with one or more specified components or systems.

4.2 Testing of non-functional software characteristics (non-functional testing)

Non-functional testing includes, but is not limited to, performance testing, load testing, stress testing, usability testing, maintainability testing, reliability testing and portability testing. It is the testing of “how” the system works.

Non-functional testing may be performed at all test levels. The term non-functional testing describes the tests required to measure characteristics of systems and software that can be quantified on a varying scale, such as response times for performance testing.

Types of Performance Tests (Non-Functional Testing)

1. Stress tests – evaluates the system when stressed to its limits. If the requirements state that a system is to handle up to a specified number of devices or users, a stress test evaluates system performance when all those devices or users are active simultaneously. This test brings out the performance during peak demand.

2. Volume tests – addresses the handling of large amounts of data in the system. This includes

                Checking of whether data structures have been defined large enough to handle all possible situations,

                Checking the size of fields, records and files to see whether they can accommodate all expected data, and

                Checking of system’s reaction when data sets reach their maximum size.

3. Configuration tests – analyzes the various software and hardware configurations specified in the requirements. (E.g. system to serve variety of audiences)

4. Compatibility tests – are needed when a system interfaces with other systems (e.g. system to retrieve information from a large database system)

5. Regression tests – are required when the system being tested is replacing an existing system (Always used during a phased development – to ensure that new system’s performance is at least as good as that of the old)

6· Security tests – ensure the security requirements (testing characteristics related to availability, integrity, and confidentiality of data and services)

7· Timing tests – include response time, transaction time, etc. Usually done with stress test to see if the timing requirements are met even when the system is extremely active.

8· Environmental tests – look at the system’s ability to perform at the installation site. If the requirements include tolerances to heat, humidity, motion, chemical presence, moisture, portability, electrical or magnetic fields, disruption of power, or any other environmental characteristics of the site, then our tests should ensure that the system performs under these conditions.

9· Quality tests – evaluate the system’s reliability, maintainability, and availability. These tests include calculation of mean time to failure and mean time to repair, as well as average time to find and fix a fault.

10· Recovery tests – addresses response to the loss of data, power, device or services. The system is subjected to loss of system resources and tested if it recovers properly.

11· Maintenance tests – addresses the need for diagnostic tools and procedures to help in finding the source of problems. To verify existence and functioning of aids like diagnostic program, memory map, traces of transactions, etc.

12· Documentation tests – ensures documents like user guides, maintenance guides and technical documentation exists and to verify consistency of information in them.

13· Human factor (or Usability) tests – investigates user interface related requirements. Display screens, messages, report formats and other aspects are examined for ease of use.

4.3: Testing of software structure/architecture (structural testing)

Structural (white-box) testing may be performed at all test levels. Structural techniques are best used after specification-based techniques, in order to help measure the thoroughness of testing through assessment of coverage of a type of structure.

Coverage is the extent that a structure has been exercised by a test suite, expressed as a percentage of the items being covered. If coverage is not 100%, then more tests may be designed to test those items that were missed and, therefore, increase coverage. Coverage techniques are covered in Chapter 4.

At all test levels, but especially in component testing and component integration testing, tools can be used to measure the code coverage of elements such as statements or decisions. Structural testing may be based on the architecture of the system, such as a calling hierarchy

4.4: Testing related to changes (confirmation testing (retesting) and regression testing)

After a defect is detected and fixed, the software should be retested to confirm that the original defect has been successfully removed. This is called confirmation. Debugging (defect fixing) is a development activity, not a testing activity.

Regression testing is the repeated testing of an already tested program, after modification, to discover any defects introduced or uncovered as a result of the change(s). These defects may be either in the software being tested, or in another related or unrelated software component. It is performed when the software, or its environment, is changed. The extent of regression testing is based on the risk of not finding defects in software that was working previously.

Tests should be repeatable if they are to be used for confirmation testing and to assist regression testing.

Regression testing may be performed at all test levels, and applies to functional, non-functional and structural testing. Regression test suites are run many times and generally evolve slowly, so regression testing is a strong candidate for automation

4.5 : Maintenance testing

Once deployed, a software system is often in service for years or decades. During this time the system and its environment are often corrected, changed or extended. Maintenance testing is done on an existing operational system, and is triggered by modifications, migration, or retirement of the software or system.

Modifications include planned enhancement changes (e.g. release-based), corrective and emergency changes, and changes of environment, such as planned operating system or database upgrades, or patches to newly exposed or discovered vulnerabilities of the operating system.

Maintenance testing for migration (e.g. from one platform to another) should include operational tests of the new environment, as well as of the changed software.

Maintenance testing for the retirement of a system may include the testing of data migration or archiving if long data-retention periods are required