Static Testing
Why Static Testing necessary?
The benefit is clear once you think about it. If you can find a problem in the requirements before it turns into a problem in the system that will save time and money. The following statistics would be mind boggling.
When to start the Static Testing?
To get value from static testing, we have to start at the right time. For example, reviewing the requirements after the programmers have finished coding the entire system may help testers design test cases. However, the significant return on the static testing investment is no longer available, as testers can't prevent bugs in code that's already written. For optimal returns, a static testing should happen as soon as possible after the item to be tested has been created, while the assumptions and inspirations remain fresh in the creator's mind and none of the errors in the item have caused negative consequences in downstream processes. Effective reviews involve the right people. Business domain experts must attend requirements reviews, system architects must attend design reviews, and expert programmers must attend code reviews. As testers, we can also be valuable participants, because we're good at spotting inconsistencies, vagueness, missing details, and the like. However, testers who attend review meetings do need to bring sufficient knowledge of the business domain, system architecture, and programming to each review. And everyone who attends a review, walkthrough or inspection should understand the basic ground rules of such events.
Below is the diagram of static testing
4.1 Static techniques and the test process
Unlike dynamic testing, which requires the execution of software; static testing techniques rely on the manual examination (reviews) and automated analysis (static analysis) of the code or other project documentation.
Reviews are a way of testing software work products (including code) and can be performed well before dynamic test execution. Defects detected during reviews early in the life cycle are often much cheaper to remove than those detected while running tests (e.g. defects found in requirements).
A review could be done entirely as a manual activity, but there is also tool support. The main manual activity is to examine a work product and make comments about it. Any software work product can be reviewed, including requirements specifications, design specifications, code, test plans, test specifications, test cases, test scripts, user guides or web pages.
Benefits of reviews include early defect detection and correction, development productivity improvements, reduced development timescales, reduced testing cost and time, lifetime cost reductions, fewer defects and improved communication. Reviews can find omissions, for example, in requirements, which are unlikely to be found in dynamic testing.
Reviews, static analysis and dynamic testing have the same objective – identifying defects. They are complementary: the different techniques can find different types of defects effectively and efficiently. Compared to dynamic testing, static techniques find causes of failures (defects) rather than the failures themselves.
Typical defects that are easier to find in reviews than in dynamic testing are: deviations from standards, requirement defects, design defects, insufficient maintainability and incorrect interface specifications.
IEEE classifies Static testing under three broad categories:
Why Static Testing necessary?
The benefit is clear once you think about it. If you can find a problem in the requirements before it turns into a problem in the system that will save time and money. The following statistics would be mind boggling.
When to start the Static Testing?
To get value from static testing, we have to start at the right time. For example, reviewing the requirements after the programmers have finished coding the entire system may help testers design test cases. However, the significant return on the static testing investment is no longer available, as testers can't prevent bugs in code that's already written. For optimal returns, a static testing should happen as soon as possible after the item to be tested has been created, while the assumptions and inspirations remain fresh in the creator's mind and none of the errors in the item have caused negative consequences in downstream processes. Effective reviews involve the right people. Business domain experts must attend requirements reviews, system architects must attend design reviews, and expert programmers must attend code reviews. As testers, we can also be valuable participants, because we're good at spotting inconsistencies, vagueness, missing details, and the like. However, testers who attend review meetings do need to bring sufficient knowledge of the business domain, system architecture, and programming to each review. And everyone who attends a review, walkthrough or inspection should understand the basic ground rules of such events.
Below is the diagram of static testing
4.1 Static techniques and the test process
Unlike dynamic testing, which requires the execution of software; static testing techniques rely on the manual examination (reviews) and automated analysis (static analysis) of the code or other project documentation.
Reviews are a way of testing software work products (including code) and can be performed well before dynamic test execution. Defects detected during reviews early in the life cycle are often much cheaper to remove than those detected while running tests (e.g. defects found in requirements).
A review could be done entirely as a manual activity, but there is also tool support. The main manual activity is to examine a work product and make comments about it. Any software work product can be reviewed, including requirements specifications, design specifications, code, test plans, test specifications, test cases, test scripts, user guides or web pages.
Benefits of reviews include early defect detection and correction, development productivity improvements, reduced development timescales, reduced testing cost and time, lifetime cost reductions, fewer defects and improved communication. Reviews can find omissions, for example, in requirements, which are unlikely to be found in dynamic testing.
Reviews, static analysis and dynamic testing have the same objective – identifying defects. They are complementary: the different techniques can find different types of defects effectively and efficiently. Compared to dynamic testing, static techniques find causes of failures (defects) rather than the failures themselves.
Typical defects that are easier to find in reviews than in dynamic testing are: deviations from standards, requirement defects, design defects, insufficient maintainability and incorrect interface specifications.
IEEE classifies Static testing under three broad categories:
- Reviews
- Walkthroughs
- Inspections
Reviews
What is “Reviews?”
A meeting at which the software element is presented to project personnel, managers, users, customers or other interested parties for comment or approval. The software element can be Project Plans, URS, SRS, Design Documents, code, Test Plans, User Manual.
What are objectives of “Reviews”?
To ensure that:
•
The software element conforms to its specifications.
•
The development of the software element is being done as per plans, standards, and guidelines applicable for the project.
•
Changes to the software element are properly implemented and affect only those system areas identified by the change specification
Reviews - Input
•
A statement of objectives for the technical reviews
•
The software element being examined
•
Software project management plan
•
Current anomalies or issues list for the software product
•
Documented review procedures
•
Earlier review report - when applicable
•
Review team members should receive the review materials in advance and they come prepared for the meeting
•
Check list for defects
Reviews – Meeting
•
Examine the software element for adherence to specifications and standards
•
Changes to software element are properly implemented and affect only the specified areas
•
Record all deviations
•
Assign responsibility for getting the issues resolved
•
Review sessions are not expected to find solutions to the deviations.
•
The areas of major concerns, status on previous feedback and review days utilized are also recorded.
•
The review leader shall verify, later, that the action items assigned in the meeting are closed
Reviews - Outputs
•
List of review findings
•
List of resolved and unresolved issues found during the later re-verification
4.2 Review process
The different types of reviews vary from very informal (e.g. no written instructions for reviewers) to very formal (i.e. well structured and regulated). The formality of a review process is related to factors such as the maturity of the development process, any legal or regulatory requirements or the need for an audit trail.
The way a review is carried out depends on the agreed objective of the review.
Phases of a formal review
A typical formal review has the following main phases:
1. Planning: selecting the personnel, allocating roles; defining the entry and exit criteria for more formal review types (e.g. inspection); and selecting which parts of documents to look at.
2. Kick-off: distributing documents; explaining the objectives, process and documents to the participants; and checking entry criteria (for more formal review types).
3. Individual preparation: work done by each of the participants on their own before the review meeting, noting potential defects, questions and comments.
4. Review meeting: discussion or logging, with documented results or minutes (for more formal review types). The meeting participants may simply note defects, make recommendations for handling the defects, or make decisions about the defects.
5. Rework: fixing defects found, typically done by the author.
6. Follow-up: checking that defects have been addressed, gathering metrics and checking on exit criteria (for more formal review types).
Roles and responsibilities
A typical formal review will include the roles below:
•
Manager: decides on the execution of reviews, allocates time in project schedules and determines if the review objectives have been met.
•
Moderator: the person who leads the review of the document or set of documents, including planning the review, running the meeting, and follow-up after the meeting. If necessary, the moderator may mediate between the various points of view and is often the person upon whom the success of the review rests.
•
Author: the writer or person with chief responsibility for the document(s) to be reviewed.
•
Reviewers: individuals with a specific technical or business background (also called checkers or inspectors) who, after the necessary preparation, identify and describe findings (e.g. defects) in the product under review. Reviewers should be chosen to represent different perspectives and roles in the review process, and should take part in any review meetings.
•
Scribe (or recorder): documents all the issues, problems and open points that were identified during the meeting.
Looking at documents from different perspectives and using checklists can make reviews more effective and efficient, for example, a checklist based on perspectives such as user, maintainer, tester or operations, or a checklist of typical requirements problems.
4.2.1 Types of review
A single document may be the subject of more than one review. If more than one type of review is used, the order may vary. For example, an informal review may be carried out before a technical review, or an inspection may be carried out on a requirements specification before a walkthrough with customers. The main characteristics, options and purposes of common review types are:
Informal review
Key characteristics:
•
no formal process;
•
there may be pair programming or a technical lead reviewing designs and code;
•
optionally may be documented;
•
may vary in usefulness depending on the reviewer;
•
main purpose: inexpensive way to get some benefit
Walkthrough
Walkthrough – Definition
A technique in which a designer or programmer leads the members of the development team and other interested parties through the segment of the documentation or code and the participants ask questions and make comments about possible errors, violation of standards and other problems.
Walkthrough - Objectives
•
To find defects
•
To consider alternative implementations
•
To ensure compliance to standards & specifications
Walkthrough – Input
•
A statement of objectives
•
The software element for examination
•
Standards for the development of the software
•
Distribution of materials to the team members, before the meeting
•
Team members shall examine and come prepared for the meeting
•
Check list for defects
Walkthrough- Meeting
•
Author presents the software element
•
Members ask questions and raise issues regarding deviations
•
Discuss concerns, perceived omissions or deviations from the specifications
•
Document the above discussions
•
Record the comments and decisions
•
The walk-through leader shall verify, later, that the action items assigned in the meeting are closed
Walkthrough – Outputs
•
List of walk-through findings
•
List of resolved and unresolved issues found during the later re-verification
Walkthrough - Key characteristics:
•
meeting led by author;
•
scenarios, dry runs, peer group;
•
open-ended sessions;
•
optionally a pre-meeting preparation of reviewers, review report, list of findings and scribe (who is not the author);
•
may vary in practice from quite informal to very formal;
•
main purposes: learning, gaining understanding, defect finding.
Technical review Key characteristics:
•
documented, defined defect-detection process that includes peers and technical experts;
•
may be performed as a peer review without management participation;
•
ideally led by trained moderator (not the author);
•
pre-meeting preparation;
•
optionally the use of checklists, review report, list of findings and management participation;
•
may vary in practice from quite informal to very formal;
•
main purposes: discuss, make decisions, evaluate alternatives, find defects, solve technical problems and check conformance to specifications and standards.
Inspection
Inspection – Definition
A visual examination of software element to detect errors, violations of development standards and other problems. An inspection is very rigorous and the inspectors are trained in inspection techniques. Determination of remedial or investigative action for a defect is a mandatory element of a software inspection, although the solution should not be determined in the inspection meeting.
Inspection – Objectives
•
To verify that the software element satisfies the specifications & conforms to applicable standards
•
To identify deviations
•
To collect software engineering data like defect and effort
•
To improve the checklists, as a spin-off
Inspection – Input
•
A statement of objectives for the inspection
•
Having the software element ready
•
Documented inspection procedure
•
Current defect list
•
A checklist of possible defects
•
Arrange for all standards and guidelines
•
Distribution of materials to the team members, before the meeting
•
Team members shall examine and come prepared for the inspection
Inspection – Meeting
•
Introducing the participants and describing their role (by the Moderator)
•
Presentation of the software element by non - author
•
Inspectors raise questions to expose the defects
•
Recording defects - a defect list details the location, description and severity of the defect
•
Reviewing the defect list - specific questions to ensure completeness and accuracy
•
Making exit decision
-
Acceptance with no or minor rework, without further verification.
-
Accept with rework verification (by inspection team leader or a designated member of the inspection team).
-
Re-inspect.
Inspection - Output
•
Defect list, containing a defect location, description and classification.
•
An estimate of rework effort and rework completion date.
Inspection - Key characteristics:
•
led by trained moderator (not the author);
•
usually peer examination;
•
defined roles;
•
includes metrics;
•
formal process based on rules and checklists with entry and exit criteria;
•
pre-meeting preparation;
•
inspection report, list of findings;
•
formal follow-up process;
•
optionally, process improvement and reader;
•
main purpose: find defects.
Walkthroughs, technical reviews and inspections can be performed within a peer group – colleagues at the same organizational level. This type of review is called a “peer review”.
Comparison of Reviews, Walk-Throughs and Inspections
Objectives:
•
Reviews - Evaluate conformance to specifications; Ensure change integrity
•
Walkthroughs - Detect defects; Examine alternatives; Forum for learning
•
Inspections - Detect and identify defects; Verify resolution
Group Dynamics:
•
Reviews: 3 or more persons ; Technical experts and peer mix
•
Walkthroughs: 2 to 7 persons Technical experts and peer mix
•
Inspections: 3 to 6 persons; Documented attendance; with formally trained inspectors
Decision Making & Change Control:
•
Reviews: Review Team requests Project Team leadership or management to act on recommendations
•
Walkthroughs: All decisions made by producer; Change is prerogative of the author
•
Inspections: Team declares exit decision – Acceptance, Rework & Verify or Rework & Re-inspect
Material Volume:
•
Reviews: Moderate to high
•
Walkthroughs: Relatively low
•
Inspections: Relatively low
Presenter
•
Reviews: Software Element Representative
•
Walkthroughs: Author
•
Inspections: Other than author
What are the Success Factors for Review?
Each review has a clear predefined objective.
•
The right people for the review objectives are involved.
•
Defects found are welcomed, and expressed objectively.
•
People issues and psychological aspects are dealt with (e.g. making it a positive experience for the author).
•
Review techniques are applied that are suitable to the type and level of software work products and reviewers.
•
Checklists or roles are used if appropriate to increase effectiveness of defect identification.
Training is given in review techniques, especially the more formal techniques, such as inspection.
•
Management supports a good review process (e.g. by incorporating adequate time for review activities in project schedules).
•
There is an emphasis on learning and process improvement.
4.4 Static analysis by tools
The objective of static analysis is to find defects in software source code and software models.
Static analysis is performed without actually executing the software being examined by the tool; dynamic testing does execute the software code. Static analysis can locate defects that are hard to find in testing. As with reviews, static analysis finds defects rather than failures. Static analysis tools analyze program code (e.g. control flow and data flow), as well as generated output such as HTML and XML.
The value of static analysis is:
•
Early detection of defects prior to test execution.
•
Early warning about suspicious aspects of the code or design, by the calculation of metrics, such as a high complexity measure.
•
Identification of defects not easily found by dynamic testing.
•
Detecting dependencies and inconsistencies in software models, such as links.
•
Improved maintainability of code and design.
•
Prevention of defects, if lessons are learned in development.
Typical defects discovered by static analysis tools include:
•
referencing a variable with an undefined value;
•
inconsistent interface between modules and components;
•
variables that are never used;
•
unreachable (dead) code;
•
programming standards violations;
•
security vulnerabilities;
•
syntax violations of code and software models.
Static analysis tools are typically used by developers (checking against predefined rules or programming standards) before and during component and integration testing, and by designers during software modeling. Static analysis tools may produce a large number of warning messages, which need to be well managed to allow the most effective use of the tool.
Advantages of Static Methods over Dynamic Methods
- Early detection of software defects
- Static methods expose defects, whereas dynamic methods show only the symptom of the defect
- Static methods expose a batch of defects, whereas it is usually one by one in dynamic methods
- Some defects can be found only by Static Testing
- Code redundancy (when logic is not affected)
- Dead code
- Violations of coding standard.
Common Issues for Reviews, Walk-throughs and Inspections
Responsibilities for Team Members
•
Leader: To conduct / moderate the review / walkthrough / inspection process
•
Reader: To present the relevant material in a logical fashion
•
Recorder: To document defects and deviations
•
Other Members: To critically question the material being presented
Communication Factors
•
Discussion is Kept within bounds
•
Not extreme in opinion
•
Reasonable and calm
•
Not directed at a personal level
•
Concentrate on finding defects
•
Not get bogged down with disagreement
•
Not discuss trivial issues
•
Not involve itself in solution-hunting
•
The participants should be sensitive to each other by keeping the synergy of the meeting very high
Being aware of, and correcting any conditions, either physical or emotional, that are draining off the participant’s attention, shall ensure that the meeting is fruitful – i.e. Maximum number of defects is found during the early stages of software development life cycle.
No comments:
Post a Comment