ISTQB:ADVANCED LEVEL EXAM
Static testing is a form of software testing where the software isn’t actually used. This is in contrast to dynamic testing. It is generally not detailed testing, but checks mainly for the sanity of the code, algorithm, or document. It is primarily syntax checking of the code or and manually reading of the code or document to find errors. This type of testing can be used by the developer who wrote the code, in isolation. Code reviews, inspections and walkthroughs are also used.
From the black box testing point of view, static testing involves review of requirements or specifications. This is done with an eye toward completeness or appropriateness for the task at hand. This is the verification portion of Verification and Validation.
Even static testing can be automated. A static testing test suite consists in programs to be analyzed by an interpreter or a compiler that asserts the programs syntactic validity.
Bugs discovered at this stage of development are less expensive to fix than later in the development cycle.
Static Code Analysis
Static code analysis is the analysis of computer software that is performed without actually executing programs built from that software (analysis performed on executing programs is known as dynamic analysis). In most cases the analysis is performed on some version of the source code and in the other cases some form of the object code. The term is usually applied to the analysis performed by an automated tool, with human analysis being called program understanding or program comprehension.
The sophistication of the analysis performed by tools varies from those that only consider the behavior of individual statements and declarations, to those that include the complete source code of a program in their analysis. Uses of the information obtained from the analysis vary from highlighting possible coding errors (e.g., the lint tool) to formal methods that mathematically prove properties about a given program (e.g., its behavior matches that of its specification).
Formal Methods
Formal methods is the term applied to the analysis of software (and hardware) whose results are obtained purely through the use of rigorous mathematical methods. The mathematical techniques used include denotational semantics, axiomatic semantics, operational semantics, and abstract interpretation.
It has been proven that, barring some hypothesis that the state space of programs is finite and small, finding possible run-time errors, or more generally any kind of violation of a specification on the final result of a program, is undecidable: there is no mechanical method that can always answer truthfully whether a given program may or may not exhibit runtime errors. This result dates from the works of Church, Gödel and Turing in the 1930s (see the halting problem and Rice’s theorem). As with most undecidable questions, one can still attempt to give useful approximate solutions.
Some of the implementation techniques of formal static analysis include:
* Model checking considers systems that have finite state or may be reduced to finite state by abstraction;
* Data-flow analysis is a lattice-based technique for gathering information about the possible set of values;
* Abstract interpretation models the effect that every statement has on the state of an abstract machine (i.e., it ‘executes’ the software based on the mathematical properties of each statement and declaration). This abstract machine overapproximates the behaviours of the system: the abstract system is thus made simpler to analyze, at the expense of incompleteness (not every property true of the original system is true of the abstract system). If properly done, though, abstract interpretation is sound (every property true of the abstract system can be mapped to a true property of the original system).
* Use of assertions in program code as first suggested by Hoare logic. There is tool support for some programming languages (e.g., the SPARK programming language (a subset of Ada) and the Java Modeling Language — JML — using ESC/Java and ESC/Java2).
Dynamic testing (or dynamic analysis) is a term used in software engineering to describe the testing of the dynamic behavior of code. That is, dynamic analysis refers to the examination of the physical response from the system to variables that are not constant and change with time. In dynamic testing the software must actually be compiled and run; this is in contrast to static testing. Dynamic testing is the validation portion of Verification and Validation.
Some of dynamic testing methodologies include unit testing, integration testing, system testing and acceptance testing.
Dynamic Testing involves working with the software, giving input values and checking if the output is as expected. These are the Validation activities. Unit Tests, Integration Tests, System Tests and Acceptance Tests are few of the Dynamic Testing methodologies.
Levels of DynamicTesting
1. Unit Testing
2. Integration Testing
3. System Testing
4. Acceptance Testing
5. Regression Testing
Unit Testing
In computer programming, unit testing is a method of testing that verifies the individual units of source code are working properly. A unit is the smallest testable part of an application. In procedural programming a unit may be an individual program, function, procedure, etc., while in object-oriented programming, the smallest unit is a method, which may belong to a base/super class, abstract class or derived/child class.
Ideally, each test case is independent from the others; Double objects like stubs, mock or fake objects as well as test harnesses can be used to assist testing a module in isolation. Unit testing is typically done by software developers to ensure that the code they have written meets software requirements and behaves as the developer intended.
Unit testing is a software development process in which the smallest testable parts of an application, called units, are individually and independently scrutinized for proper operation. Unit testing is often automated but it can also be done manually. This testing mode is a component of Extreme Programming (XP), a pragmatic method of software development that takes a meticulous approach to
building a product by means of continual testing and revision.
Unit testing involves only those characteristics that are vital to the performance of the unit under test. This encourages developers to modify the source code without immediate concerns about how such changes might affect the functioning of other units or the program as a whole. Once all of the units in a program have been found to be working in the most efficient and error-free manner possible, larger
components of the program can be evaluated by means of integration testing.
Unit testing can be time-consuming and tedious. It demands patience and thoroughness on the part of the development team. Rigorous documentation must be maintained. Unit testing must be done with an awareness that it may not be possible to test a unit for every input scenario that will occur when the program is run in a real-world environment.
Benefits
The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. A unit test provides a strict, written contract that the piece of code must satisfy. As a result, it affords several benefits. Unit tests find problems early in the development cycle.
Facilitates change
Unit testing allows the programmer to refactor code at a later date, and make sure the module still works correctly (i.e. regression testing). The procedure is to write test cases for all functions and methods so that whenever a change causes a fault, it can be quickly identified and fixed.
Readily-available unit tests make it easy for the programmer to check whether a piece of code is still working properly. Good unit test design produces test cases that cover all paths through the unit with attention paid to loop conditions.
In continuous unit testing environments, through the inherent practice of sustained maintenance, unit tests will continue to accurately reflect the intended use of the executable and code in the face of any change. Depending upon established development practices and unit test coverage, up-to-the-second
accuracy can be maintained.
Simplifies integration
Unit testing helps to eliminate uncertainty in the units themselves and can be used in a bottom-up testing style approach. By testing the parts of a program first and then testing the sum of its parts, integration testing becomes much easier.
A heavily debated matter exists in assessing the need to perform manual integration testing. While an elaborate hierarchy of unit tests may seem to have achieved integration testing, this presents a false sense of confidence since integration testing evaluates many other objectives that can only be proven through the human factor. Some argue that given a sufficient variety of test automation systems,
integration testing by a human test group is unnecessary. Realistically, the actual need will ultimately depend upon the characteristics of the product being developed and its intended uses. Additionally, the human or manual testing will greatly depend on the availability of resources in the organization.
Documentation
Unit testing provides a sort of living documentation of the system. Developers looking to learn what functionality is provided by a unit and how to use it can look at the unit tests to gain a basic understanding of the unit API.
Unit test cases embody characteristics that are critical to the success of the unit. These characteristics can indicate appropriate/inappropriate use of a unit as well as negative behaviors that are to be trapped by the unit. A unit test case, in and of itself, documents these critical characteristics, although many software development environments do not rely solely upon code to document the product in development.
On the other hand, ordinary narrative documentation is more susceptible to drifting from the implementation of the program and will thus become outdated (e.g. design changes, feature creep, relaxed practices to keep documents up to date).
Design
When software is developed using a test-driven approach, the Unit-Test may take the place of formal design. Each unit test can be seen as a design element specifying classes, methods, and observable behaviour. The following Java example will help illustrate this point.
Here is a test class that specifies a number of elements of the implementation. First, that there must be an interface called Adder, and an implementing class with a zero-argument constructor called AdderImpl. It goes on to assert that the Adder interface should have a method called add, with two integer parameters,
which returns another integer. It also specifies the behaviour of this method for a small range of values.
public class TestAdder {
public void testSum() {
Adder adder = new AdderImpl();
assertTrue(adder.add(1,1) == 2);
assertTrue(adder.add(1,2) == 3);
assertTrue(adder.add(2,2) == 4);
assertTrue(adder.add(0,0) == 0);
assertTrue(adder.add(-1,-2) == -3);
assertTrue(adder.add(-1,1) == 0);
assertTrue(adder.add(1234,988) == 2222);
}
}
In this case the unit test, having been written first, acts as a design document specifying the form and behaviour of a desired solution, but not the implementation details, which are left as an exercise for the programmer. Following the ‘do the simplest thing that could possibly work’ practice, the easiest solution that will make the test pass is shown below.
interface Adder {
int add(int a, int b);
}
class AdderImpl implements Adder {
int add(int a, int b) {
return a + b;
}
}
Unlike other diagram-based design methods, using a unit-test as a design has one significant advantage.
The design document (the unit-test itself) can be used to verify that the implementation adheres to the design.
UML suffers from the fact that although a diagram may name a class Customer, the developer can call the class Wibble and nothing in the system would note this discrepancy. With the unit-test design method, the tests will never pass if the developer does not implement the solution according to the design.
It is true that unit-testing lacks some of the accessibility of a diagram, but UML diagrams are now easily generated for most modern languages by free tools (usually available as extensions to IDEs). Free tools, like those based on the xUnit framework, outsource to another system the graphical rendering of a view for human
consumption.
Separation of interface from implementation
Because some classes may have references to other classes, testing a class can frequently spill over into testing another class. A common example of this is classes that depend on a database: in order to test the class, the tester often writes code that interacts with the database. This is a mistake, because a unit test
should never go outside of its own class boundary. Instead, the software developer should create an abstract interface around the database connection, and then implement that interface with their own mock object. By abstracting this necessary attachment from the code (temporarily reducing the net effective coupling), the
independent unit can be more thoroughly tested than may have been previously achieved. This results in a higher quality unit that is also more maintainable.
Limitations of unit testing
Testing cannot be expected to catch every error in the program – it is impossible to evaluate all execution paths for all but the most trivial programs. The same is true for unit testing. Additionally, by definition unit testing only tests the functionality of the units themselves. Therefore it will not catch integration errors, or broader
system level errors (such as functions performed across multiple units, or non-functional test areas such as performance).
Unit testing is more effective if it is used in conjunction with other software testing activities. Like all forms of software testing, unit tests can only show the presence of errors; it cannot show the absence of errors.
Software testing is a combinatorial problem. For example, every boolean decision statement requires at least two tests:
one with an outcome of “true” and one with an outcome of “false”. As a result, for every line of code written, programmers often need 3 to 5 lines of test code.
To obtain the intended benefits from unit testing, a rigorous sense of discipline is needed throughout the software development process. It is essential to keep careful records not only of the tests that have been performed, but also of all changes that have been made to the source code of this or any other unit in the software. Use of a version control system is essential. If a later version of the unit fails a particular test that it had previously passed, the version-control software can provide a list of the source code changes (if any) that have been applied to the unit since that time.
It is also essential to implement a sustainable process for ensuring that test case failures are reviewed daily and addressed immediately. If such a process is not implemented and ingrained into the team’s workflow, the application
will evolve out of sync with the unit test suite ― increasing false positives and reducing the effectiveness of the test suite.
Applications
Extreme Programming
Unit testing is the cornerstone of Extreme Programming (XP), which relies on an automated unit testing framework.
This automated unit testing framework can be either third party, e.g. xUnit, or created within the development group.
Extreme Programming uses the creation of unit tests for test-driven development. The developer writes a unit test that exposes either a software requirement or a defect. This test will fail because either the requirement isn’t implemented yet, or because it intentionally exposes a defect in the existing code. Then, the developer writes the simplest code to make the test, along with other tests, pass.
Most code in a system is unit tested, but not necessarily all paths through the code. XP mandates a ‘test everything that can possibly break’ strategy, over the tradition ‘test every execution path’ method. This leads XP developers to develop fewer tests than classical methods, but this isn’t really a problem, more a restatement of fact, as classical methods have rarely ever been followed methodically enough for all execution paths to have been thoroughly tested.
XP simply recognizes that testing is rarely exhaustive (because often that is too expensive and time consuming to be economically viable), and provides guidance on how to effectively focus the limited resources we can afford expend on
the problem.
Crucially, the test code is considered a first class project artifact in that it is maintained at the same quality as the implementation code, with all duplication removed. Developers release unit testing code to the code repository in conjunction
with the code it tests. XP’s thorough unit testing allows the benefits mentioned above, such as simpler and more confident code development and refactoring, simplified code integration, accurate documentation, and more modular designs.
These unit tests are also constantly run as a form of regression test.
Six Rules of Unit Testing
1. Write the test first
2. Never write a test that succeeds the first time
3. Start with the null case, or something that doesn’t work
4. Don’t be afraid of doing something trivial to make the test work
5. Loose coupling and testability go hand in hand
6. Use mock objects
What is Regression testing?
Regression testing is any type of software testing which seeks to uncover software regressions. Such regressions occur whenever software functionality that was previously working correctly stops working as intended. Typically regressions occur as an unintended consequence of program changes.
Common methods of regression testing include re-running previously run tests and checking whether previously fixed faults have re-emerged.
Background
Experience has shown that as software is developed, this kind of reemergence of faults is quite common. Sometimes it occurs because a fix gets lost through poor revision control practices (or simple human error in revision control), but often a fix for a problem will be “fragile” in that it fixes the problem in the narrow case where it was first observed but not in more general cases which may arise over the lifetime of the software. Finally, it has often been the case that when some feature is redesigned, the same mistakes will be made in the redesign that were made in the original implementation of the feature.
Therefore, in most software development situations it is considered good practice that when a bug is located and fixed, a test that exposes the bug is recorded and regularly retested after subsequent changes to the program. Although this may be done through manual testing procedures using programming techniques, it is often done using automated testing tools. Such a test suite contains software tools that allow the testing environment to execute all the regression test cases automatically; some projects even set up automated systems to automatically re-run all regression tests at specified intervals and report any failures (which could imply a regression or an out-of-date test). Common strategies are to run such a system after every successful compile (for small projects), every night, or once a week. Those strategies can be automated by an external tool, such as BuildBot.
Regression testing is an integral part of the extreme programming software development method. In this method, design documents are replaced by extensive, repeatable, and automated testing of the entire software package at every stage in the software development cycle.
Regression test generation
Effective regression tests generate sufficient code execution coverage to exercise all meaningful code branches. Therefore, software testing is a combinatorial problem. However, in practice many combinations are unreachable so the problem size is greatly reduced. Every boolean decision statement requires at least two tests: one with an outcome of “true” and one with an outcome of “false”. As a result, for every line of code written, programmers often need 3 to 5 lines of test code.
Traditionally, in the corporate world, regression testing has been performed by a software quality assurance team after the development team has completed work. However, defects found at this stage are the most costly to fix. This problem is being addressed by the rise of developer testing. Although developers have always written test cases as part of the development cycle, these test cases have generally been either functional tests or unit tests that verify only intended outcomes. Developer testing compels a developer to focus on unit testing and to include both positive and negative test cases.
When regression test generation is supported by a sustainable process for ensuring that test case failures are reviewed daily and addressed immediately, the end result is a regression test suite that evolves with the application, and becomes more robust and more intelligent each day. If such a process is not implemented and ingrained into the team’s workflow, the application may evolve out of sync with the generated test suite — increasing false positives and reducing the effectiveness of the test suite.
Uses
Regression testing can be used not only for testing the correctness of a program, but it is also often used to track the quality of its output. For instance in the design of a compiler, regression testing should track the code size, simulation time and compilation time of the test suite cases.
Exception Handling Testing
Exception handling is a programming language construct or computer
hardware mechanism designed to handle the occurrence of some
condition that changes the normal flow of execution. For signalling
conditions that are part of the normal flow of execution see the concepts
of signal and event handler.
Exception or error handling refers to the anticipation, detection, and
resolution of programming, application, and communications errors.
Specialized programs, called error handlers, are available for some
applications. The best programs of this type forestall errors if possible,
recover from them when they occur without terminating the application,
or (if all else fails) gracefully terminate an affected application and save
the error information to a log file.
In programming, a development error is one that can be prevented.
Such an error can occur in syntax or logic. Syntax errors, which are
typographical mistakes or improper use of special characters, are
handled by rigorous proofreading. Logic errors, also called bugs,
occur when executed code does not produce the expected or desired
result. Logic errors are best handled by meticulous program debugging.
This can be an ongoing process that involves, in addition to the traditional
debugging routine, beta testing prior to official release and customer feedback
after official release.
A run-time error takes place during the execution of a program, and usually
happens because of adverse system parameters or invalid input data. An
example is the lack of sufficient memory to run an application or a memory
conflict with another program. On the Internet, run-time errors can result from
electrical noise, various forms of malware or an exceptionally heavy demand on
a server. Run-time errors can be resolved, or their impact minimized, by the use
of error handler programs, by vigilance on the part of network and server
administrators, and by reasonable security countermeasures on the part of
Internet users. In runtime engine environments such as Java or .NET there exist
tools that attach to the runtime engine and every time that an exception of interest
occurs they record debugging information that existed in memory at the time the
exception was thrown (call stack and heap values). These tools are called
Automated Exception Handling or Error Interception tools and they provide
‘root-cause’ information for exceptions.
Usage
* It determines the ability of applications system to process the incorrect
transactions properly
* Errors encompass all unexpected conditions.
* In some system approx. 50% of programming effort will be devoted to
handling error condition.
Objective
* Determine Application system recognizes all expected error conditions.
* Determine Accountability of processing errors has been assigned and
procedures provide a high probability that errors will be properly
corrected.
* Determine During correction process reasonable control is maintained
over errors.
How to Use:
* A group of knowledgeable people is required to anticipate what can go
wrong in the application system.
* It is needed that all the application knowledgeable people assemble to
integrate their knowledge of user area, auditing and error tracking.
* Then logical test error conditions should be created based on this
assimilated information.
When to Use:
* Throughout SDLC.
* Impact from errors should be identified and should be corrected to
reduce the errors to acceptable level.
* Used to assist in error management process of system development
and maintenance.
Example:
* Create a set of erroneous transactions and enter them into the
application system then find out whether the system is able to
identify the problems.
* Using iterative testing enters transactions and trap errors.
Correct them. Then enter transactions with errors, which were
not present in the system earlier.
Load Testing
Load testing generally refers to the practice of modelling the expected
Usage of a software program by simulating multiple users accessing
the program’s services concurrently. As such, this testing is most
relevant for multi-user systems, often one built using a client/server
model, such as web servers. However, other types of software systems
can be load-tested also. For example, a word processor or graphics
editor can be forced to read an extremely large document; or a
financial package can be forced to generate a report based on several
years’ worth of data. The most accurate load testing occurs with
actual, rather than theoretical, results.
When the load placed on the system is raised beyond normal
usage patterns, in order to test the system’s response at unusually
high or peak loads, it is known as stress testing. The load is
usually so great that error conditions are the expected
result, although no clear boundary exists when an activity ceases
to be a load test and becomes a stress test.
There is little agreement on what the specific goals of load testing are.
The term is often used synonymously with performance testing, reliability
testing, and volume testing.
Testing
Load and Performance testing is to test software intended for a multi-user
audience for the desired performance by subjecting it with an equal amount
of virtual users and then monitoring the performance under the specified load,
usually in a test environment identical to the production, before going live.
For example if a web site with a shopping cart is intended for 100 concurrent
users who are doing the following functions:
* 25 VUsers are browsing through the items and logging off
* 25 Vusers are adding items to the shopping cart and checking out
and logging off
* 25 VUsers are returning items previously purchased and logging off
* 25 VUsers are just logged in without any activity
Using various tools available to generate these VUsers the application is
subjected to a 100 VUser load as shown above and its performance is
monitored. The pass fail criteria is different for each individual organization
and there are no standards on what an acceptable criteria should be, across
the board.
It is a common misconception that these are record and playback tools like
regression testing tools, however it must be clarified that the similarity ends
just there. The Load testing tools work at the protocol level where as the
regression testing tools work at the GUI object level. To give an example
a regression testing tool will simulate a mouse click on an OK button on
the browser, but a load testing tool will send out the hypertext that the browser
will send after the user clicks the OK button, and again it will send out the
hypertext for multiple users each having a unique login ID and password.
Tools
Various tools are also available to find out the causes for slow performance
which could be in the following areas:
* Application
* Database
* Network
* Client side processing
* Load Balancer
Input
The following are useful inputs for load-testing a Web application:
· Performance-critical usage scenarios
· Workload models
· Performance acceptance criteria
· Performance metrics associated with the acceptance criteria
· Interview feedback from the designer or developer of the Web application
· Interview feedback from end users of the application
· Interview feedback from the operations personnel who will maintain and manage the application
Output
The main outcomes that load testing helps you to accomplish are:
· Updated test plans and test designs for load and performance testing
· Various performance measures such as throughput, response time, and resource utilization
· Potential bottlenecks that need to be analyzed in the white-box testing phase
· The behavior of the application at various load level
Approach for Load Testing
The following steps are involved in load-testing a Web application:
Step 1 – Identify performance acceptance criteria
Step 2 – Identify key scenarios
Step 3 – Create a workload model
Step 4 – Identify the target load levels
Step 5 – Identify metrics
Step 6 – Design specific tests
Step 7 – Run tests
Step 8 – Analyze the results
Summary
Load testing helps to identify the maximum operating capacity of the
application and any bottlenecks that might be degrading performance.
The basic methodology for performing load testing on a Web
application is to identify the performance-critical key scenarios; identify
the workload profile for distributing all the load among the key scenarios;
identify metrics that you want to collect in order to verify them against
your performance objectives; create test cases that will be used to simulate
the load test; use tools to simulate the load according to the test cases and
capture the metrics; and finally, analyze the metrics data captured during
the tests.
Stress Testing
Stress testing often refers to tests that put a greater
emphasis on robustness, availability, and error handling under a heavy
load, rather than on what would be considered correct behavior under
normal circumstances. In particular, the goals of such tests may be to
ensure the software doesn’t crash in conditions of insufficient computational
resources (such as memory or disk space), unusually high concurrency, or
denial of service attacks.
Stress testing is a type of performance testing focused on determining an
application’s robustness, availability, and reliability under extreme conditions.
The goal of stress testing is to identify application issues that arise or become
apparent only under extreme conditions. These conditions can include heavy
loads, high concurrency, or limited computational resources. Proper stress
testing is useful in finding synchronization and timing bugs, interlock problems,
priority problems, and resource loss bugs. The idea is to stress a system to the
breaking point in order to find bugs that will make that break potentially
harmful. The system is not expected to process the overload without adequate
resources, but to behave (e.g., fail) in an acceptable manner
(e.g., not corrupting or losing data).
Stress tests typically involve simulating one or more key production scenarios
under a variety of stressful conditions. For example, you might deploy your
application on a server that is already running a processor-intensive application;
in this way, your application is immediately “starved” of processor resources
and must compete with the other application for processor cycles. You can also
stress-test a single Web page or even a single item such as a stored procedure
or class.
Examples of Stress Conditions
Examples of stress conditions include:
* Excessive volume in terms of either users or data; examples might include
a denial of service (DoS) attack or a situation where a widely viewed
news item prompts a large number of users to visit a Web site during a
three-minute period.
* Resource reduction such as a disk drive failure.
* Unexpected sequencing.
* Unexpected outages/outage recovery.
Examples of Stress-Related Symptoms
Examples of stress-related symptoms include:
* Data is lost or corrupted.
* Resource utilization remains unacceptably high after the
stress is removed.
* Application components fail to respond.
* Unhandled exceptions are presented to the end user.
Input
To perform stress testing, you are likely to use as reference
one or more of the following items:
* Results from previous stress tests
* Application usage characteristics (scenarios)
* Concerns about those scenarios under extreme conditions
* Workload profile characteristics
* Current peak load capacity (obtained from load testing)
* Hardware and network architecture and data
* Disaster-risk assessment (e.g., likelihood of blackouts, earthquakes, etc.)
Output
Output from a stress test may include:
* Measures of the application under stressful conditions
* Symptoms of the application under stress
* Information the team can use to address robustness, availability,
and reliability
Approach for Stress Testing
The following steps are involved in stress-testing a Web application:
Step1 – Identify test objectives. Identify the objectives of stress
testing in terms of the desired outcomes of the testing activity.
Step2 – Identify key scenario(s). Identify the application scenario or
cases that need to be stress-tested to identify potential problems.
Step3 – Identify the workload. Identify the workload that you want to
apply to the scenarios identified during the “Identify objectives” step.
This is based on the workload and peak load capacity inputs.
Step4 – Identify metrics. Identify the metrics that you want to collect about
the application’s performance. Base these metrics on the potential
problems identified for the scenarios you identified during the
“Identify objectives” step.
Step 5 – Create test cases. Create the test cases in which you define steps
for running a single test, as well as your expected results.
Step 6 – Simulate load. Use test tools to simulate the required load for
each test case and capture the metric data results.
Step 7 – Analyze results. Analyze the metric data captured during the test.
Summary
Stress testing allows you to identify potential application issues that surface only
under extreme conditions. Such conditions range from exhaustion of system
resources such as memory, processor cycles, network bandwidth, and disk
capacity to excessive load due to unpredictable usage patterns, common in
Web applications.
Stress testing centers around objectives and key user scenarios with an
emphasis on the robustness, reliability, and stability of the application.
The effectiveness of stress testing relies on applying the correct methodology
and being able to effectively analyze testing results. Applying the correct
methodology is dependent on the capacity for reproducing workload
conditions for both user load and volume of data, reproducing key scenarios,
and interpreting the key performance metrics
In software engineering, performance testing is testing that is
performed, from one perspective, to determine how fast
some aspect of a system performs under a particular
workload. It can also serve to validate and verify other quality
attributes of the system, such as scalability, reliability and
resource usage. Performance testing is a subset of
Performance engineering, an emerging computer science
practice which strives to build performance into the design
and architecture of a system, prior to the onset of actual
coding effort.
Performance testing can serve different purposes. It can
demonstrate that the system meets performance criteria.
It can compare two systems to find which performs better.
Or it can measure what parts of the system or workload
cause the system to perform badly. In the diagnostic case,
software engineers use tools such as profilers to measure
what parts of a device or software contribute most to the
poor performance or to establish throughput levels
(and thresholds) for maintained acceptable response time.
It is critical to the cost performance of a new system, that
performance test efforts begin at the inception of the
development project and extend through to deployment.
The later a performance defect is detected, the higher the
cost of remediation. This is true in the case of functional
testing, but even more so with performance testing, due to
the end-to-end nature of its
scope.
In performance testing, it is often crucial (and often difficult
to arrange) for the test conditions to be similar to the
expected actual use. This is, however, not entirely possible
in actual practice. The reason is that production systems
have a random nature of the workload and while
the test workloads do their best to mimic what may happen
in the production environment, it is impossible to exactly
replicate this workload variability – except in the most
simple system.
Loosely-coupled architectural implementations (e.g.: SOA)
have created additional complexities with performance testing.
Enterprise services or assets (that share common infrastructure
or platform) require coordinated performance testing (with all
consumers creating production-like transaction volumes and
load on shared infrastructures or platforms) to truly replicate
production-like states. Due to the complexity and financial and
time requirements around this activity, some organizations now
employ tools that can monitor and create production-like conditions
(also referred as “noise”) in their performance testing environments
(PTE) to understand capacity and resource requirements and
verify / validate quality attributes.
Risks Addressed Through Performance Testing
Speed-Related Risks
Speed-related risks are not confined to end-user satisfaction, although
that is what most people think of first. Speed is also a factor in certain
business- and data-related risks. Some of the most common speed-related
risks that performance testing can address include:
* Is the application fast enough to satisfy end users?
* Is the business able to process and utilize data collected by the
application before that data becomes outdated? (For example,
end-of-month reports are due within 24 hours of the close of
business on the last day of the month, but it takes the application
48 hours to process the data.)
* Is the application capable of presenting the most current information
(e.g., stock quotes) to its users?
* Is a Web Service responding within the maximum expected response
time before an error is thrown?
Scalability-Related Risk-Mitigation Strategies
The following strategies are valuable in mitigating scalability-related risks:
* Compare measured speeds under various loads. (Keep in mind that
the end user does not know or care how many other people are
using the application at the same time that he/she is.)
* Design load tests that replicate actual workload at both normal and
anticipated peak times.
* Conduct performance testing with data types, distributions, and volumes
similar to those used in business operations during actual production (e.g.,
number of products, orders in pending status, size of user base). You can
allow data to accumulate in databases and file servers, or additionally
create the data volume, before load test execution.
* Use performance test results to help stakeholders make informed
architecture and business decisions.
* Work with more meaningful performance tests that map to the
real-world requirements.
* When you find a scalability limit, incrementally reduce the load and retest
to help you identify a metric that can serve as a reliable indicator that the
application is approaching that limit in enough time for you to apply
countermeasures.
* Conduct performance tests beyond expected peak loads and observe
behavior by having representative users and stakeholders access the
application manually during and after the performance test.
Stability-Related Risks
Stability is a blanket term that encompasses such areas as reliability, uptime,
and recoverability. Although stability risks are commonly addressed with
high-load, endurance, and stress tests, stability issues are sometimes detected
during the most basic performance tests. Some common stability risks
addressed by means of performance testing include:
* Can the application run for long periods of time without data corruption,
slowdown, or servers needing to be rebooted?
* If the application does go down unexpectedly, what happens to partially
completed transactions?
* When the application comes back online after scheduled or unscheduled
downtime, will users still be able to see/do everything they expect?
* When the application comes back online after unscheduled downtime,
does it resume at the correct point? In particular, does it not attempt to
resume cancelled transactions?
* Can combinations of errors or repeated functional errors cause a
system crash?
* Are there any transactions that cause system-wide side effects?
* Can one leg of the load-balanced environment be taken down and
still provide uninterrupted service to users?
* Can the system be patched or updated without taking it down?
Stability-Related Risk-Mitigation Strategies
The following strategies are valuable in mitigating stability-related risks:
* Make time for realistic endurance tests.
* Conduct stress testing with the key scenarios. Work with key
performance indicators (network, disk, processor, memory)
and business indicators such as number of orders lost, user login
failures, and so on.
* Conduct stress testing with data feeds that replicate similar business
operations as in an actual production environment (e.g., number of
products, orders in pending status, size of user base). You can allow
data to accumulate in databases and file servers, or additionally create
the data volume, before stress test execution. This will allow you to
replicate critical errors such as database or application deadlocks and
other stress failure patterns.
* Take a server offline during a test and observe functional, performance,
and data-integrity behaviors of the remaining systems.
* Execute identical tests immediately before and after a system reboot.
Compare the results. You can use an identical approach for recycling
services or processes.
* Include error or exception cases in your performance test scenarios
(for example, users trying to log on with improper credentials).
* Apply a patch to the system during a performance test.
* Force a backup and/or virus definition update during a performance
test.
* Validate the functional accuracy of the application under various loads
by checking database entries created or validating content returned in
response to particular user requests.
Summary
Almost all application- and business-related risks can be addressed through
performance testing, including user satisfaction and the application’s ability
to achieve business goals.
Generally, the risks that performance testing addresses are categorized in
terms of speed, scalability, and stability. Speed is typically an end-user
concern, scalability is a business concern, and stability is a technical or
maintenance concern.
Identifying project-related risks and the associated mitigation strategies
where performance testing can be employed is almost universally viewed
as a valuable and time-saving practice.
Filed under: Uncategorized | Tagged: Software Testing
good on this