Let�s use this system to understand and clarify the
characteristics of black box and white box testing.
People: Who does the testing?
Some people know how software works (developers) and others just use it (users).
Accordingly, any testing by users or other nondevelopers is sometimes called
�black box� testing. Developer testing is called �white box� testing. The
distinction here is based on what the person knows or can understand.
Coverage: What is tested?
If we draw the box around the system as a whole, �black box� testing becomes
another name for system testing. And testing the units inside the box becomes
white box testing. This is one way to think about coverage.
Another is to contrast testing that aims to cover all the requirements with
testing that aims to cover all the code. These are the two most commonly used
coverage criteria. Both are supported by extensive literature and commercial
tools. Requirements-based testing could be called �black box� because it makes
sure that all the customer requirements have been verified. Code-based testing
is often called �white box� because it makes sure that all the code (the
statements, paths, or decisions) is exercised.
Risks: Why are you testing?
Sometimes testing is targeted at particular risks. Boundary testing and other
attack-based techniques are targeted at common coding errors. Effective security
testing also requires a detailed understanding of the code and the system
architecture. Thus, these techniques might be classified as �white box.�
Another set of risks concerns whether the software will actually provide value
to users. Usability testing focuses on this risk, and could be termed �black
box.�
Activities: How do you test?
A common distinction is made between behavioral test design, which defines tests
based on functional requirements, and structural test design, which defines
tests based on the code itself. These are two design approaches. Since
behavioral testing is based on external functional definition, it is often
called �black box,� while structural testing�based on the code internals�is
called �white box.� Indeed, this is probably the most commonly cited definition
for black box and white box testing.
Another activity-based distinction contrasts dynamic test execution with formal
code inspection. In this case, the metaphor maps test execution (dynamic
testing) with black box testing, and maps code inspection (static testing) with
white box testing.
We could also focus on the tools used. Some tool vendors refer to code-coverage
tools as white box tools, and tools that facilitate applying inputs and
capturing inputs�most notably GUI capture replay tools�as black box tools.
Testing is then categorized based on the types of tools used.
Evaluation: How do you know if you�ve found a bug?
There are certain kinds of software faults that don�t always lead to obvious
failures. They may be masked by fault tolerance or simply luck. Memory leaks and
wild pointers are examples. Certain test techniques seek to make these kinds of
problems more visible. Related techniques capture code history and stack
information when faults occur, helping with diagnosis. Assertions are another
technique for helping to make problems more visible. All of these techniques
could be considered white box test techniques, since they use code
instrumentation to make the internal workings of the software more visible.
These contrast with black box techniques that simply look at the official
outputs of a program.
To summarize, black box testing can sometimes describe user-based testing
(people); system or requirements-based testing (coverage); usability testing
(risk); or behavioral testing or capture replay automation (activities). White
box testing, on the other hand, can sometimes describe developer-based testing
(people); unit or code-coverage testing (coverage); boundary or security testing
(risks); structural testing, inspection or code-coverage automation
(activities); or testing based on probes, assertions, and logs (evaluation).
