OneStopTesting - Quality Testing Jobs, eBooks, Articles, FAQs, Training Institutes, Testing Software, Testing downloads, testing news, testing tools, learn testing, manual testing, automated testing, load runner, winrunner, test director, silk test, STLC

Forum| Contact Us| Testimonials| Sitemap| Employee Referrals| News| Articles| Feedback| Enquiry
Testing Resources
  • Testing Articles
  • Testing Books
  • Testing Certification
  • Testing FAQs
  • Testing Downloads
  • Testing Interview Questions
  • Career In Software Testing
  • Testing Jobs
  • Testing Job Consultants
  • Testing News
  • Testing Training Institutes
  • Introduction
  • Designing Test Cases
  • Developing Test Cases
  • Writing Test Cases
  • Test Case Templates
  • Purpose
  • What Is a Good Test Case?
  • Test Specifications
  • UML
  • Scenario Testing
  • Test Script
  • Test Summary Report
  • Test Data
  • Defect Tracking
    Software testing
  • Testing Forum
  • Introduction
  • Testing Start Process
  • Testing Stop Process
  • Testing Strategy
  • Risk Analysis
  • Software Listings
  • Test Metrics
  • Release Life Cycle
  • Interoperability Testing
  • Extreme Programming
  • Cyclomatic Complexity
  • Equivalence Partitioning
  • Error Guessing
  • Boundary Value Analysis
  • Traceability Matrix
    SDLC Models
  • Introduction
  • Waterfall Model
  • Iterative Model
  • V-Model
  • Spiral Model
  • Big Bang Model
  • RAD Model
  • Prototyping Model
    Software Testing Types
  • Static Testing
  • Dynamic Testing
  • Blackbox Testing
  • Whitebox Testing
  • Unit Testing
  • Requirements Testing
  • Regression Testing
  • Error Handling Testing
  • Manual support Testing
  • Intersystem Testing
  • Control Testing
  • Parallel Testing
  • Volume Testing
  • Stress Testing
  • Performance Testing
  • Agile Testing
  • Localization Testing
  • Globalization Testing
  • Internationalization Testing
    Test Plan
  • Introduction
  • Test Plan Development
  • Test Plan Template
  • Regional Differences
  • Criticism
  • Hardware Development
  • IEEE 829-1998
  • Testing Without a TestPlan
    Code Coverage
  • Introduction
  • Measures
  • Working
  • Statement Coverage
  • Branch Coverage
  • Path Coverage
  • Coverage criteria
  • Code coverage in practice
  • Tools
  • Features
    Quality Management
  • Introduction
  • Components
  • Capability Maturity Model
  • CMMI
  • Six Sigma
    Project Management
  • Introduction
  • PM Activities
  • Project Control Variables
  • PM Methodology
  • PM Phases
  • PM Templates
  • Agile PM
    Automated Testing Tools
  • Quick Test Professional
  • WinRunner
  • LoadRunner
  • Test Director
  • Silk Test
  • Test Partner
  • Rational Robot
    Performance Testing Tools
  • Apache JMeter
  • Rational Performance Tester
  • LoadRunner
  • NeoLoad
  • WAPT
  • WebLOAD
  • Loadster
  • OpenSTA
  • LoadUI
  • Appvance
  • Loadstorm
  • LoadImpact
  • QEngine
  • Httperf
  • CloudTest
  • Perl Testing
  • Python Testing
  • JUnit Testing
  • Unix Shell Scripting
    Automation Framework
  • Introduction
  • Keyword-driven Testing
  • Data-driven Testing
    Configuration Management
  • History
  • What is CM?
  • Meaning of CM
  • Graphically Representation
  • Traditional CM
  • CM Activities
  • Tools
  • What Is Software Testing?
  • Effective Defect Reports
  • Software Security
  • Tracking Defects
  • Bug Report
  • Web Testing
  • Exploratory Testing
  • Good Test Case
  • Write a Test
  • Code Coverage
  • WinRunner vs. QuickTest
  • Web Testing Tools
  • Automated Testing
  • Testing Estimation Process
  • Quality Assurance
  • The Interview Guide
  • Upgrade Path Testing
  • Priority and Severity of Bug
  • Three Questions About Bug
    Home » Testing Articles » Load, Stress & Performance Testing Articles » Automation for the people: Continuous testing

    Automation for the people: Continuous testing

    A D V E R T I S E M E N T

    Running unit tests from an IDE like Eclipse or an Ant build script, for example, is a great start toward ensuring the quality of an application; however, running unit tests any time source code changes in a version control repository, like Subversion, on a separate, clean build machine can help verify problems throughout a development life cycle. Moreover, running different types of developer tests, such as component, functional and performance, can shine a light on problem areas earlier in the development life cycle.

    Developer tests, which when run often in a Continuous Integration (CI) environment, effectively act like a spot light on code quality. This is because these tests, if written effectively, can find issues like defects almost as soon as they're created. Tests that aren't run often are less effective because the time between when a defect is coded and when it can be discovered is long, but running tests continually (that is, every time code changes) ensures that unintended behavior is discovered quickly.

    This article covers the following:

    • Running JUnit tests via Ant
    • Executing longer running component tests using JUnit and DbUnit
    • Determining which methods take too long to execute using JUnitPerf
    • Running Web-based functional tests with Selenium
    • Assessing code coverage with Cobertura
    • Testing continually with CruiseControl

    I provide an overview of the different types of developer tests along with examples that you can add to your build process and run continually using a Continuous Integration system.

    Unit testing a la JUnit

    You say unit test, I say component test

    What is often referred to as a unit test is more likely a component-level test. A component test typically verifies more than one class and has a dependency on something like a database or other heavy mechanism, like a file system. Most importantly, however, on a test by test basis, component tests take much longer to run than a unit test.

    Sometimes I hear developers lump the term developer tests as simple unit tests; however, I've found that it's helpful to refine the term unit testing as something more specific. To me, unit tests are fast running tests that typically test individual classes that don't have heavy external dependencies like databases. For example, Listing 1 defines a unit test that uses JUnit to verify a stubbed out data class called BeerDaoStub. The technique of testing against an interface that doesn't actually connect to a database, for example, is an approach that you can use to verify business functionality without incurring expensive setup costs. Plus, doing so keeps the test as a true unit test.

    Listing 1. A simple unit test
    public void setUp() {    beerService = new BeerDaoStub();  }    public void testUnitGetBeer() {    Collection beers = beerService.findAll();    assertTrue(beers != null && beers.size() > 0);  }  

    Once you've written a few unit tests, you can always run them through an IDE, but you'll also want to run them as part of a build process. Ensuring that tests run successfully through a build process means these same tests can be kicked off in the context of a CI build too.

    Listing 2 is a snippet of an Ant script that demonstrates the junit task to execute a batch of unit tests. The beauty with this task, working in concert with JUnit, is that any tests that I've defined are now automatically run and if any of these tests fail, the build will fail also -- by using the haltonfailure attribute.

    Listing 2. Running a unit test in Ant
    haltonfailure="true" dir="${basedir}" printsummary="yes">                                                  

    Notice that test.unit.dir designates the location of the tests. This is an effective way to segregate these tests (unit, in this case) from other tests. By utilizing this technique, you can run faster tests first, followed by slower tests (such as component, functional, and system tests) by defining additional Ant targets.

    Collecting component tests

    Because unit tests execute so quickly, they are easy to run often as a part of a build. These tests, however, don't achieve a high rate of code coverage -- their isolated nature means they only test a portion of the functionality. Writing tests that reach more code and therefore exercise more functionality usually takes more legwork in the form of adjunct frameworks. Once you start to use these helper frameworks to write tests, these tests start to become higher level ones that I tend to categorize as component tests.

    Component tests are essentially tests that verify more than one class and, typically, rely on external dependencies such as a database. Component tests are written in much the same way as unit tests, except that instead of mocking or stubbing classes to force isolation, these tests bite the bullet, so to speak, and enlist frameworks to facilitate working with external dependencies. For example, I often employ the DbUnit framework to help manage a database so that my component tests can verify the functionality of code that relies on the database's data.

    Controlling database state with DbUnit

    DbUnit is a framework that makes the process of testing against a database much simpler. It provides a standard XML format for defining test data that can be used to select, update, insert, and remove data from a database. Keep in mind, DbUnit doesn't replace a database; it provides a more efficient mechanism for handling test data. With DbUnit, you can write tests that rely on specific data, which DbUnit ensures is present in an underlying database.

    You can use DbUnit programmatically in JUnit or you can utilize it as part of a build process. The framework comes with an Ant task that provides a way to manipulate, export, or compare data in a database using XML files. For example, Listing 3 demonstrates the dbunit task, which, in my case, inserts test data into my target database and then removes the data after having run all my component tests:

    Listing 3. Running a component test in Ant
                  <dbunit driver="com.mysql.jdbc.Driver"        url="jdbc:mysql://localhost:3306/brewery"        userid="${db.username.system}"  	  classpathref="db.lib.path"        password="${db.password.system}">                                                                                                                                                                                                      

    As you can see in Listing 3, my component tests can now rely on specific data residing in the database during their execution. What's more, this process is now repeatable because I've removed all the data after all tests successfully execute.

    Seeding the database

    You can use the INSERT and DELETE operation types for the dbunit task in conjunction with a seed file that contains XML elements representing database tables and associated rows. For example, Listing 4 is the contents of the seedFile.xml file referenced in Listing 3. Each BEER element represents a database table also named BEER and each of the BEER element's attributes and their values map to corresponding database column names and values.

    Listing 4. A DbUnit seed file

    As you may have noticed from Listing 3, you can reuse DbUnit's seed files for different operations. In my case, I used the file in listing 4 to seed the database before running my component tests and then used the same file to indicate which data to delete from the database upon test completion.

    Partaking in performance tests

    Performance testing is often performed (no pun intended) long after developers have finished coding -- yet it's often the case that performance issues could have been found (and most likely solved) much earlier in the development cycle. Luckily, there is a way to solve this problem: continuous testing, or more specifically, continuously running JUnitPerf tests.

    JUnitPerf is perfect for performance testing

    JUnitPerf is a framework that works in concert with JUnit to execute test cases within a predefined time limit: If a method under test takes longer than a desired threshold, the test is considered a failure. By integrating performance tests into an automated build, you can effectively monitor application performance and even fail a build if performance issues pop up.

    I prefer to use JUnitPerf as a simple way to discover early performance problems rather than as a mechanism to measure execution time though; tools like profilers are much more capable at providing this type of measurement. In essence, you can think of JUnitPerf as an early warning system.

    In Listing 5, I define a JUnit test that uses JUnitPerf to verify the execution time of a test called testLongRunningMethod in the BeerServicePerformanceTest test class. If the test method happens to take longer than 1000 milliseconds to execute, it fails.

    Listing 5. A performance-based test using JUnitPerf
    package;  import com.clarkware.junitperf.*;  import junit.framework.Test;    public class ExampleTimedTest {    public static Test suite() {              long maxElapsedTime = 1000;              Test testCase = new BeerServicePerformanceTest("testLongRunningMethod");      Test timedTest = new TimedTest(testCase, maxElapsedTime);              return timedTest;    }  	    public static void main(String[] args) {;    }    }  

    Be careful in using precise timings as a gauge for the execution time of a method; the setup and tear down time of a test is included in the overall execution time. Plus, precise measuring of execution speed is more of an art form than a science when it comes to early performance testing.

    Functional tests with Selenium

    You can write all of the unit and component tests you want, but if you're writing an application that provides a user interface of some type (like a Web application, for example), you'll need to test the presentation layer. In the case of a Web application, you'll want to verify that the navigation of a user scenario and additionally verify that the functionality of the scenario pans out. Until recently, however, this sort of testing has proven to be a burden as typically, you'd need to purchase tools that promoted late cycle testing. What's more, these tools rarely fit into a build process even if the tests were built early enough.

    Enter Selenium

    In the last few years, however, a number of open source tools have sprung up that address functional testing; moreover, you can easily employ these tools early in the development life cycle. Tools such as Selenium and Watir are open source; what's more, they have been built with developers in mind. Besides programmatically defining Selenium tests in various languages (like Java� programming and Python, for example), Selenium also provides an easy to learn table-driven format that can be used by non-technical types as well.

    The Selenium framework uses JavaScript to execute Web-based acceptance tests that open a browser and run table-driven tests. For example, Listing 6 demonstrates an HTML table that represents a simple Selenium test. The various steps of the test open a Web application and then perform a login with a valid username and password. The test's results are generated in an HTML table that can be viewed after Selenium completes running all the tests.

    Listing 6. Functional test with Selenium
    open /beer/
    type username admin
    type password password
    clickAndWait //input[@value='Login']
    verifyTextPresent Logged in as admin

    You can define multiple acceptance tests using the table-based format in Listing 6. You can also group tests into suites and execute an entire suite at a time.

    Driving Selenium through Ant

    What's great about Selenium is that it was created from the ground up with CI in mind because you can run Selenium tests from build tools like Ant. What's more, because of the forward thinking of the framework's designers, if any Selenium acceptance tests fail, you can also fail the entire build. For example, Listing 7 demonstrates an Ant task that uses the Selenium Remote Control server to execute a series of table-driven tests against a Web application:

    Listing 7. Running Selenium with Ant

    When executing Selenium tests, don't be alarmed when the framework opens a Web browser, executes your tests at lightning speed, and then closes the browser and generates an HTML report -- it's one more quick and easy way you can learn about problems early in the development life cycle (when they are easier to fix).

    Code coverage with Cobertura

    To be at 100% or not, that is the question

    When running tools like Cobertura or Emma, it is important to remember one important aspect: Just because you have 100% line coverage in a particular method, it doesn't mean the method is defect free or whether it's been entirely tested. For instance, if you've written a test for an if statement that contains a logical And and your tests exercise the left side of the expression, then a tool like Cobertura will report 100% line coverage, but in reality, you've only exercised 50% of the statement; therefore you've only achieved 50% branch coverage.

    Now that you've written a bunch of tests, how do you determine what all those tests execute? Luckily, answering this question is where a code coverage tool like Cobertura shines. A code coverage tool reports test coverage -- either in the form of line or branch coverage -- that represents the amount of code that is being touched when a test is run.

    Listing 8 demonstrates an Ant script that uses Cobertura to generate an HTML report of the code coverage achieved by running a series of JUnit tests:

    Listing 8. Code coverage using Ant and Cobertura

    Cobertura produces an HTML report like the one in Figure 1. Notice the line and branch coverage percentages by package. You can click on each package to get class level line and path percentages and even see which source code lines were executed and how many times they was run.

    Figure 1. HTML report produced using Cobertura and Ant
    HTML report produced using Cobertura and Ant

    How much code coverage do you need?

    Ideally, you want to have one test for each path. In other words, if your entire code base has a cyclomatic complexity of 20,000, you should have 20,000 tests. I've never run across a project with 100% path coverage, although I've seen teams with close to 100% line coverage.

    You've seen various types of tests and even how to measure the coverage of those tests -- but how do you ensure that the execution of these tests happens at regular intervals? As it happens, this is where a CI server -- like CruiseControl steps up to the plate, as I'll show you next.

    Running tests continually

    Once you've incorporated the execution of these various developer test types into a build process, you can run some (or all) of these tests as part of a CI process. For example, in Listing 9, which is a snippet of CruiseControl's config.xml file, I define a number of things. First, I have CruiseControl monitoring a Subversion repository for any changes every two minutes. If it discovers any changes have occurred, CruiseControl kicks off a delegating build script (typically, you will see this written in Ant) called build-${}.xml. The delegating build script calls the project's build script, which executes a compile and runs any tests.

    I've also defined some logic to merge the results of all my different types of tests into a CruiseControl log file. Furthermore, I'm also using CruiseControl's capability to link (using the artifactspublisher tag) the reports generated by different tools into the Build Artifacts link, which is available from the CruiseControl dashboard application.

    Listing 9. CI with CruiseControl
    ...                                              <artifactspublisher       dir="projects/${}/_reports/"       dest="projects/artifacts/${}"/>    ...  

    You won't necessarily run every test defined with every source change applied to your version control repository. For instance, you can set up your CI system to execute a build that only runs unit tests during code check-ins (which is often referred to as a commit build). You can complement commit builds with more heavy-weight style builds that run component, functional, performance test and even code inspections, for example (Resources). These builds can be run more infrequently (like once a day, for instance). Alternatively, you can run these tests and inspections immediately after your commit build.

    More Load, Stress & Performance Testing Articles
    1 2 3 4 5 Next

    discussionDiscussion Center


    Yahoo Groups
    Y! Group
    Sirfdosti Groups
    Contact Us

    Looking for Software Testing eBooks and Interview Questions? Join now and get it FREE!
    A D V E R T I S E M E N T

    Members Login

    Email ID:

    Forgot Password
    New User
    Testing Interview Questions
  • General Testing
  • Automation Testing
  • Manual Testing
  • Software Development Life Cycle
  • Software Testing Life Cycle
  • Testing Models
  • Automated Testing Tools
  • Silk Test
  • Win Runner
    Testing Highlights

  • Software Testing Ebooks
  • Testing Jobs
  • Testing Frequently Asked Questions
  • Testing News
  • Testing Interview Questions
  • Testing Jobs
  • Testing Companies
  • Testing Job Consultants
  • ISTQB Certification Questions
    Interview Questions

  • WinRunner
  • LoadRunner
  • SilkTest
  • TestDirector
  • General Testing Questions

  • Testing Forum
  • Downloads
  • E-Books
  • Testing Jobs
  • Testing Interview Questions
  • Testing Tools Questions
  • Testing Jobs
  • A-Z Knowledge
    Study ABROAD ?

    Study Abroad

    Vyom Network : Free SMS, GRE, GMAT, MBA | Online Exams | Freshers Jobs | Software Downloads | Programming & Source Codes | Free eBooks | Job Interview Questions | Free Tutorials | Jokes, Songs, Fun | Free Classifieds | Free Recipes | Bangalore Info | GATE Preparation | MBA Preparation | Free SAP Training
    Privacy Policy | Terms and Conditions
    Sitemap | Sitemap (XML)
    Job Interview Questions | Placement Papers | SMS Jokes | C++ Interview Questions | C Interview Questions | Web Hosting
    German | French | Portugese | Italian