OneStopTesting - Quality Testing Jobs, eBooks, Articles, FAQs, Training Institutes, Testing Software, Testing downloads, testing news, testing tools, learn testing, manual testing, automated testing, load runner, winrunner, test director, silk test, STLC

Forum| Contact Us| Testimonials| Sitemap| Employee Referrals| News| Articles| Feedback| Enquiry
 
Testing Resources
 
  • Testing Articles
  • Testing Books
  • Testing Certification
  • Testing FAQs
  • Testing Downloads
  • Testing Interview Questions
  • Career In Software Testing
  • Testing Jobs
  • Testing Job Consultants
  • Testing News
  • Testing Training Institutes
  •  
    Fundamentals
     
  • Introduction
  • Designing Test Cases
  • Developing Test Cases
  • Writing Test Cases
  • Test Case Templates
  • Purpose
  • What Is a Good Test Case?
  • Test Specifications
  • UML
  • Scenario Testing
  • Test Script
  • Test Summary Report
  • Test Data
  • Defect Tracking
  •  
    Software testing
     
  • Testing Forum
  • Introduction
  • Testing Start Process
  • Testing Stop Process
  • Testing Strategy
  • Risk Analysis
  • Software Listings
  • Test Metrics
  • Release Life Cycle
  • Interoperability Testing
  • Extreme Programming
  • Cyclomatic Complexity
  • Equivalence Partitioning
  • Error Guessing
  • Boundary Value Analysis
  • Traceability Matrix
  •  
    SDLC Models
     
  • Introduction
  • Waterfall Model
  • Iterative Model
  • V-Model
  • Spiral Model
  • Big Bang Model
  • RAD Model
  • Prototyping Model
  •  
    Software Testing Types
     
  • Static Testing
  • Dynamic Testing
  • Blackbox Testing
  • Whitebox Testing
  • Unit Testing
  • Requirements Testing
  • Regression Testing
  • Error Handling Testing
  • Manual support Testing
  • Intersystem Testing
  • Control Testing
  • Parallel Testing
  • Volume Testing
  • Stress Testing
  • Performance Testing
  • Agile Testing
  • Localization Testing
  • Globalization Testing
  • Internationalization Testing
  •  
    Test Plan
     
  • Introduction
  • Test Plan Development
  • Test Plan Template
  • Regional Differences
  • Criticism
  • Hardware Development
  • IEEE 829-1998
  • Testing Without a TestPlan
  •  
    Code Coverage
     
  • Introduction
  • Measures
  • Working
  • Statement Coverage
  • Branch Coverage
  • Path Coverage
  • Coverage criteria
  • Code coverage in practice
  • Tools
  • Features
  •  
    Quality Management
     
  • Introduction
  • Components
  • Capability Maturity Model
  • CMMI
  • Six Sigma
  •  
    Project Management
     
  • Introduction
  • PM Activities
  • Project Control Variables
  • PM Methodology
  • PM Phases
  • PM Templates
  • Agile PM
  •  
    Automated Testing Tools
     
  • Quick Test Professional
  • WinRunner
  • LoadRunner
  • Test Director
  • Silk Test
  • Test Partner
  • Rational Robot
  •  
    Performance Testing Tools
     
  • Apache JMeter
  • Rational Performance Tester
  • LoadRunner
  • NeoLoad
  • WAPT
  • WebLOAD
  • Loadster
  • OpenSTA
  • LoadUI
  • Appvance
  • Loadstorm
  • LoadImpact
  • QEngine
  • Httperf
  • CloudTest
  •  
    Languages
     
  • Perl Testing
  • Python Testing
  • JUnit Testing
  • Unix Shell Scripting
  •  
    Automation Framework
     
  • Introduction
  • Keyword-driven Testing
  • Data-driven Testing
  •  
    Configuration Management
     
  • History
  • What is CM?
  • Meaning of CM
  • Graphically Representation
  • Traditional CM
  • CM Activities
  • Tools
  •  
    Articles
     
  • What Is Software Testing?
  • Effective Defect Reports
  • Software Security
  • Tracking Defects
  • Bug Report
  • Web Testing
  • Exploratory Testing
  • Good Test Case
  • Write a Test
  • Code Coverage
  • WinRunner vs. QuickTest
  • Web Testing Tools
  • Automated Testing
  • Testing Estimation Process
  • Quality Assurance
  • The Interview Guide
  • Upgrade Path Testing
  • Priority and Severity of Bug
  • Three Questions About Bug
  •    
     
    Home » Testing Articles » Manual Testing Articles » Agile Testing

    Agile Testing

    A D V E R T I S E M E N T


    This post was inspired by an article I read in the Feb. 2005 issue of Better Software: "Double Duty" by Brian Button. The title refers to having unit tests serve the double role of testing and documentation. Brian calls this Agile Documentation. For Python developers, this is old news, since the doctest module already provides what is called "literate testing" or "executable documentation". However, Brian also introduces some concepts that I think are worth exploring: Test Lists and Tests Maps.

    Test Lists

    A Test List tells a story about the behavior expected from the module/class under test. It is composed of one-liners, each line describing what a specific unit test tries to achieve. For example, in the case of a Blog management application, you could have the following (incomplete) Test List:

    • Deleting all entries results in no entries in the blog.
    • Posting single entry results in single valid entry.
    • Deleting a single entry by index results in no entries in the blog.
    • Posting new entry results in valid entry and increases the number of entries by 1.
    • Etc.

    I find it very valuable to have such a Test List for every Python module that I write, especially if the list is easy to generate from the unit tests that I write. I will show later in this post how the combination of doctest and epydoc makes it trivial to achieve this goal.

    Test Maps

    A Test Map is a list of unit tests associated with a specific function/method under test. It helps you see how that specific function/method is being exercised via unit tests. A Test Map could look like this:

    Testmap for method delete_all_entries:

    • test_delete_all_entries
    • test_delete_single_entry
    • test_post_single_entry
    • test_post_two_entries
    • test_delete_first_of_two_entries
    • test_delete_second_of_two_entries

    Generating Test Lists

    As an example of a module under test, I will use the Blog management application that I discussed in several previous posts. The source code can be found here. I have a directory called blogmgmt which contains a module called blogger.py. The blogger module contains several classes, the main one being Blogger, and a top-level function called get_blog. I also created an empty __init__.py file, so that blogmgmt can be treated as a package. I wrote a series of doctest-based tests for the blogger module in a file I called testlist_blogger.py. Here is part of that file:



    """
    Doctest unit tests for module L{blogger}
    """

    def test_get_blog():
    """
    get_blog() mimics a singleton by always returning the same object.

    Function(s) tested:
    - L{blogger.get_blog}


    >>> from blogger import get_blog
    >>> blog1 = get_blog()
    >>> blog2 = get_blog()
    >>> id(blog1) == id(blog2)
    True

    """

    def test_get_feed_title():
    """
    Can retrieve the feed title.

    Method(s) tested:
    - L{blogger.Blogger.get_title}

    >>> from blogger import get_blog
    >>> blog = get_blog()
    >>> print blog.get_title()
    fitnessetesting

    """

    def test_delete_all_entries():
    """
    Deleting all entries results in no entries in the blog.

    Method(s) tested:
    - L{blogger.Blogger.delete_all_entries}
    - L{blogger.Blogger.get_num_entries}

    >>> from blogger import get_blog
    >>> blog = get_blog()
    >>> blog.delete_all_entries()
    >>> print blog.get_num_entries()
    0

    """

    def test_post_new_entry():
    """
    Posting new entry results in valid entry and increases the number of entries by 1.

    Method(s) tested:
    - L{blogger.Blogger.post_new_entry}
    - L{blogger.Blogger.get_nth_entry_title}
    - L{blogger.Blogger.get_nth_entry_content_strip_html}
    - L{blogger.Blogger.get_num_entries}

    >>> from blogger import get_blog
    >>> blog = get_blog()
    >>> init_num_entries = blog.get_num_entries()
    >>> rc = blog.post_new_entry("Test title", "Test content")
    >>> print rc
    True
    >>> print blog.get_nth_entry_title(1)
    Test title
    >>> print blog.get_nth_entry_content_strip_html(1)
    Test content
    >>> num_entries = blog.get_num_entries()
    >>> num_entries == init_num_entries + 1
    True

    """

    Each unit test function is composed of a docstring and nothing else. The docstring starts with a one-line description of what the unit test tries to achieve. The docstring continues with a list of methods/functions tested by that unit test. Finally, the interactive shell session output is copied and pasted into the docstring so that it can be processed by doctest.

    For the purpose of generating a Test List, only the first line in each docstring is important. If you simply run

    epydoc -o blogmgmt testlist_blogger.py

    you will get a directory called blogmgmt that contains the epydoc-generated documentation. I usually then move this directory somewhere under the DocumentRoot of one of my Apache Virtual Servers. When viewed in a browser, this is what the epydoc page for the summary of the testlist_blogger module looks like this (also available here):

    Module blogmgmt.testlist_blogger

    Doctest unit tests for module blogger


    Function Summary


    test_delete_all_entries()
    Deleting all entries results in no entries in the blog.


    test_delete_first_of_two_entries()
    Posting two entries and deleting entry with index 1 leaves oldest entry in place.


    test_delete_second_of_two_entries()
    Posting two entries and deleting entry with index 2 leaves newest entry in place.


    test_delete_single_entry()
    Deleting a single entry by index results in no entries in the blog.


    test_get_blog()
    get_blog() mimics a singleton by always returning the same object.


    test_get_feed_posting_host()
    Can retrieve the feed posting host.


    test_get_feed_posting_url()
    Can retrieve the feed posting URL.


    test_get_feed_title()
    Can retrieve the feed title.


    test_post_new_entry()
    Posting new entry results in valid entry and increases the number of entries by 1.


    test_post_single_entry()
    Posting single entry results in single valid entry.


    test_post_two_entries()
    Posting two entries results in 2 valid entries ordered most recent first.


    This is exactly the Test List we wanted. Note that epydoc dutifully generated it for us, since in the Function Summary section it shows the name of every function it finds, plus the first line of that function's docstring. The main value of this Test List for me is that anybody can see at a glance what the methods of the Blogger class are expected to do. It's a nice summary of expected class behavior that enhances the documentation.

    So all you need to do to get a nicely formatted Test List is to make sure that you have the test description as the first line of the unit test's docstring; epydoc will then do the grungy work for you.

    If you click on the link with the function name on it, you will go to the Function Detail section and witness the power of doctest/epydoc. Since all the tests are copied and pasted from an interactive session and included in the docstring, epydoc will format the docstring very nicely and it will even color-code the blocks of code. Here is an example of the detail for test_delete_all_entries.

    Generating Test Maps

    Each docstring in the testlist_blogger module contains lines such as these:


    Method(s) tested:
    - L{blogger.Blogger.post_new_entry}
    - L{blogger.Blogger.get_nth_entry_title}
    - L{blogger.Blogger.get_nth_entry_content_strip_html}
    - L{blogger.Blogger.get_num_entries}

    (the L{...} notation is epydoc-specific and represents a link to another object in the epydoc-generated documentation)

    The way I wrote the unit tests, each of them actually exercises several functions/methods from the blogger module. Some unit test purists might think these are not "real" unit tests, but in practice I found it is easier to work this way. For example, the get_blog function is called by each and every unit test in order to retrieve the same "blog" object. However, I am not specifically testing get_blog in every unit test, only calling it as a helper function. The way I see it, a method is tested when there is an assertion made about its behavior. All the other methods are merely called as helpers.

    So whenever I write a unit test, I manually specify the list of methods/functions under test. This makes it easy to then parse the testlist file and build a mapping from each function/method under test to a list of unit tests that test it, i.e. what we called the Test Map.

    For example, in the testlist_blogger module, the Blogger.delete_all_entries method is listed in the docstrings of 6 unit tests: test_delete_all_entries, test_delete_single_entry, test_post_single_entry, test_post_two_entries, test_delete_first_of_two_entries, test_delete_second_of_two_entries. These 6 unit test represent the Test Map for Blogger.delete_all_entries. It's easy to build the Test Map programatically by parsing the testlist_blogger.py file and creating a Python dictionary having the methods under tests as keys and the unit test lists corresponding to them as values.

    An issue I had while putting this together was how to link a method in the Blogger class (for example Blogger.delete_all_entries) to its Test Map. One way would have been to programatically insert the Test Map into the docstring for that method. But this would mean that every time a new unit test is added that tests that method, the Test Map will change and thus the module containing the Blogger class will get changed. This is unfortunate especially when the files are under source control. I think a better solution, and the one I ended up implementing, is to have a third module called for example testmap_blogger that will be automatically generated from testlist_blogger. A method M in the Blogger class will then link to a single function in testmap_blogger. That function will contain in its docstring the Test Map for the Blogger method M.

    Again, an example to make all this clearer. Here is the docstring of the Blogger.delete_all_entries method in the blogger module:

    """
    Delete all entries in the blog

    Test map (set of unit tests that exercise this method):
    - L{testmap_blogger.testmap_Blogger_delete_all_entries}
    """


    Here is the epydoc-generated documentation for the Blogger.delete_all_entries method (in the Method Details section):

    delete_all_entries(self)

    Delete all entries in the blog

    Test map (set of unit tests that exercise this method):


    I manually inserted in the docstring an epydoc link to a function called testmap_Blogger_delete_all_entries in a module called testmap_blogger. Assuming that the testmap_blogger module was already generated and epydoc-documented, clicking on the link will bring up the epydoc detail for that particular function, which contains the 6 unit tests for te delete_all_entries method:

    testmap_Blogger_delete_all_entries()

    Testmap for blogger.Blogger.delete_all_entries:

    Here is the programatically-generated testmap_blogger.py file.

    To have all this mechanism work, I use some naming conventions:

    • The module containing the Test Maps for module blogger is called testmap_blogger
    • In testmap_blogger, the function containing the Test Map for method Blogger.M from the blogger module is called testmap_Blogger_M
    • In testmap_blogger, the function containing the Test Map for function F from the blogger module is called testmap_F
    • In the docstring of the testmap function itself there is a link which points back to the method Blogger.M; the name of the link needs to be blogger.Blogger.M, otherwise epydoc will not find it

    Here's an end-to-end procedure for using the doctest/epydoc combination to write Agile Documentation:

    1. We'll unit test a Python module we'll call P which contains a class C.

    2. We start by writing a unit test for the method C.M1 from the P module. We write the unit test by copying and pasting a Python shell session output in another Python module called testlist_P. We call the unit test function test_M1. It looks something like this:

    def test_M1():
    """
    Short description of the behavior we're testing for M1.

    Method(s) tested:
    - L{P.C.M1}

    >>> from P import C
    >>> c = C()
    >>> rc = c.M1()
    >>> print rc
    True

    """

    The testlist_P module has a "main" section of the form:

    if __name__ == "__main__":
    import doctest
    doctest.testmod()

    This is the typical doctest way of running unit tests. To actually execute the tests, we need to run "python testlist_P.py" at a command line (for more details on doctest, see a previous blog post).

    3. At this point, we fleshed out an initial implementation for method M1 in module P. In its docstring, we add a link to the test map:

    def M1(self):
    """
    Short description of M1

    Test map (set of unit tests that exercise this method):
    - L{testmap_P.testmap_C_M1}

    """

    Note that I followed the naming convention I described earlier.

    4. We programatically generate the Test Map for module P by running something like this: build_testmap.py. It will create a file called testmap_P.py with the following content:

    def testmap_C_M1():
    """
    Testmap for L{P.C.M1}:

    - L{testlist_P.test_M1}
    """


    5. We run epydoc:

    epydoc -o P_docs P.py testlist_P.py testmap_P.py

    A directory called P_docs will be generated; we can move this directory to a public area of our Web server and thus make the documentation available online. When we click on the testlist_P
    module link, we will see the Test List for module P. It will show something like:

    Module P_docs.testlist_P

    Doctest unit tests for module P


    Function Summary


    test_M1()
    Short description of the behavior we're testing for M1.


    When we click on the test map link inside the docstring of method C.M1, we see:

    testmap_C_M1()

    Testmap for P.C.M1:


    Now repeat steps 2-5 for method M2:

    6. Let's assume we now unit test method M2, but in the process we also test method M1. The function test_M2 will look something like this:

    def test_M2():
    """
    Short description of the behavior we're testing for M2.

    Method(s) tested:
    - L{P.C.M1}
    - L{P.C.M2}

    >>> from P import C
    >>> c = C()
    >>> rc = c.M1()
    >>> print rc
    True
    >>> rc = c.M2()
    >>> print rc
    True

    """

    We listed both methods in the "Method(s) tested" section.

    7. We add a link to the testmap in method M2's docstring (in module P):

    def M2(self):
    """
    Short description of M2

    Test map (set of unit tests that exercise this method):
    - L{testmap_P.testmap_C_M2}
    """

    8. We recreate the testmap_P file by running build_testmap.py. The testmap for M1 will now contain 2 functions: test_M1 and test_M2, while the testmap for M2 will contain test_M2:

    def testmap_C_M1():
    """
    Testmap for L{P.C.M1}:

    - L{testlist_P.test_M1}
    - L{testlist_P.test_M2}
    """

    def testmap_C_M2():
    """
    Testmap for L{P.C.M2}:

    - L{testlist_P.test_M2}
    """


    9. We run epydoc again:

    epydoc -o P_docs P.py testlist_P.py testmap_P.py

    Now clicking on testlist_P will show:

    Module P_docs.testlist_P

    Doctest unit tests for module P


    Function Summary


    test_M1()
    Short description of the behavior we're testing for M1.


    test_M2()
    Short description of the behavior we're testing for M2.


    Clicking on the test map link inside the docstring of method C.M1 shows:

    testmap_C_M1()

    Testmap for P.C.M1:

    10. Repeat steps 2-5 for each unit test that you add to the testlist_P module.



    More Manual Testing Articles



    discussionDiscussion Center
    Discuss
    Discuss

    Query

    Feedback
    Yahoo Groups
    Y! Group
    Sirfdosti Groups
    Sirfdosti
    Contact Us
    Contact

    Looking for Software Testing eBooks and Interview Questions? Join now and get it FREE!
     
    A D V E R T I S E M E N T
       
       

    Members Login


    Email ID:
    Password:


    Forgot Password
    New User
       
       
    Testing Interview Questions
  • General Testing
  • Automation Testing
  • Manual Testing
  • Software Development Life Cycle
  • Software Testing Life Cycle
  • Testing Models
  • Automated Testing Tools
  • Silk Test
  • Win Runner
  •    
       
    Testing Highlights

  • Software Testing Ebooks
  • Testing Jobs
  • Testing Frequently Asked Questions
  • Testing News
  • Testing Interview Questions
  • Testing Jobs
  • Testing Companies
  • Testing Job Consultants
  • ISTQB Certification Questions
  •    
       
    Interview Questions

  • WinRunner
  • LoadRunner
  • SilkTest
  • TestDirector
  • General Testing Questions
  •    
       
    Resources

  • Testing Forum
  • Downloads
  • E-Books
  • Testing Jobs
  • Testing Interview Questions
  • Testing Tools Questions
  • Testing Jobs
  • A-Z Knowledge
  •    
    Planning
    for
    Study ABROAD ?


    Study Abroad


    Vyom Network : Free SMS, GRE, GMAT, MBA | Online Exams | Freshers Jobs | Software Downloads | Programming & Source Codes | Free eBooks | Job Interview Questions | Free Tutorials | Jokes, Songs, Fun | Free Classifieds | Free Recipes | Bangalore Info | GATE Preparation | MBA Preparation | Free SAP Training
    Privacy Policy | Terms and Conditions
    Sitemap | Sitemap (XML)
    Job Interview Questions | Placement Papers | SMS Jokes | C++ Interview Questions | C Interview Questions | Web Hosting
    German | French | Portugese | Italian