|
Introduction
Microsoft Web Capacity Analysis Tool (WCAT) is a free lightweight HTTP load generation tool, part of IIS 6.0 Resource toolkit that is used to test performance and scalability of IIS and ASP.NET application. This tool is used internally by IIS team and NT Performance team use to conduct internal performance and scalability testing.
Visual Studio Team System offers a nice way with GUI to stress test web applications which is very easy to use and has its advantages but might consider WCAT in following cases:
� looking for a free tool without compromising the features;
� tool for IT Pros.
� tool that can be used in Continuous Integration (CI) setup;
� granular transactions setup (setting several URL usually executed together by users).
As WCAT learning curve is not very flat and the tool seems complicated at the beginning you should consider Visual Studio Team System for quick and easy load test.
This article aims at including only essential information to run WCAT test. For more detailed information please refer to WCAT Documentation [2].
Configuration samples in this article apply to WCAT v6.3.
Stress testing basics
Stress testing (or Load testing) is process of creating a demand on service or a device and measuring its response [4].
In order to make testing more accurate additional information about the system should be collected as user activities, think times, usage patterns [5].
A couple ways can be applied on stress testing depending on your approach:
� Test existing production configuration � There is a production environment and the business wants to know how long current configuration will be reliable having in mind trends of the users. In this case is recommended to have a staging environment which should be as close to production as possible. The stress test will determine max capabilities of existing environment (hardware and software) and would help where to put resources (e.g. buying more hardware or optimizing some application components).
� Plan production environment � in this case new product should be released and according marketing researches production environment is planned. In this case different configurations are considered in order to achieve best ROI (Return on investment) on hardware and best performance and scalability. (For example these scenarios include from having the application on single machine to separate major components on different machines to spread the load ) This approach will give a clear idea what kind of hardware and configuration is needed to host required load as well as max load for specific hardware/configuration.
WCAT Components
In order to start a load test against web application several components are needed. More or less same components are needed for all stress test tools.
For big commercial websites it is recommended to have a separate machine for each component or you want to squeeze the component as much as possible.
Web Server
As this article discuss web application testing the web server is the very vital part of the test. This is the machine which performance will be tested. While narrow definition would include simple web site hosted in Internet Information Services (IIS), wider one would include all components and connected systems that make one application useful to the customer (or client applications if it is a service).
Database server (optional)
Database server could be considered as a part of the web server component (described above) but for clarifying purposes it is explicitly described. In many cases the web application response is in acceptable range but with time it goes outside this range. This is because the data volume accumulated during the time. Having database on back-end of web application should be load tested as well as it could be the bottleneck in the system.
Client
The client is a machine(s) that is responsible for making many requests to the web server. This component generates HTTP traffic directly against web server. An ideal WCAT test environment will contain an adequate number of WCAT Client machines such that the CPU or disk of the Web Server can be fully saturated. Selecting the right number of WCAT Client machines is a chicken-and-egg type of problem. The number of WCAT Client machines required will depend on the capacity of the web server hardware and the resource requirements of the web site content. However, the capacity of the Web Server is unknown until WCAT is run.
WCAT Clients and Virtual Clients
In a "real world" web server, HTTP requests are generated from potentially thousands of unique physical machines. WCAT is designed to simulate multiple unique users concurrently making requests to a web server. WCAT uses the notion of a "physical client" and a "virtual client". A physical client is a real computer running one instance of the WCAT Client process. Each process can simulate thousands of users. These users are called "virtual clients". When WCAT is executed, it accepts a parameter "-virtualclients", which specifies the number of concurrent users to simulate from each physical client. For example, if a test environment has 10 physical clients and the test specifies 8 virtual clients per physical client, then the web server will receive 80 concurrent connections [2].
For more information see "Determining the Number of WCAT Client Machines" in Section 3. Setting Up a Test Environment in WCAT Documentation [2].
Controller
The controller send signals to clients when to start sending HTTP load to the server as well as feeds clients with test data (like URL, how often to send the load etc.). The controller is responsible for manage the clients and collecting performance counters from web server. It is possible to use different performance measurement and in this case you won't need communication between the controller and web server � see see 'Setup environment' (also this will invalidate some sections of the report).
Load Test Configuration
Environment
Having clients and a controller makes us asking the question "Where should I host them?" Here are some scenarios [2]:
Single Machine Environment
The single machine environment is the simplest test environment. It is also the cheapest. It comprises of a single machine with all four of the basic software components running concurrently. The web server content is local along with a database if necessary. WCAT controller and client software are run on the web server machine itself.
WARNING: This configuration is not recommended. The drawback to a single machine environment is that the database and WCAT both consume significant resources on the server.
Dual Machine Environment
The dual machine environment is the simplest environment that provides accurate measurements of a web server's performance capacity. It consists of a single dedicated web server and another machine that runs the rest of the software components. The drawback to this environment is that the resources on the WCAT machine need to be monitored to ensure that it is not bottlenecking the system.
Multiple Machines Isolated Environment
The multiple machines, isolated environment is the recommended configuration for enterprise and commercial capacity testing. The network isolation guarantees that external factors such as random network traffic cannot affect the results of tests.
This is just a short description of possible scenarios. For more detailed description see Section 3. Setting Up a Test Environment in WCAT Documentation [2].
Stages
The load test has three main stages [2]:
Warm Up
WCAT uses a "warm-up" period in order to allow the Web Server to achieve steady state before taking measurements of throughput, response time and performance counters. For instance there is a slight delay on first request on ASP.NET sites when Just-In-Time (JIT) compilation is performed.
WCAT will divide the warm-up phase into two parts. For the first half of the warm-up period WCAT will slowly add virtual clients until all virtual clients have been activated. The second half is pure load generation.
Actual test
In this stage the load data is sent to the server and the performance is measured.
Cool down
In order to ensure that measurements end before load generation ends a cool down period in seconds must be specified. Recommended setting is 20 seconds.
Settings
The following settings can be set in settings.ubr flat file or passed as parameters. Here is sample settings file with most interesting settings.
Example WCAT Settings File
settings
{
clientfile = "home.ubr";
server = "192.168.1.113";
clients = 10;
virtualclients = 200;
counters
{
interval = 10;
counter = "Processor(_Total)\\% Processor Time";
counter = "Processor(_Total)\\% Privileged Time";
counter = "Processor(_Total)\\% User Time";
counter = "Processor(_Total)\\Interrupts/sec";
counter = "Memory\\Available KBytes";
counter = "Process(w3wp)\\Working Set";
counter = "System\\Context Switches/sec";
counter = "System\\System Calls/sec";
counter = "Web Service(_Total)\\Bytes Received/sec" ;
counter = "ASP.NET Applications(__Total__)\\Anonymous Requests/Sec" ;
counter = "ASP.NET Applications(__Total__)\\Request Execution Time" ;
counter = "ASP.NET\\Request Wait Time" ;
counter = "ASP.NET\\Requests Queued" ;
counter = "ASP.NET\\Requests Rejected" ;
}
registry
{
path = "System\\CurrentControlSet\\Services\\Tcpip\\Parameters";
name = "SynAttackProtect";
type = REG_DWORD;
}
}
� Client file � a relative path to a flat file that describe test scenario.
� Server � a name or IP address of the web server.
� Clients � number of physical clients that will sent HTTP load to the server.
� Virtual Clients - number of 'threads' per physical client.
� Counters � section with performance counters that will be checked
o Interval � interval in seconds that shows how often performance counters values are pulled
o Counter � multiple rows with the name and category of the performance counter.
� Registry � monitor registry value
o Path � path to registry node
o Name � name of the value
o Type � type of the value. Only simple strings and dwords are all that are currently supported
Note: When WCAT is installed there are plenty of comments inside sample settings file.
Scenario
The scenario file consists of three main pieces. The first is the default request element. This describes attributes to be used as defaults for all requests defined in the scenario file. Examples of typical settings include the default HTTP headers, HTTP version, expected status code and connection close/keep-alive behavior. The second section (which is optional) is the library section. The final piece of the scenario file is all of the transactions to be executed by WCAT Client. Transactions have weights and are chosen randomly according to the weight given to them. Typically a transaction consists of a linear list of HTTP requests to execute against the target web server. A transaction can also contain other "actions" such as connection closes, sleeps and branches within the current transaction. [2]
Example WCAT Scenario File
scenario
{
name = "Galcho.com Home Page";
warmup = 30;
duration = 120;
cooldown = 10;
default
{
// send keep-alive header
setheader
{
name = "Connection";
value = "keep-alive";
}
// set the host header
setheader
{
name = "Host";
value = server();
}
// HTTP1.1 request
version = HTTP11;
// keep the connection alive after the request
close = ka;
}
transaction
{
id = "Default Web Site Homepage";
weight = 1;
request
{
url = "/default.aspx";
}
request
{
url = "/blog";
}
//
// specifically close the connection after both files are requested
//
close
{
method = reset;
}
}
transaction
{
id = "Articles";
weight = 1;
request
{
url = "/articles";
verb = "POST";
}
close
{
method = reset;
}
}
}
� name � name of the test/scenario.
� warmup � length in seconds for warm-up stage.
� duration � length in seconds for duration of the actual test.
� cooldown - length in seconds for cool down period.
� default � set default settings for all request. All settings can be overridden if specified on request element
o version � version of HTTP � HTTP10/HTTP11
o close-close connection after the request
o setheader � set value to HTTP Header.
� Name � name of the header
� Value � value - WCAT functions can be used.
� transaction � set of request to be executed.
o Id � unique name of transaction
o Weight � integer number that shows how often given transaction will be executed during load test. WCAT calculates the correct percentage dynamically relative to the total weight of all transactions in a scenario.
o Request � describe single request.
� url � relative or absolute URL address to the resource.
� Verb � HTTP verb � (GET, POST). "HEAD" request is not supported.
� Postdata � data to be sent to the server
� And many others� see the documentation [2]
Best Practices
1. Run tests on an isolated network.
2. Check that load generator machines are not CPU, memory or network bound.
3. Always test the web server in a known good state (i.e. record all changes to the web server such as registry changes, installed software, etc�).
4. Run multiple iterations of the test to validate that variance in results is low. If the standard deviation of requests/second between consecutive tests is greater than 2% then increase run time and warm up time.
5. Validate that the warm up period in WCAT tests is sufficient. Web Servers can take multiple minutes to fully populate all relevant caches. Experiment with larger warm up periods until no difference in throughput occurs.
6. Check the results of WCAT to ensure that no unexpected errors occurred. Ideally, the number of errors will be zero, or potentially only a fraction of 1% [2].
Setup a Load Test
Setup environment
A very important thing is using a synchronized Windows account (an account with the same username / same password) on all machines that host clients and on the controller. Having same account on web server would allow collecting information from performance counters. If this is set the rest is easy.
Setup clients
Clients requires a couple registry settings that should be made
HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort = 0xfffe
HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\TcpTimedWaitDelay = 0x2
Run the test
Here are the steps for running WCAT Load Test successfully. If on wcclient.exe �b switch is used the order doesn't matter
1. Install WCAT 6.3
2. Prepare scenario (home.ubr) and settings (settings.ubr) files
3. Run the controller with following command
cscript "C:\Program Files\wcat\wcat.wsf" -x -run -clients WCATClientMachineName -f E:\Projects\IIS\WCAT\galchocom\settings.ubr -x �singleip
WCATClientMachineName � this is the comma separated names or IPs of clients. (win2003test, 192.168.1.1).
4. Run clients one by one:
wcclient.exe controllerName �b
controllerName � the name or IP of the machine that host the controller
-b - switch that let client to wait for controller's command.
Wait until the execution completes and look for log.xml file in current folder.
Summary
Web Capacity Analysis Tool (WCAT) is a very lightweight and yet powerful tool that can be used from IT professionals and developers. Its rich settings can be used for most cases. Its biggest disadvantage is lack of GUI to create and manage load test scenarios.
WACT is extremely powerful tool that doesn't need any installation beside XCOPY to test and adjust web server environment.
|
