Load Testing with JMETER - introduction

Load testing is the process of applying load to an application to see if it can perform as intended under normal conditions.

It is normally done with specialized tools like Load Runner or JMeter.

This type of testing is much more complex than manual testing or even test automation as it requires very diverse knowledge and skills.

The performance tester should have good understanding of not only the application under test but also of 

- http protocol, web requests and responses

- server configuration and monitoring

- scripting

- regular expressions

- log parsing

- application analytics

etc


Some performance testing theory is useful to see the big picture.

The following info has been borrowed from the following free, online resource:
Performance Testing Guidance for Web Applications


Purpose of performance testing


  • Apply normal load to an application to see if it can perform as intended under normal   conditions
  • Ensures that a given function, program, or system can simply handle what it’s designed to handle
  • Related to Stress Testing: overload things until they break, applying unrealistic or unlikely load scenarios
  • Measures:

1. response times

2. throughput rates

3. resource-utilization levels

4. identify your application’s breaking point


 Approach for performance testing

1.Identify the Test Environment


    • Identify the physical test environment and the production environment as well as the tools and resources available to the test team.
    • Physical environment:
      • Hardware
      • Software
      • Network configurations


2. Identify Performance Acceptance Criteria


      • the response time (user concern)
      • throughput (business concern)
      • resource utilization goals and constraints (system concern)


3. Plan and Design Tests


    • key scenarios
    • determine variability among representative users
    • how to simulate that variability
    • define test data
    • establish metrics to be collected. 


4. Configure the Test Environment, Tools and Resources

Ensure that the test environment is instrumented for resource monitoring.


5. Implement the Test Design

Develop the performance tests in accordance with the test design.


6. Execute the Test


    • Run and monitor your tests
    • Validate the tests, test data, and results collection
    • Execute validated tests while monitoring the test and the test environment.


7. Analyse Results, Report, and Retest

    • Consolidate and share results data
    • Analyse the data both individually and as a cross-functional team
    • Re-prioritize the remaining tests and re-execute them as needed
    • When all of the metric values are within accepted limits, none of the set thresholds have been violated, and all of the desired information has been collected, you have finished testing that particular scenario on that particular configuration.


Why Do Performance Testing?


  • Assessing release readiness
    • Helps estimating the performance characteristics of an application
    • Helps determining if an application is capable of handling future growth
  • Providing data indicating the likelihood of user dissatisfaction with the performance characteristics of the system.
  • Assessing infrastructure adequacy
    • Evaluating the adequacy of current capacity
    • Determining the acceptability of stability
    • Determining the capacity of the application’s infrastructure, as well as determining the future resources required
  • Assessing adequacy of developed software performance
    • Determine the application’s desired performance characteristics before and after changes to the software.
    • Provide comparisons between the application’s current and desired performance characteristics.
  • Improving the efficiency of performance tuning
    • Analyzing the behavior of the application at various load levels
    • Identifying bottlenecks in the application
    • Providing information related to the speed, scalability, and stability of a product prior to production release

Baselines


    • Process of running a set of tests to capture performance metric data for the purpose of evaluating the effectiveness of subsequent performance-improving changes to the system or application. 
    • A critical aspect of a baseline is that all characteristics and configuration options except those specifically being varied for comparison must remain invariant.
    • A baseline can be created for a system, component, or application. 
    • A baseline can set the standard for comparison, to track future optimizations or regressions.  



The load testing tools are operating differently from browsers and test automation tools.

They are 

- not browsers

- do not perform all actions supported by browsers

- do not render html pages

- do not execute javascript code included in the html pages

- work at the protocol level (http protocol for web sites), submit web requests to the web server and then process the response from the server.


Most of the load testing tools work for multiple protocols so they can test

- web sites

- database systems

- web services

- client-server apps


Usually, the following components will be found in a load testing environment:

- target machine that hosts the application under test


- master load testing server (load controller): generates the load requests that either send them directly to the target machine or sends them to the slave load testing servers


- slave load testing servers (load generator): get the web requests from the master load testing server and send them to the target machine





Load Runner environment: master (controller) server, slave (load generators) servers, target servers



Finally, a bit of Http protocol theory is needed to explain briefly the concept of http transaction, request, response header and response body.


  • HTTP uses the client-server model: An HTTP client opens a connection and sends a request message to an HTTP server; the server then returns a response message, usually containing the resource that was requested. After delivering the response, the server closes the connection (making HTTP a stateless protocol, i.e. not maintaining any connection information between transactions).
  • The format of the request and response messages are similar, and English-oriented. Both kinds of messages consist of:

    • an initial line,
    • zero or more header lines,
    • a blank line (i.e. a CRLF by itself), and
    • an optional message body (e.g. a file, or query data, or query output).
  • Put another way, the format of an HTTP message is:
       <initial line, different for request vs. response>
       Header1: value1
       Header2: value2
       Header3: value3

       <optional message body goes here, like file contents or query data;
      it can be many lines long, or even binary data $&*%@!^$@>

The above info is borrowed from http://www.jmarshall.com/easy/http/
Please read more about the HTTP protocol on the hosting site.


Having discussed about all of the above, we can start looking at JMETER.


Dynamic WHERE clauses for SQL queries

Let's assume that the following example query is needed for getting the number or orders and the sum of order values from some tables:

select count(orders.orderid), sum(orders.value)
from orders
join customers on orders.customerid = customers.customerid
join stores on orders.storeid = stores.storeid
join cities on customers.cityid = cities.cityid
join products on orders.productid = products.productid
where orders.value > 0
and stores.type = 'wholesale'
and customers.id is not null
and cities.name is not null
and products.type = 'fruit'
and orders.value > 1000000

The query uses only data from wholesale stores, fruit products and orders with value greater than 1000000.

Since it is possible that this query will run for different parameter values, let's change it in a stored procedure:

create procedure getInfo
@StoreType as varchar(50),
@ProductType as varchar(50)

as

begin

select count(*), sum(value)
from orders
join customers on orders.customerid = customers.customerid
join stores on orders.storeid = stores.storeid
join cities on customers.cityid = cities.cityid
join products on orders.productid = products.productid
where orders.value > 0
and stores.type = @StoreType
and customers.id is not null
and cities.name is not null
and products.type = @ProductType
and orders.value > 1000000

end

Having the stored procedure, it is very easy to run it

declare @StoreType as varchar(50) = 'wholesale'
declare @ProductType as varchar(50) = 'fruit'

exec getInfo @StoreType, @ProductType



Let's assume now that for a certain product type (vegetable), we need a very similar query that has the operator in the last where clause different (< instead of >):

select count(*), sum(value)
from orders
join customers on orders.customerid = customers.customerid
join stores on orders.storeid = stores.storeid
join cities on customers.cityid = cities.cityid
join products on orders.productid = products.productid
where orders.value > 0
and stores.type = 'wholesale'
and customers.id is not null
and cities.name is not null
and products.type = 'vegetable'
and orders.value < 1000000


Since it is not a good idea to create another stored procedure, lets see how we can change the existing one:

alter procedure getInfo
@StoreType as varchar(50),
@ProductType as varchar(50)

as

begin

if @ProductType = 'fruit'

select count(*), sum(value)
from orders
join customers on orders.customerid = customers.customerid
join stores on orders.storeid = stores.storeid
join cities on customers.cityid = cities.cityid
join products on orders.productid = products.productid
where orders.value > 0
and stores.type = @StoreType
and customers.id is not null
and cities.name is not null
and products.type = @ProductType
and orders.value > 1000000

else
if @ProductType = 'vegetable'

select count(*), sum(value)
from orders
join customers on orders.customerid = customers.customerid
join stores on orders.storeid = stores.storeid
join cities on customers.cityid = cities.cityid
join products on orders.productid = products.productid
where orders.value > 0
and stores.type = @StoreType
and customers.id is not null
and cities.name is not null
and products.type = @ProductType
and orders.value < 1000000

else

return


end


This works but it is not very good.

What happens if we will have yet another special case as follows?

select count(*), sum(value)
from orders
join customers on orders.customerid = customers.customerid
join stores on orders.storeid = stores.storeid
join cities on customers.cityid = cities.cityid
join products on orders.productid = products.productid
where orders.value > 0
and stores.type = 'wholesale'
and customers.id is not null
and cities.name is not null
and products.type = 'herb'
and orders.value < 1000000
and orders.paid > 1000000

The stored procedure will become very complicated with lots of conditional statements and almost identical queries.



What is needed is a way of selecting the correct clause depending on the value of the Product Type parameter.

Please see below the final version of the stored procedure:


alter procedure getInfo
@StoreType as varchar(50),
@ProductType as varchar(50)

as

begin

select count(*), sum(value)
from orders
join customers on orders.customerid = customers.customerid
join stores on orders.storeid = stores.storeid
join cities on customers.cityid = cities.cityid
join products on orders.productid = products.productid
where orders.value > 0
and stores.type = @StoreType
and customers.id is not null
and cities.name is not null
and products.type = @ProductType
and
(
(isnull(@ProductType, 0) = 'fruit' and orders.value > 1000000)
OR
(isnull(@ProductType, 0) = 'vegetable' and orders.value < 1000000)
OR
(isnull(@ProductType, 0) = 'herb' and orders.value < 1000000 and orders.paid > 1000000)
)

end

Using the new technique, the stored procedure remains compact and efficient.

Why does this work?

This new clause uses the rules of Boolean algebra.








Lets see how the condition evaluates for @ProductType = 'fruit':

(ISNULL('fruit', 0) = 'fruit' AND orders.value > 1000000)
OR
(ISNULL('fruit', 0) = 'vegetable' AND orders.value < 1000000)
OR
(ISNULL('fruit', 0) = 'herb' AND orders.value < 1000000 and orders.paid > 1000000)

becomes

'fruit' = 'fruit' AND orders.value > 1000000
OR
'fruit' = 'vegetable' AND orders.value < 1000000)
OR
'fruit' = 'herb' AND orders.value < 1000000 and orders.paid > 1000000)

becomes

true AND orders.value > 1000000
OR
false AND orders.value < 1000000)
OR
false AND orders.value < 1000000 and orders.paid > 1000000)

becomes

true AND orders.value > 1000000
OR
false
 OR
false

becomes

true AND orders.value > 1000000 

becomes

orders.value > 1000000

The condition is evaluated similarly for the other values of the parameters.

Cross Browser testing in Web Driver

There are a few options that can be used for the cross browser test automation in Web Driver:

1. use the browsers installed on the local computer (free)

2. use Selenium Grid and browsers installed on remote computers (free)

3. use Sauce Labs (not free)

Options 1 and 2 are limited as for 1, only browsers for one operating system can be used.

For option 2, browsers for various operating systems can be used but the remote computers have to be set up and maintained.

With Sauce Labs, you get access to many operating systems and many browser versions.

Knowledge of Maven is needed for the Sauce Labs option.



WebDriver scripts with parameters

Some test automation projects require the ability to run the same test script multiple times using different values for the script parameters.

To use an example, the test script can do the following:

1. open the www.vpl.ca site
2. do a keyword search using java as the keyword
3. validate that the number of results is greater than 0
4. on the results page, change the sort order from the default sort order to Author
5. verify that the new sort order is correct

A few changes are needed for running this script multiple times using parameters:

1. open the www.vpl.ca site
2. do a keyword search using a {keyword parameter}
3. validate that the number of results is greater than 0
4. on the results page, change the sort order from the default sort order to a {sort parameter value}
5. verify that the new sort order is correct

A list of values for the 2 parameters (search keyword and sort order) is needed too:

search keyword sort order
java                  Author
vancouver Title
cats                  Rating
dogs                 Relevance

The script with parameters will be run as follows:

1. the first pair of values is selected from the list of values
2. the keyword parameter from the test script will take the keyword value from the pair of values
3. the sort order parameter from the test script will take the sort order value from the pair of values
4. the test script is executed using the selected values
5. the next pair of values is selected
6. continue from step 2

Going back to web driver, this is how the initial script looks like:

import static org.junit.Assert.*;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;

public class tests {


public WebDriver driver;

//use this class member for the search keyword value

private String keywordValue = "java";

//use this class member for the new sort order

private String sortOrderValue = "Author";

private String fullPatternValue = "Found (\\d+)(,*)(\\d+) items";
private String numberPatternValue = "(\\d+)(,*)(\\d+)";

@Before
public void setUp() 
{

System.setProperty("webdriver.chrome.driver",            "C:\\Users\\asiminiuc\\AppData\\Local\\Google\\Chrome\\Application\\chromedriver.exe"); 

driver = new ChromeDriver();         
       
}
   
@After
public void tearDown() 
{
driver.quit();
}

@Test
public void tests1() throws InterruptedException
{  

HomePage homePage = new HomePage(driver);
ResultsPage resultsPage = homePage.search(keywordValue);

assertTrue(new PatternExtract(resultsPage.getItemsNumber(), 
                                                 fullPatternValue).matches() == true);    
PatternExtract patternExtract = new PatternExtract(resultsPage.getItemsNumber(), 
                                                                  numberPatternValue);  

assertTrue(patternExtract.matches() == true);      
assertTrue(patternExtract.getValue() > 0);

resultsPage = resultsPage.changeSortOrder(sortOrderValue);
assertTrue(resultsPage.getSortOrderValue().equals(sortOrderValue));

}



The script with parameters looks like this:



import static org.junit.Assert.*;
import java.io.IOException;
import java.util.Arrays;
import java.util.Collection;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;

@RunWith(value = Parameterized.class)
public class testsPARAMETERS {

public WebDriver driver;

private String stringValue;

private String keywordValue, sortOrderValue;

private String fullPatternValue = "Found (\\d+)(,*)(\\d+) items";
private String numberPatternValue = "(\\d+)(,*)(\\d+)";  

public testsPARAMETERS(String stringValue) {
this.stringValue = stringValue;
}

@Parameters
public static Collection<Object[]> data() throws IOException {
       
Object[][] data = {
          {"java-Author"}, 
       {"vancouver-Title"}, 
       {"cats-Rating"}, 
       {"dogs-Relevance"}
                  };

return Arrays.asList(data);   
}

@Before
public void setUp(){
System.setProperty("webdriver.chrome.driver", 
"C:\\Users\\asiminiuc\\AppData\\Local\\Google\\Chrome\\Application\\chromedriver.exe"); 

driver = new ChromeDriver();               
}
   
@After
public void tearDown() 
{
driver.quit();
}
@Test
public void tests1() throws InterruptedException {

String[] values = stringValue.split("-");
keywordValue = values[0];
sortOrderValue = values[1];

HomePage homePage = new HomePage(driver);
ResultsPage resultsPage = homePage.search(keywordValue);

assertTrue(new PatternExtract(resultsPage.getItemsNumber(),  
            fullPatternValue).matches() == true);  

PatternExtract patternExtract = new PatternExtract(resultsPage.getItemsNumber(), 
                                                                  numberPatternValue);  

assertTrue(patternExtract.matches() == true);      
assertTrue(patternExtract.getValue() > 0);

resultsPage = resultsPage.changeSortOrder(sortOrderValue);
assertTrue(resultsPage.getSortOrderValue().equals(sortOrderValue));
}

}


These are the differences between the 2 scripts (highlighted above in red):

- a few additional libraries are added:

import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

- the test class uses a special annotation:

@RunWith(value = Parameterized.class)

- a new member is added to the class:

private String stringValue;

This is the variable that will store the current pair of values for the test script parameters

- a new constructor is added to the test class:

public testsPARAMETERS(String stringValue) 
{
this.stringValue = stringValue;
}

It is in the constructor that the stringValue is initialized with the current pair of values.

- a new method is added to the class:

@Parameters
public static Collection<Object[]> data() throws IOException

This method defines the list of the parameter data and provides the selected pairs of values to the test script

- code is added to the test method for breaking the pair of values in the keyword and the sort order:

String[] values = stringValue.split("-");
keywordValue = values[0];
sortOrderValue = values[1];





_____________________________________________________________________________


Do you want to learn more about test automation and Java?
I will start an SELENIUM online group training on October 15.
Please see the training details here.

Why are stored procedures sometimes so slow compared with plain SQL statements?

I have been doing lately lots of database testing by checking if data displayed in business intelligence reports is correct.

The reports data goes from the original database to an intermediate database (through some processing), then a cube is generated and finally, the reports.

The testing consists in writing SQL queries against the original database and comparing the query results with the BI reports data.

So far so good.

Some of the queries were very similar so I tried to re-use the code by creating stored procedures that have parameters.

For example, a simple SELECT statement like this

SELECT table1.field1, table2.field2
FROM table1
JOIN table2 ON table1.key = table2.key
WHERE table1.filter1 = 'value1' and table2.filter2 = 'value2'

was converted in the following stored procedure:

CREATE PROCEDURE doSomething

@Filter1Value as varchar(10)
, @Filter2Value as varchar(10)

AS
BEGIN

SET NOCOUNT ON;

        SELECT table1.field1, table2.field2
        FROM table1
        JOIN table2 ON table1.key = table2.key
        WHERE table1.filter1 = @Filter1Value and table2.filter2 = @Filter2Value

END

The stored procedure can then be executed as follows instead of running directly the SQL statement:

EXEC doSomething 'value1', 'value2'


So easy, right?

The problem is that the performance of the stored procedure was terrible compared with the performance of the SQL statement.

In some cases, the results of the SQL statement would be generated in 10 seconds and I would still wait for the results of the stored procedure after 15 minutes!

The SQL code could not be the cause of the performance problem since it is identical.

Why is the performance of the stored procedure so bad then?

The answer is that SQL Server does something called parameter sniffing for the parameters of stored procedures for performance optimization.

How can this be fixed?

See below the same stored procedure with a minor change:

CREATE PROCEDURE doSomething

@Filter1Value as varchar(10)
, @Filter2Value as varchar(10)

AS
BEGIN

       DECLARE @LocalFilter1Value as varchar(10)
       DECLARE @LocalFilter2Value as varchar(10)

       SET @LocalFilter1Value = @Filter1Value
       SET @LocalFilter2Value = @Filter2Value

SET NOCOUNT ON;

        SELECT table1.field1, table2.field2
        FROM table1
        JOIN table2 ON table1.key = table2.key
        WHERE table1.filter1 = @LocalFilter1Value and table2.filter2 = @LocalFilter2Value



END

The change consists in creating local variables in the stored procedure, initialize them with the values of the parameters and then use the local variables in the SQL statement.


As soon as this change is done, the performance of the stored procedure is the same with the performance of the plain SQL statement.