Selenium over QTP

Here is an excerpt from a recommendation made for my current project, where we chose to use Selenium rather than the established (and, in our case, mandated) QTP.

Reasons for:

·         Developers will be able to easily run tests in local sandbox environment

·         Tests can be written in Java

o   Tests can be dev-friendly; developers could assist with tests

o   Java is tidier and easier to deal with than VBS (QTP’s language), and considerably more powerful in application

o   Social element, as dev and test have another thing in common

·         Tests will be faster to execute

o   QTP has a time lag on loading (when libraries being loaded are sizeable, and also when acquiring a license from the license server)

·         Browser support includes possibility of iOS and Android emulators

·         Eclipse is the better IDE

o   Tests will be easier to create and maintain

o   Debug time will likely be faster as a result

·         Integration with continuous build systems. Selenium tests can be easily initiated from command line already, this is not quite so easy with QC/QTP.   

Agile Automation - tooling overview

I've spent a lot of time working with QTP over the years, and have successfully driven tests using a combination of batch processes, VBS processes and QC OTA API calls. While QTP works nicely (has it's niche and so on) I have a personal preference for Selenium WebDriver[0] for integrating testing within an agile team (light weight, cost, versatility of tool, reporting, debugging, targeting tests, ease of writing and updating assets, scaling is actually affordable, actually means working with developers, and so on). 

[0] Full list of tools involved in current project's implementation

• Selenium RC/Grid for distributing tests, 
• TestNG for driving the tests, 
• Subversion for assets, 
• Cruise Control at present, 
• moving to Jenkins for building all artefacts, 
• Python for serving up results, and 
• PSExec, VBS and batch scripts for deploying/initiating/plumbing everything that's needed. 

Automation and Continuous Integration

Familiar story: A few devs commit some lovely code, one dev commits some lovely bugs. Testers deploy the latest code compiled from all devs that evening. Testers begin running some tests the next morning. A bug is located. Who made it requires a little bit of chasing. What was the change that made it requires a bit of thinking. Dropping current tasks to pick up bug fixing requires a little bit of context-switch-time.  That's inefficient. Finding bugs early is cheap. Any delay and costs start ramping (illustration of a graph with a curve available here) 

What's pretty cool/sleek/shiny is the far-from-new-or-novel idea that we perform a bunch of tests against everything a developer creates. Immediately. For every change. Just run stuff. 

Amazon are typically the prime example of having the sleekest of processes, as they are able to go from a developer making a change to actually seeing this change in production within 1/2 an hour[0], having completed a full regression in this time. 

[0]this is from a conversation I had with some random person at some random pub, credibility is definitely lacking. But the idea is there, captured in this (possibly fictitious) scenario, that with the right testing in place we can provide however much feedback is needed. 

So, what do we want? We want a range of tests which are easy to create, easy to maintain, that cover off a fairly vast portion of the application, and detect defects quickly. This just requires having a healthy build process within a team, and plugging in a test automation framework that meets the requirements. It'll all need to be easy, largely (100%?) automated, and have valuable feedback to the team. 

There's nothing profound with any of this; it's a simple idea: test early and often. It's cheaper, and will allow everyone to spend their energies on making the best darn product they can. 

Grouping Tests - TestNG

One of the many brilliant TestNG features is the ability to group tests.

This is a high-level overview of how I structure this; I will try and flesh this out when I get some down time.

From command line, invoke TestNG
java org.testng.TestNg FullRegression.xml
(from memory) (note, classpaths will probably need to be set up)

My file, FullRegression.xml just references other files.


<suite name="MyProjectTestSuite" verbose="3" >
 <suite-files>
  <suite-file path="./Regression_Story101.xml" />
  <suite-file path="./Regression_Story102.xml" />
  <suite-file path="./Regression_Story103.xml" />
  <suite-file path="./Regression_Story104.xml" />
  <suite-file path="./Regression_Story105.xml" />
  <suite-file path="./Regression_Story106.xml" />
  <suite-file path="./Regression_Story107.xml" />
  <suite-file path="./Regression_Other.xml" />
  <suite-file path="./Regression_Failing.xml" />
 </suite-files>
</suite>

Example of a subsequent file looks something like this

<suite name="Story104" verbose="3" >
 <test name="Regression1">
  <groups>
   <run>  
    <exclude name="overkill" />
    <exclude name="failing" />
<include name="story104" />
   </run>
  </groups>
  <packages>
   <package name="com.whiteboxit.mattTestA.tests" />
  </packages>
 </test>
</suite>

Why have I done this

Well, visibility. My auto-generated html run results are now broken down by story. This looks awesome. And it is really easy for me to keep on top of something which has started failing, or whatever.

How I've approached test result reporting with TestNG

This is a look at how I've approached result reporting on my current project. Results, current and historic, are served up on my company's intranet at a URL.

Using:
Selenium, TestNG, Java and Python

So, I run my TestNG tests and these result in some cool looking results automatically (win). These look a bit like this:

Two limitations though:

1. I want everyone to be able to view this easily, and 
2. I want a historical record of past runs to be present

Solution, I'm saving these in to a folder which is hosted up by a simple HTTP server, which is built in to Python. 

Python / simple HTTP server

Go to directory. 

Invoke the following:

c:\dev\opt\python-2.7\python -m SimpleHTTPServer 8000

Obviously I've put in the path to Python, obviously you'll need to change the path to wherever your Python is. 

Historical Record

This was achieved by just creating a copy of the assets at the location being served up whenever I wanted. 

By navigating to the root dir in a browser I could see all of my result folders. 

 Nice, now I just click on a link. 

Batch Scripting a folder copy for historical run results

Just because I am too lazy to copy a folder and rename the new one to a date stamp I created the following batch script which would do it for me. 

@echo off

ECHO Generating timestamp to use in folder name

For /f "tokens=2-4 delims=/ " %%a in ('date /t') do (set mydate=%%c%%b%%a)

For /f "tokens=1-2 delims=/:" %%a in ("%TIME%") do (set mytime=%%a%%b)

echo %mydate%-%mytime%

SET newResultFolderName=%mydate%-%mytime%

ECHO New folder name will be %newResultFolderName%

SET locationOfLatestRunResults="C:\dev\tests\DrivingXml\test-output\"

SET uriToPublishResults="\\myserver01\selenium-test-output\welc\%newResultFolderName%\"

ECHO Location of latest run results: %locationOfLatestRunResults%

ECHO URI to publish results to: %uriToPublishResults%

XCOPY /S /V /Z %locationOfLatestRunResults%* %uriToPublishResults%

EXPLORER %uriToPublishResults%

Executing TestNG Selenium tests via Command Line

My Java's a little rusty.
I'm new to Selenium.
This will be a guide for me: the 'above novice programmer' and 'below dev'.

Starting Assumptions

Java jre is installed.
Class path for Java has been set up.
You know how to add/edit class paths (for most (all?) Windows machines this is: Start > right click My Computer > Properties > Advanced tab > Environment Variables)

There is a Selenium project kicking around, already set up with TestNG. 

We're on a Windows computer.

I'm going to assume that you've been able to run these from your IDE - in my case, Eclipse. For help on this, or getting TestNG set up originally, refer to my older post on Implementing TestNG. 

What two things we need to set up

Two classpaths are required. Please be aware that a computer reboot is required in order for a class path / environment variable to be set.

Class path to Selenium jar

So I have a file on my computer: C:\Selenium\selenium-server-standalone-2.25.0.jar
So I add in to my class path
c:\Selenium\*
From memory I think this didn't work without the \* - I can't really be bothered analysing why, or double checking this. 

Class path to my project

And I have a project at C:\dev\Workspace\MyProject\
So I will add the bin to my class path
C:\dev\Workspace\MyProject\bin;

This seemed to throw a wobbly when I added \* ..again, I am a bit perplexed but can't really justify analysing why, or double checking this. 

Command to Run

Refer to the official documentation for examples on creating a driving xml file.

C:\Selenium\DrivingXml>java org.testng.TestNG myDrivingXmlFile.xml

Work log (and issues encountered)

Could not find or load main class org.testng.TestNG

Check your class paths. I had this because I failed to include my class path to my selenium jar. Or had an error in one of my class paths.

Exception in thread "main" java.lang.NoClassDefFoundError: org/testng/TestNG

Computer probably doesn't hvae TestNG jar file. Will test this on Monday though as today is POETS day. 

QTP/vbs - Dynamically loading executable code - pt II.

I very recently posted an article on loading vbs code dynamically, and noted that this had a shortcoming with functions loaded not being globally available.

Solution

ExecuteFile "c:\myFile.vbs"

Where myFile.vbs is a file filled with a bunch of executable commands and such. 

QTP/vbs - Dynamically loading executable code - pt I.

There is a cool function in vbs which allows you to execute a string as regular vbs code.
This function is Execute.
The downside is that the Execute command is not persistent; i.e. the variables declared in memory, and functions 'registered', will not be available globally[0]. I didn't realize this in 3 years of using QTP until I came to reimplement it on another project.

[0] Example
strMyVbs = "strSomeText = ""Hello World"" "


f_loadMyVariable
msgbox (strSomeText) 'This does not print Hello World to the screen, because the executed code was/is local to the function

Function f_loadMyVariable()
  Execute strMyVbs
  msgbox (strSomeText) 'This will print Hello World to the screen
End Function

Hopefully, for the sake of my productivity today, there will be a post following showing how to make this a global thing. -- solved, QTP/vbs - Dynamically loading executable code - pt II. --

Implementation

A quick elaboration on implementation
Dim uri
uri = PathFinder.Locate("[QualityCenter\Resources] Resources\SomeOtherFolder\myFile.qfl")

Dim oFileSystem, oFile
Set oFileSystem = CreateObject("Scripting.fileSystemObject")
Set oFile = oFileSystem.OpenTextFile(uri)
Dim strInputText
strInputText = oFile.ReadAll
Execute strInputText
oFile.Close