Scripting languages comparison

Related article:
Thoughts on test automation (Brett Pettichord)

Last week, I went through a lot of printed articles waiting to be read and among them were a few from Brett Pettichord on test automation.

If you don't know who he is, he wrote LESSONS LEARNED IN SOFTWARE TESTING with James Bach and Cem Kaner and is one of the experts in test automation.

The HOMEBREW TEST AUTOMATION article (available on has interesting comparisons between scripting languages as follows:

Ruby, Python and Perl are all supported by SELENIUM.

The article is rather old (2004) but the comparison is still useful especially for testers that consider learning their first scripting language.

This convinced me of learning Ruby so will give it a try in the next months.

So far, the only language I used for test automation is VB Script with QTP.

But I am open to better language alternatives, especially since QTP is so expensive and Selenium is free :)


Brett Pettichord - thoughts on test automation

The following were selected from various articles of Brett Pettichord from 2002 and 2004. 
- Focus the automation strategy on key features or concerns that can benefit from the added power of automated testing

- Find a way to leverage the staff skills available for testing

- Test automation is a software development process:

    1. establish milestones and deliverables
    2. define requirements
    3. manage source code, test data, tools
    4. design before coding
    5. review the code
    6. test the code
    7. track automation bugs
    8. document for other users

- Test automation is a significant investment:

    1. staff time commitment can dwarf tool cost
    2. prepare for 10 times manual effort
    3. maintenance effort can be large

- Automated regression tests find a minority of the bugs and most of them are found when the tests are created

- Can you automate at times when you can't otherwise test?

- Capture-replay fails; tests are too sensitive for user interface changes

- Test tools are buggy

- The problem with bad automation is that no one may notice:

    1. tests may not do what you thought
    2. tests may no longer be interesting
    3. coverage may be poor
    4. false alarms may be common
    5. test results may be false

- Test automation projects require skills in
    1. testing
    2. development
    3. project management

- Don't mandate 100 percent automation

- Use pilot projects to prove feasibility

- Use abstraction techniques for the user interface changes:

    1. window maps
    2. data driven tests
    3. task libraries
    4. keyword driven tests
    5. api driven tests

- Avoid complex logic in your tests

    1. keep tests linear: easy to understand, hard to make mistakes
    2. encapsulate necessary complexity

- Don't build libraries simply to avoid repeating code; hodge-podge libraries make tests hard to understand and maintain; design your libraries, don't just whip them together on the fly

- Use standard scripting languages, avoid proprietary languages

- Have testers and programmers jointly charter automation projects

- Speed the development process instead of trying to save a few dollars on testing

- Test automation tool components:

    1. languages

        * tie other components together
        * describe tests
        * provide dedicated libraries

    2. interface drivers

        *interact with the product software under test

    3. test harnesses

        * collect and execute multiple tests
        * report test results

    4. remote agents

        * provide test related services on remote machines

    5. test parsers

        * interpret and execute tests expressed in convenient form

    6. test generators

        * create new tests based on models and algorithms

- Scripting languages for testing

        1. well established
        2. vast libraries


        1. well established
        2. compact
        3. popular with embedded systems


        1. concise support for object oriented programming
        2. integrates well with java

    RUBY - best

        1. everything's an object
        2. principle of least surprise


        1. popular
        2. integrate well with Microsoft technologies
        3. not open source

    TCL, PYTHON and RUBY have line interpreters.

    Another up and coming scripting language is GROOVY. Like Jython, it runs inside the JVM.

    PERL has the most libraries of any of the languages listed here. For many people, this is a good reason to use PERL.

    The lack of a line interpreter for PERL is to me its greatest weakness.

    The lack of native support for general templating is PYTHON's greatest weakness.

- Understand the motivations of the test tool vendors

    1. want you to think they are experts in testing: guess what? they are not
    2. encourage belief in tool magic
    3. achieve lock-in when your test library is in their language
    4. focus on improvements that people will see during the eval period: enhancements for power users go on the back burner

- Test automation architectural patterns
        1. scripting frameworks
        2. data driven scripts
        3. screen based tables
        4. action keywords
        5. test first programming
        6. api tests
        7. thin gui
        8. consult an oracle
        9. automated monkey
        10. assertions and diagnostics
        11. quick and dirty

- Test automation common problems

        1. spare time test automation
        2. lack of clear goals
        3. lack of experience
        4. testing the wrong stuff
        5. high turnover
        6. reaction to desperation
        7. reluctance to think about testing
        8. technology focus
        9. working in isolation

- Rules of software development

        1. define requirements
        2. manage source code, test data, tools
        3. design before coding
        4. review test automation code
        5. test the code
        6. track automation bugs
        7. document for other users
        8. establish milestones and deliverables


        1. follow the rules of software development
        2. improve the testing process
        3. define requirements
        4. prove the concept
        5. champion product testability
        6. design for sustainability
        7. plan for deployment

- 2 focuses for test automation


            - reduce testing costs
            - reduce time spent in testing phase
            - improve test coverage
            - make testers look good
            - reduce impact on the bottom line


            - tighten build cycles
            - enable refactoring and other risky practices
            - prevent de-stabilization
            - make developers look good
            - increase management confidence in the product

        Automation projects with a service focus are more successful.

- User interface change: your gui test automation will likely break. what can you do?

        1. prevent developers from changing them
        2. design your automation so it is adaptable
        3. test via non-user interfaces

- Quality attributes for test automation

        1. maintainability
        2. reviewability
        3. repeatability
        4. integrity
        5. reliability
        6. reusability
        7. independence
        8. performance
        9. simplicity

- Top 10 reasons for automating tests

    1. manual testing sucks
    2. tool vendor said that capture replay works
    3. afterwards, we can fire all those pesky testers
    4. everybody else is doing it
    5. big bucks already spent on the test tool
    6. looks good on the resume
    7. keep the intern busy

- Gradual test automation: test automation benefits from a gradual approach; build some tests and se how they run before adding complexity

- Use a pilot project:

    1. validate your tools and approach
    2. demonstrate that your investment in automation is well spent
    3. quickly automate some real tests
    4. get a trial license for any test tools
    5. scale your automation project in steps

- Reasons for automating

    1. speed up testing
    2. allow more frequent testing
    3. reduce manual labor costs
    4. improve test coverage
    5. ensure consistency
    6. simplify testing
    7. just want testing to go away
    8. define the testing process
    9. make testing more interesting and challenging
    10. develop programming skills
    11. justify cost of the tools

    - high reuse
    - speed development
    - expand reach
    - smooth development

    - simplify testing
    - force organization
    - 100% automation
    - justify tool purchase

    - regression testing
    - build skill and morale

- What are your success criteria

        1. the automation runs
        2. the automation does real testing
        3. the automation finds defects
        4. the automation saves time


    1. is automation or testing a label for other problems?
    2. are testers trying to use automation to prove their prowess?
    3. can testability features be added to the product code?
    4. do testers and developers work cooperatively and with mutual respect?
    5. is automation developed on an iterative basis?
    6. have you defined the requirements and success criteria for automation?
    7. are you open to different concepts of what test automation can mean?
    8. is test automation lead by someone with an understanding of both programming and testing?


    1. keep it simple
            - test automation tends to complicate things
            - the test suite itself will need to be tested
            - make sure the test suite meets the original goals

    2. build flexibly and incrementally
            - build and deliver test automation in stages
            - deliver a proof of concept early
            - deliver automation updates regularly
            - package the automation so that it can be installed easily
            - document all dependancies

    3. work with development
            - test automation is development; get help from your development experts (developers)
            - incorporate automation milestones into the development plan

    4. keep tests visible
            - visibility facilitates review
            - review encourages realism about test coverage
            - test suites require review and improvement
            - dont assume that old features are covered by existing tests
            - assess test suite weaknesses and product risks
            - are these tests still useful?

    5. use multiple approaches
            - it is better to use several approaches half-way than to use one perfectly
            - test expertise and manual testing are still required
            - manual testing is a sanity test for the automation
            - there will always be need for exploratory, non-repeated testing

    6. get an early start
            - an early start makes it more likely you can improve testability and get test APIs
            - your tes strategy will include automation from the start

    7. commitment is essential
            - it is easy for test automation to be designed as a side project. it wont get the resources it needs
            - commitment ensures that test automation gets the resources, cooperation and attention that it needs
                    * from management
                    * from development

Thoughts on testing - James Whittaker

- Take your testing up a level from test cases to techniques

Test cases are a common unit of measurement in the testing world. we count the number of them that we run, and when we find a bug, we proudly present the test case that we used to find it.

I hate it when we do that.

I think test cases - and most discussions about them - are generally meaningless. 
I propose we talk in more high-level terms about test techniques instead. for example, when one of my testers finds a bug, they often come to my office to demo it to me (i am a well-known connoisseur of bugs and these demos are often reused in lectures i give around campus).

Most of them have learned not to just repro the bug by re-running the test case. They know i am not interested in the bug per se but the context of its exposure. there are the type of questions i like to ask during bug demos:

1. what made you think of this particular test case?
2. what was your goal when you ran that test case?
3. did you notice something interesting while running the test case that changed what you were doing?
4. at what point did you know you had found a bug?

At Microsoft, we have begun using the concept of test tours as a way to categorize test techniques.

- It is our task to use testing as an instrument of improvement

- This is the true job of a tester: to make developers and development better; we don't ensure better software - we enable developers to build better software; it isn't about finding bugs, because the improvement caused is temporal

The true measure of a great tester is that they find bugs, analyze them thoroughly, report them skillfully and end up creating a development team that understands the gaps in their own skill and knowledge

Making developers better, helping them understand failures and the factors that cause them will mean fewer bugs to find in the future

The real value of tests is not that they detect bugs in the code, but that they detect inadequecies in the methods, concentration and skill of those who design and produce the code (Tony Hoare)

- Testing without innovation is a great way to lose talent

Testing sucks. now let me qualify that statement: running test cases over and over - in the hope that bugs will manifest - sucks. it's boring, uncreative work. 

What is interesting about testing is strategy, deciding what to test and how to combine multiple features and environmental consideration in a single test. the tactical part of testing, actually running test cases and logging bugs, is the least interesting part

High Stakes, No Prisoners - Charles Ferguson

I just finished this great book about the first 5 years of the Internet and the many changes it brought to the high technology companies and everybody's life in general.

Written by one of the founders of Vermeer Technology (creator of Front Page, later sold to Microsoft), the  book is an excellent presentation of the rise and fall of Netscape, the Internet revolution and of course Microsoft, both the smartness of the company and the predatory behaviour.

There are so many interesting things in the book and on so many topics like:

- everything about a high tech start up
- why is Microsoft so successful at destroying other companies
- venture capitalist practices
- how to create a software product architecture
- how to hire and not to hire a CEO
- and many others

Some of my favourite paragraphs are below:

- Whether through haste, overconfidence or ignorance, in 1994 and 1995 Netscape 
made a series of catastrophic technical and strategic errors that eventually proved their 
undoing. These included sloppy, indeed almost non-existent, technical architecture; 
foolish, immature hype that awakened Microsoft; failure to create proprietary advantage; 
failure to generate 3rd party support and lock-in;poor testing and quality control; 
excessive attention to minore markets to the neglect of windows

- the next mistake was the hiring of a non-technical CEO

- the CEO of a serious high technology company must have a serious technical 
background, or at least the ability to understand technology and a deep appreciation of 
its importance in strategic and organizational decisions

- i have a generally low opinion of professional CEOs, although some are superb. They 
frequently have poor technical backgrounds, no concern for anything other than their 
bank accounts, and skills that are more political than substantive. And their track records 
are not great. Of the industry's most successful companies, there's only one - Cisco - that 
has been run for a long time by a conventional professional CEO. Intel, Microsoft, Oracle, 
Aol, Compaq, Dell and Gateway are all run by founders, early employeesand/or former 
academics who somehow do okay despite not having Harvard mbas and perfect resumes. 
When you stack them up against the people responsible for the last 20 years performance 
of Apple, IBM, Netscape and Lotus and many others, you come away thinking that 
unqualified, socially dysfunctional, impatient, ruthless, egomaniacal founders arent so 
bad after all

- writing a clever piece of code that works is one thing; designing something that can 
support a long-lasting business is quite another. Commercial software design and 
production is, or should be, a rigurous, capital - intensive activity. Software products 
should be based on a broad, deep structure that can support much more than whatever 
the product contains at any given time. In addition to code that works, you need 
documentation, help functions, error handling, multi-platform support, and multiple 
languages. You also need an underlying architecture that allows you to add and 
change features, purchase and integrate external software components, and allow 
other software vendors to make their products talk to yours, add customized widgets to 
it, or embed your product into something larger of their own. A good architecture, one 
that will carry you for a decade's worth of unpredictable technology and market 
changes, takes months to develop. But if  you skip this step, as Netscape did, you 
have made a truly Faustian bargain

- one goal of good architecture is to enable systems to be partitioned so that people 
can work effectively in parallel, their progress can be measured periodically and 
everyone can finish at the same time

- There were lots of kids and Unix people at Netscape, but few seasoned technologists 
with PC software experience. Andreessen's job as chief technology officer was 
supposedly deciding what Netscape's future products should be. But he had no 
experience, and there was no chief architect; so for architectural change, who was 
minding the store? Moreover, of the entire top management team, only one person 
had any experience with a pc succssful software company

- the mangling of large companies by polished, politically astute, non-technical CEOs, 
which is a major part of what happened at Netscape, is a movie that Silicon Valley 
has seen many times before. ….. firing highly visible CEOs is all too rare, no matter 
how bad they are. Ben Rosen has done it twice, when he replaced the CEO of 
Compaq in 1991 and again in 1999. More often, however, everyone sits around, 
afraid to be the first to blow the whistle and provoke a fight. But by the time an obvious 
crisis forces action, it's often too late. 

- Netscape's story also provides a remarkable case study in how to run a high tech 
company, and how not to, at both strategic and operational levels. 

- Perhaps never in American history has a new company faced larger opportunities 
and such profound choices as Netscape did.

- Netscape's browser was a mess from the very beginning. Their first browser was 
relatively simple, so choosing hacker style and spaghetti code to reach the market 
fast was tempting. But there is a general rule in software engineering that the cost of 
fixing an error doubles with every step down the engineering sequence, from initial 
conception to architecture to commercial release, and these costs continue to double 
with subsequent releases. 

- Navigator 1.0 had almost no architecture. In Netscape's rush to market, it was 
conceivably justifiable to do this. Even if so, the failure to fix it immediately by developing 
a real architecture for the next product was utterly, totally fatal. Navigator 1.0 was 
throwaway code that they didnt throw away.

- underlying each of the next browsers (navigator 2.0, 3.0, 4.0) was the non-architected 
faustian bargain of the navigator 1.0. With each release, Netscape's cost structure, 
technical limitations and development times worsened relative to Microsoft's, as did its 
ability to support new features and interfaces, with the result that Netscape's lead, which 
was at least a year, was eliminated within two years

- Netscape's failure to architect its browser was just one of many engineering errors. 
Another, also serious, concerned testing. Software quality assurance is a complex activity, 
with a range of best practices and capital-intensive software tools for automating testing, 
bug tracking and error correction. For complex products, you even design the testing 
systems into the original architecture of the product. That's not the way Netscape did it, to 
put it mildly. According to some reports, Navigator 1.0 testers were hired by placing notes 
on bulletin boards offering to hire random college students at ten dollars per hour. A 
dangerous casual attitude toward QA persisted even well after Netscape had focused 
on mission critical application and server products for conservative, quality-conscious
 Fortune 500 clients

- during the development of Communicator 4.0, Netscape allocated about half of its 
browser development team to an effort to redesign the browser from scratch, using the 
java language. It would be difficult to exxagerate the idiocy of this choice. At this writing, 
nobody yet uses Java to write large Windows applications, because its performance is 
still mediocre and its tools, environment and user interfaces are insufficiently mature. 
After a few months, Netscape canceled the effort

- Netscape went to great lengths to make its own life more difficult and succeeded.

- Netscape decided to develop its early browsers so that they can be developed and 
released for many operating systems simultaneously, not just the multiple versions of
 Windows but also the McIntosh, OS/2 and a half dozen varieties of Unix. These operating 
systems and their user interfaces are fundamentally different from one another. 
Consequently, Netscape developed a major piece of software, the Netscape portable 
runtime layer that covered up the specific features of various operating systems. Netscape 
developers then wrote code targeted to this generic intermediate layer rather than to 
specific operating systems. This decision was an enormous strategic error. …. it was a 
major technological and engineering disaster. It meant that Netscape deprived itself of the 
best available tools for developing software, particularly for Windows.