The following were selected from various articles of Brett Pettichord from 2002 and 2004.
- Focus the automation strategy on key features or concerns that can benefit from the added power of automated testing
- Find a way to leverage the staff skills available for testing
- Test automation is a software development process:
1. establish milestones and deliverables
2. define requirements
3. manage source code, test data, tools
4. design before coding
5. review the code
6. test the code
7. track automation bugs
8. document for other users
- Test automation is a significant investment:
1. staff time commitment can dwarf tool cost
2. prepare for 10 times manual effort
3. maintenance effort can be large
- Automated regression tests find a minority of the bugs and most of them are found when the tests are created
- Can you automate at times when you can't otherwise test?
- Capture-replay fails; tests are too sensitive for user interface changes
- Test tools are buggy
- The problem with bad automation is that no one may notice:
1. tests may not do what you thought
2. tests may no longer be interesting
3. coverage may be poor
4. false alarms may be common
5. test results may be false
- Test automation projects require skills in
1. testing
2. development
3. project management
- Don't mandate 100 percent automation
- Use pilot projects to prove feasibility
- Use abstraction techniques for the user interface changes:
1. window maps
2. data driven tests
3. task libraries
4. keyword driven tests
5. api driven tests
- Avoid complex logic in your tests
1. keep tests linear: easy to understand, hard to make mistakes
2. encapsulate necessary complexity
- Don't build libraries simply to avoid repeating code; hodge-podge libraries make tests hard to understand and maintain; design your libraries, don't just whip them together on the fly
- Use standard scripting languages, avoid proprietary languages
- Have testers and programmers jointly charter automation projects
- Speed the development process instead of trying to save a few dollars on testing
- Test automation tool components:
1. languages
* tie other components together
* describe tests
* provide dedicated libraries
2. interface drivers
*interact with the product software under test
3. test harnesses
* collect and execute multiple tests
* report test results
4. remote agents
* provide test related services on remote machines
5. test parsers
* interpret and execute tests expressed in convenient form
6. test generators
* create new tests based on models and algorithms
- Scripting languages for testing
PERL
1. well established
2. vast libraries
TCL
1. well established
2. compact
3. popular with embedded systems
PYTHON
1. concise support for object oriented programming
2. integrates well with java
RUBY - best
1. everything's an object
2. principle of least surprise
VB & VBSCRIPT
1. popular
2. integrate well with Microsoft technologies
3. not open source
TCL, PYTHON and RUBY have line interpreters.
Another up and coming scripting language is GROOVY. Like Jython, it runs inside the JVM.
PERL has the most libraries of any of the languages listed here. For many people, this is a good reason to use PERL.
The lack of a line interpreter for PERL is to me its greatest weakness.
The lack of native support for general templating is PYTHON's greatest weakness.
- Understand the motivations of the test tool vendors
1. want you to think they are experts in testing: guess what? they are not
2. encourage belief in tool magic
3. achieve lock-in when your test library is in their language
4. focus on improvements that people will see during the eval period: enhancements for power users go on the back burner
- Test automation architectural patterns
1. scripting frameworks
2. data driven scripts
3. screen based tables
4. action keywords
5. test first programming
6. api tests
7. thin gui
8. consult an oracle
9. automated monkey
10. assertions and diagnostics
11. quick and dirty
- Test automation common problems
1. spare time test automation
2. lack of clear goals
3. lack of experience
4. testing the wrong stuff
5. high turnover
6. reaction to desperation
7. reluctance to think about testing
8. technology focus
9. working in isolation
- Rules of software development
1. define requirements
2. manage source code, test data, tools
3. design before coding
4. review test automation code
5. test the code
6. track automation bugs
7. document for other users
8. establish milestones and deliverables
- SEVEN STEPS TO TEST AUTOMATION SUCCESS
1. follow the rules of software development
2. improve the testing process
3. define requirements
4. prove the concept
5. champion product testability
6. design for sustainability
7. plan for deployment
- 2 focuses for test automation
EFFICIENCY
- reduce testing costs
- reduce time spent in testing phase
- improve test coverage
- make testers look good
- reduce impact on the bottom line
SERVICE
- tighten build cycles
- enable refactoring and other risky practices
- prevent de-stabilization
- make developers look good
- increase management confidence in the product
Automation projects with a service focus are more successful.
- User interface change: your gui test automation will likely break. what can you do?
1. prevent developers from changing them
2. design your automation so it is adaptable
3. test via non-user interfaces
- Quality attributes for test automation
1. maintainability
2. reviewability
3. repeatability
4. integrity
5. reliability
6. reusability
7. independence
8. performance
9. simplicity
- Top 10 reasons for automating tests
1. manual testing sucks
2. tool vendor said that capture replay works
3. afterwards, we can fire all those pesky testers
4. everybody else is doing it
5. big bucks already spent on the test tool
6. looks good on the resume
7. keep the intern busy
- Gradual test automation: test automation benefits from a gradual approach; build some tests and se how they run before adding complexity
- Use a pilot project:
1. validate your tools and approach
2. demonstrate that your investment in automation is well spent
3. quickly automate some real tests
4. get a trial license for any test tools
5. scale your automation project in steps
- Reasons for automating
1. speed up testing
2. allow more frequent testing
3. reduce manual labor costs
4. improve test coverage
5. ensure consistency
6. simplify testing
7. just want testing to go away
8. define the testing process
9. make testing more interesting and challenging
10. develop programming skills
11. justify cost of the tools
REASONABLE REASONS
- high reuse
- speed development
- expand reach
- smooth development
UNREASONABLE REASONS
- simplify testing
- force organization
- 100% automation
- justify tool purchase
MIXED BAG
- regression testing
- build skill and morale
- What are your success criteria
1. the automation runs
2. the automation does real testing
3. the automation finds defects
4. the automation saves time
- ARE YOU READY TO AUTOMATE?
1. is automation or testing a label for other problems?
2. are testers trying to use automation to prove their prowess?
3. can testability features be added to the product code?
4. do testers and developers work cooperatively and with mutual respect?
5. is automation developed on an iterative basis?
6. have you defined the requirements and success criteria for automation?
7. are you open to different concepts of what test automation can mean?
8. is test automation lead by someone with an understanding of both programming and testing?
- WHAT HAVE WE LEARNED ABOUT TEST AUTOMATION
1. keep it simple
- test automation tends to complicate things
- the test suite itself will need to be tested
- make sure the test suite meets the original goals
2. build flexibly and incrementally
- build and deliver test automation in stages
- deliver a proof of concept early
- deliver automation updates regularly
- package the automation so that it can be installed easily
- document all dependancies
3. work with development
- test automation is development; get help from your development experts (developers)
- incorporate automation milestones into the development plan
4. keep tests visible
- visibility facilitates review
- review encourages realism about test coverage
- test suites require review and improvement
- dont assume that old features are covered by existing tests
- assess test suite weaknesses and product risks
- are these tests still useful?
5. use multiple approaches
- it is better to use several approaches half-way than to use one perfectly
- test expertise and manual testing are still required
- manual testing is a sanity test for the automation
- there will always be need for exploratory, non-repeated testing
6. get an early start
- an early start makes it more likely you can improve testability and get test APIs
- your tes strategy will include automation from the start
7. commitment is essential
- it is easy for test automation to be designed as a side project. it wont get the resources it needs
- commitment ensures that test automation gets the resources, cooperation and attention that it needs
* from management
* from development
Share this
