Warm Springs Computer Works
Fremont, California

This application note is the first step in formalizing the rules, topics, and experiences distilled from a career centered on automatic test equipment.  The following list of topics forms an application note on how to approach the design of an automatic test system.  The plan is to expand this application note into a book in which a page or two is devoted to each topic.  Supporting material and examples will be added.

All suggestions, comments, and critiques are welcome. E-mail to new-book@wscw.com.

1. The Essentials of Automated Testing

Description of what automated testing is
Engineering effort comparable to product development
A necessary part of the product's life cycle
Part of the development and manufacturing process
More than just extending a manual process
Integrated into the production process
Integrated into the corporate culture and business model
Chip
Board
Box
System
Reliability
Repeatability
Data capture
Trend capture and analysis
Manufacturing process control and feedback
Warranty documentation
Manufacturing test verification
Repair requirements
Diagnostic aid
Directed probing
Troubleshooting
Fault isolation
Tracking of design changes
Product improvement
Product evolution
Product growth
 
 
2. Types of Testing

Bare-board faults
Individual component testing
Component testing after loading onto board
Board loading (in circuit test)
Active board tests
Power-on infant mortality
Burn-in performance monitoring
Validation after burn-in
Tracking changes in performance
A last step in the actual manufacturing process
Matched-set selection
Feedback-driven circuit adjustments
Adjusting of component values
Selection of components for offsetting tolerances
Open box
Closed box
Design verification
Manufacturing verification
Product evaluation
Acceptance testing
Directed probing for fault isolation
Extent of testing
Test to fault or exhaustive test
Degree of testing
Test each combination or test each element
 
 
3. Types of Testers

Turn-key
Custom-built
Standard commercial equipment
Rack-and-stack
Fixtures and adapters
Cabling
Mix and match
Degree of operator involvement
 
 
4. Test System Technology

Discrete components
Discrete instruments
Traditional instrumentation
Customizing of standard equipment
Special-purpose equipment
Fixtures and adapters
Test cables
Long signal paths
Switching systems for routing power and signal
Concessions to automated techniques
Computer control
Computer plug-in components
Plug-in instruments
Remote instrument clusters
Remote control
Software as the instrument
Serial port, Parallel port, Universal Serial Bus, Fiber Optic
GPIB, VME, VXI
Ethernet, LAN, WAN, Internet
Other interfaces
 
 
5. Test System Economics

Retaining a perspective on the cost and value of individual tests
Test duration and throughput
Load time and test time
Multiple fixtures
Build or buy decisions
High-priced and low-priced instruments
Locking of resources
Multiple threads, multiple processes, multiple processors
Sharing of high-priced instruments among multiple low-priced systems
Retain perspective on performing some tests manually or not at al
Cost of implementing a test in an automatic mode instead of a manual mode
Performing a test manually on the workbench and then reporting the results
Creative insight into alternative ways to gather essential data
Creative ways to determine actual UUT operation
Exhaustive test vs. sampling, by UUT or by lot
 
 
6. Test System Calibration

As individual instruments or components
As an integrated system
Transfer standards
Self-calibration from internal or external standards
Fixture and cable calibration
Calibration fixture
Calibration procedure
Calibration interval
Daily confidence check
Confidence check after a failed test
Diagnostics for isolation of test system malfunctions
Test system repair procedure
Test system operation at a reduced capability
Calibration of the computer as an instrument
Program execution time
Software time-dependencies altered by multiple programs and the operating system
Computer clock time
Computer software validity
Data file of calibration data for adjusting test measurements
 
 
7. Operator Requirements

Skill level
Training
Training methods
Certifications
Physical activities
Physical limitations
Unattended operation
 
 
8. Operator Interface

Step-by-step instructions
Separate or on-line instructions and tutorials
Separate or on-line manuals or books
Help file
Graphical displays
Model of what to look for
Auxiliary displays
Audio output for commands
Speech recognition and speech learning
Operator input devices, mouse, touch screen, keyboard, numeric keypad
Auxiliary devices, button, foot-switch, key-switch, token, dongle
Enforced hands-clear interlocks
 
 
9. Operator Actions

Load and unload the UUT
Initiate test procedure
Connecting and disconnecting
Probing
Interpretation of results
Reporting of events, conditions, observations
Feedback to operator
Operator hazards
Operator safety
 
 
10. Test Environment

Voltage and current sources
Signal sources
Loads
Energy storage in capacitors, inductors, batteries, mechanics
Power dissipation
Interaction among sources and sinks
EMF radiation
EMF susceptibility
Light
Sound
Switching, order, path, intermediate states
Heating and cooling
Liquids and gasses
Air, water, glycol, oil, nitrogen, helium
Inert and reactive
Chemicals
Vacuum
Hazards to and from the operator
Mass
Mechanical
Shielding
Safety grounds
Signal grounds
Circuit and instrument protections
Tolerant of UUT faults
Tolerant of test system faults
 
 
11. Test Requirements and Standards

Origin of specifications
Customer's specification
Manufacturer's specification
Product specification
Acceptance specification
Appropriateness of specifications
Beginning of life values
End of life values
Return-to-service values
Tolerance proportional to use or wear
Thoroughness of specifications
References to recognized standards for safety and test methods
Sources for the standards and the source's address
ISO 9000
Government requirements, OSHA, NASA, Military, FDA, NTSB
 
12. UUT Considerations

Connections
Mechanical
Electrical
Operations
Test duration
Steady state or intermittent
Transient tests
Stress testing
Testing outside normal operating ranges
Tests that cannot be performed automatically
Tests that must be performed automatically
Testing provisions as part of the UUT's design
Designed for testing
Intermediate or internal test-point access
Test connectors
Additional pins on regular connectors
Built-in-test (BIT)
Normal operating modes
Failure modes
Parts that are inaccessible from standard connections
Parts that may or may not be accessible to probes
Parts that can fail and not affect a test result
 
 
13. Risk Factors

Risks that are known
Risks that are unknown
Surprise factors
What you "know" that "is not so"
Alternatives
Prototype to validate a design approach
Engineering model and prototype
Production prototype
Expectations beyond technology
New technology
Single or multiple test systems
 
 
14. Test System Design Relative to Hardware

Different orientation from product design
One- or few-of-a-kind engineering
Whether to model after bench testing
Single- or multiple-UUT capable
Specifications of the UUT
Specifications of the test equipment
Boundary between testing the UUT and testing the tester
Manual operations
Automatic operations
Indicator lights
UUT indicators
Panel meters
Error budget
Sources of errors
Accuracy ratios
Wiring losses
Wiring lengths
Noise sources
Noise susceptibility
Propagation time constraints
Physical constraints
Test environment vs. normal-use environment
Automated mechanical actions
Changes as test process evolves
Changes as tests are added or removed
Computer vision for image processing
Motion control
Interlocks and timeouts
ID of cables, adapters, fixtures, proper connections
 
 
15. Unexpected Shutdown

Overload
Timeout
Power failure
Orderly shutdown vs. catastrophic shutdown
Residual effects
Release of UUT from fixture
Stored energy
Residual heat if cooling is interrupted
No-power means of clearing the test system and rendering it safe
Purging of liquids and gasses
Standby power, UPS, battery, flashlight
Manual and no-power means to extract and safeguard UUT
 
 
16. Test System Design Relative to Software

Customer endorsement
Test philosophy
Operator mode
Supervisor mode
Decision tree
Abort, stop, pause, or continue on...
      First fail, fail-count, specific failure, combinations of failures
Abort procedure
Force fail or pass
Order of tests
Low-cost tests
Low-value tests
High-cost tests
High-value tests
Subset of tests
Benign tests
Stressful tests
Dangerous tests
Simulation mode
Changes as test process evolves
Changes as tests are added or removed
 
 
17. Test System Construction

Proof of concept
Prototype
Make it work first
Make it neat second
Building multiple testers
Differences among different testers
Software identification of host system
Proof of proper design
Proof of proper construction
 
 
18. Test Data

UUT ID
Test system ID
Fixture ID
Adapter ID
Cable(s) ID
Operator ID
Supervisor ID
Facility ID
Operator commentary before, during, and after testing
Time and date of test start
Test duration
Power-on duration
Power on-off cycle count
Measured data
Calculated data
Inferred data
Operator report of an observed condition
Data reduction
Post-test data manipulation
Data interpretation
Data formatting and presentation
Data archiving and preservation
Exporting to other programs (database, graphics, statistics)
 
 
19. Post-Test Actions

Assign serial numbers
Correlation to work-in-process numbers
Adjust inventory levels
Automated marking
Automated conveyor or bin selection
Ship, rework, repair, scrap, salvage
 
 
20. Security

Sensitivity as related to proprietary or classified data
Test procedures and test reports as sensitive documents
Performing tests without revealing sensitive data
Clearing instruments of sensitive data
Associating sensitive information with a UUT
Choice of operator
Trust of operator
Access control built into the software
Access tokens or dongles
Limiting program access
Limiting physical access to the computer
Limiting access to data
Security in the presence and absence of power
Unexpected events
UUT access
 
 
21. Test System Software

Proper software engineering is vital
Software development environment
Software development standards
Final arbiter of system function and performance
Host computer
Operating system
Test executive
Test sequencer
Test generator
Programming languages
     Basic, Visual Basic, HP Basic, Test Basic, ATS Basic
     Fortran, C, C++, Ada, Atlas
     LabView, LabWindows/CVI, Agilent-VEE, DT-VEE
     Test Point, DASYLab, Labtech, SnapMaster
Software requirements
Input data
Output data
Measurement data
Use of calibration data to adjust measured data
Pass and fail limits in program or data file
Hardware or data file means to identify host system
Media
Networking
Coordination with other activities
Test result interpretation
Test results archiving
Origin of test-input data
Operator interface
Hardware-software interfaces
Instrument interfaces
Interface hardware
Device drivers
Instrument drivers
Interfacing to third-party software packages for data export
 
 
22. Long-Term Tests

Different approach from short-term tests
Test program started or re-activated on a time schedule
Test program designed to work and report valid data despite interruptions and re-starts
Single program vs. multiple programs for measurements at different intervals
Test state and status recorded in a file
Open, write or append, and close data files.
 
 
23. Configuration Management

Operating system software
Test, calibration, and validation software
Test instruments
Test system adapters, cables, and fixtures
UUT
Maintaining compatibility with past and future versions of everything
Log book of all activities
Always retain all old versions of specifications and drawings
 
 
24. Company Politics and Experiences

Opinions and attitudes
Prior experiences, good and bad
Existing test system or equipment
Existing test software
Budgets and schedules
How good is good enough?
Lessons learned
 
 
25. Appendix

About the Author
About the Company
 
 
26. Notes

 
27. Glossary

 
28. Bibliography

Brindley, Kieth, Automatic Test Equipment. Oxford: Butterworth-Heinemann Ltd., 1991.

Kaner, Chem, Jack Falk, Hung Quoc Nguyen, Testing Computer Software, Second Edition. Boston: International Thompson Computer Press, 1993.

Lenk, Ron, Practical Design of Power Supplies. Piscataway: NJ, IEEE Press, 1998.

Liguori, Fred, Automatic Test Equipment: Hardware, Software, and Management. New York: IEEE Press, 1974.

Stover, Allan C., ATE: Automatic Test Equipment. New York: McGraw-Hill Book Company, 1984.
 
 

29. Index

All suggestions, comments, and critiques are welcome. E-mail to new-book@wscw.com.