In the news has been Facebook’s announcement that it was paying a bounty for security bugs reported by the public. Details of Facebook’s program are here. It’s a variation on processes long used by web and software companies. Done well, beta programs help engineering find bugs they might otherwise have released to the public. Done poorly, a beta program is a quagmire of time and resources, providing you no new insights into your product.
When I worked at Intuit, the engineering group had in place a comprehensive beta program. Intuit sent out hundreds to thousands of beta versions of Quicken. With an organized and streamlined system for getting feedback to engineering, Intuit’s model became renowed within the industry. Of course it continues: Intuit now has a entire web portal devoted to its beta program. The beta program was also followed up by a post release ‘Follow Me Home’ program. Intuit engineers literally followed Quicken purchasers home (with their consent, of cousre), and watched them install and use Quicken. One story: a new user was confused by the opening screen option to ‘Register’: did this mean register the software or use the Quicken checkbook register? This ambiguity was fixed in the next Quicken release.
At Frame there was a built in beta group: the documentation team used FrameMaker to author all Frame related books and manuals. When a new version of FrameMaker was nearing completion, the doc team would start to use that new, not yet released version of FrameMaker. Of course, even having a beta group in the same building, at the same company, on the same floor, we still needed processes and protocols to make sure any bugs or issues found were promptly and clearly reported to engineering. There was also an external beta program for FrameMaker.
At Visioneer in addition to a software beta program for Paperport there was a concurrent hardware beta program for the Strobe sheetfed scanner. Here are the descendants of that original product. With hardware there was more overhead: shipping and keeping track of all the hardware units; replacing defective units (there were a few); getting clear, useful, and reproducible bug reports (a combined hardware/software program is much more complex than simply a pure software program); then getting all the hardware back when the program was over. It was one of the more logistically challenging beta programs I have been involved with.
The increasing trend of delivering software over the web, the agile model of development, and the increased role of the browser at the center of computing activity has changed the nature of betas. And the public’s willingness to accept a lower quality experience. More and more products and services come out with a ‘beta’ qualifier. Users and customers become de facto beta sites, used to find bugs that should have been found prior to release. Facebook’s attempt to turn hackers into betas and bug finders is another example of this process aikido.
Don’t confuse the beta programs with company wide bugathons or bug-o-ramas. These events are intended to get the rest of the company involved with, and exposed to, what engineering has been working on for months. Usually done for a few hours on a Saturday, other departments (technical support, documentation, marketing, sales, etc.) are invited to use the soon to be release product or service. In my experience these events do not turn up any critical bugs. Instead the company bug hunt is a combination of motivational rally and marginally useful project milestone.
If you’re running a beta program, here are a few things to keep in mind:
- Make sure it’s worth you time to set up and run. If you’re too close to releasing and won’t be able to incorporate any feedback, spend your time tuning and focusing your internal testing.
- Make sure your software or website are ready for beta; releasing too early will mean all your betas will be reporting the same bugs.
- Pre-qualify your beta users so you know you are getting knowledgeable users and domain experts, who will be a good auxillary to your qa team.
- Be realistic about the amount of feedback you will get; beta sites often seem enthusiastic, but they have jobs and lives too, and may not devote much time to testing.
- Tell your betas about known bugs; this may prevent them from reporting known issues.
- Goes without saying: make sure you have a streamlined system for capturing all information needed to reproduce a bug.
- Designate a single point of contact for the beta sites.
- Give clear wrap up instruction to your betas: how to clean systems of the beta software, return hardware, remove plug-ins, etc.
- Thank each and all betas, regardless of their degree of participation. If they are eligible for free products or upgrades, make sure they get that.
- And, if you’re paying them a bounty, like Facebook, cut them a check.