Writing a Good Bug Report
A bad bug report can be worse than no bug report. A bad bug report leads to frustrated developers, which leads to developers marking bugs as “Cannot Reproduce” and sending it back, which leads to bug tennis, where the developer and the QA tester whip the bug back and forth to each other, crying “It’s broken!” and “No, it’s not!” all day. Good bug reports require only a little more effort to produce and make lives an order of magnitude easier for everyone in QA and development.
As an example of just how bad things can get, here’s an example of a terrible bug report:
I make an order and click Next but get told error about not supporting billing.
As a developer looking to test that bug, I would have absolutely no idea how to start. I’ve likely submitted dozens of orders already and none of them have failed, least of all with some ambiguous error regarding the billing system. This is the kind of hair-pulling complaint I expect to see from my Mom when she’s having trouble buying from Amazon or setting up online banking.
Bug reports absolutely require 4 pieces of information:
- Steps to reproduce
- What you expected to see
- What you actually saw
Let’s examine them individually.
Anyone scanning through long lists of bugs need to be able to instantly identify what a bug is about, so the summary has to be concise and uniquely identify the bug in question. Try to limit to 80 characters or less as a guideline. Avoid the urge to go into detail here – this can be included in the actual bug report itself.
Example: “Cannot submit Mastercard transactions with card expiration dates after 08/11”
2) Steps to reproduce
This is probably the most critical part of the bug report. A bug that can’t be found can’t be fixed, and a bug without listed repro steps is very hard to find. Provide these steps in an ordered list, ensuring that they are easy to follow and contain ALL information required to reproduce the bug, including setup steps that need to be taken to trigger it (like turning an option on or off).
- Add Pink Cat Collar #3444 to cart.
- Click check-out button.
- Use development Mastercard (3557) with expiration date 08/12 to submit transaction.
3) What you expected to see
What you should have seen if there was no bug.
Example: “Order complete/receipt page.”
4) What you actually saw
What you saw due to the bug. If the program spits out any debug information, copy/paste it into your bug report. If it’s over 20 lines long, or if you need to include an entire HTML page or file, include it as an attachment.
Example: “PHP Error: ‘Line 42: Overflow in $cardChecksum'”
Optional – Further description
If you have more stuff to say about the bug than what’s listed above, don’t try to mash it into the fields above. Place it under the header “Additional Information” or similar. If you can find conditions the bug suspiciously doesn’t happen under, list them here. Don’t muddy the waters with ideas on how to fix the bug here, unless you’re absolutely certain you know what’s wrong. Your job is not to be the doctor and diagnose, but to be the patient and report symptoms.
Sloppy writing and conjecture isn’t encouraged anywhere in a bug report, but the one place you can get away with it is in the “Further description” field – provide as much information as you think is relevant. Developers would far prefer to have to read extra material than have to hunt you down to figure out just what your bug report meant.
Optional – Severity/Priority
The severity or priority of a bug indicates how damaging it is to the product. A properly set severity level is important, as often, bugs can either be fixed or not fixed based on a set schedule, and the severity level is used to identify which fixes make it in and which don’t. QA Testers are rarely the people who set severity levels of bugs, but I find they’re so often misunderstood that covering them here would probably help.
There are many schools of thought on how to properly categorize severity levels, but in the interests of brevity I’ll just cover one, a fairly common 5-level classification:
- Blocker – Blocks development or testing. Examples: Installer doesn’t work/application fails to launch.
- Critical – Loss of data, crashes, public exposure of private data. Examples: Renaming an account deletes it, listing the credit cards on your account lists all credit cards on all accounts.
- Major – Major loss of function, or function loss where no work-around exists. Examples: Layering function in a graphics editor doesn’t work, or unable to change font in a text editor.
- Minor – Minor loss of function, or function loss wher an easy work-around exists. Examples: “Close” button on tabs don’t work, but Menu->Close does, italics doesn’t work with little-used font.
- Trivial – Cosmetic problem that does not impede functionality. Examples: “Exit when finished” translates to “Please leave now” in Korean, where you sell five units a year, border around statistics doesn’t show up under certain browsers.
Keep in mind that severity doesn’t necessarily equate to priority, and these are typically separate fields in many bug tracking systems. A Critical bug that affects only one customer worth $200/month should definately have a lower priority than a Major bug that affects multiple customers worth multiple thousands.
The aim of the bug report is to make the failure you encountered as instantly obvious to the developer assigned to fix it as it is to you. Approach the whole thing scientifically. If I follow the same steps as you followed, I should get the same result, on which we can build (i.e. fix). You should aim to be specific and clear about what you’re reporting (never “it” or “the textbox”, use “the cursor” or “the address textbox” instead), concise in your summary and repro steps, and accurate.
With only a little more effort and attention paid to including the right information, you can turn a rushed, bad bug report into a thoughtful, good bug report. Your developers will thank you, your managers will thank you, heck, Amnesty International will probably thank you. Okay, well maybe not, but still, it’s a good idea.