Coding

Writing a Good Bug Report


A bad bug report can be worse than no bug report. A bad bug report leads to frustrated developers, which leads to developers marking bugs as “Cannot Reproduce” and sending it back, which leads to bug tennis, where the developer and the QA tester whip the bug back and forth to each other, crying “It’s broken!” and “No, it’s not!” all day. Good bug reports require only a little more effort to produce and make lives an order of magnitude easier for everyone in QA and development.

As an example of just how bad things can get, here’s an example of a terrible bug report:

I make an order and click Next but get told error about not supporting billing.

As a developer looking to test that bug, I would have absolutely no idea how to start. I’ve likely submitted dozens of orders already and none of them have failed, least of all with some ambiguous error regarding the billing system. This is the kind of hair-pulling complaint I expect to see from my Mom when she’s having trouble buying from Amazon or setting up online banking.

Bug reports absolutely require 4 pieces of information:

  1. Summary
  2. Steps to reproduce
  3. What you expected to see
  4. What you actually saw

Let’s examine them individually.

1) Summary

Anyone scanning through long lists of bugs need to be able to instantly identify what a bug is about, so the summary has to be concise and uniquely identify the bug in question. Try to limit to 80 characters or less as a guideline. Avoid the urge to go into detail here – this can be included in the actual bug report itself.

Example: “Cannot submit Mastercard transactions with card expiration dates after 08/11”

2) Steps to reproduce

This is probably the most critical part of the bug report. A bug that can’t be found can’t be fixed, and a bug without listed repro steps is very hard to find. Provide these steps in an ordered list, ensuring that they are easy to follow and contain ALL information required to reproduce the bug, including setup steps that need to be taken to trigger it (like turning an option on or off).

Example:

  1. Add Pink Cat Collar #3444 to cart.
  2. Click check-out button.
  3. Use development Mastercard (3557) with expiration date 08/12 to submit transaction.

3) What you expected to see

What you should have seen if there was no bug.

Example: “Order complete/receipt page.”

4) What you actually saw

What you saw due to the bug. If the program spits out any debug information, copy/paste it into your bug report. If it’s over 20 lines long, or if you need to include an entire HTML page or file, include it as an attachment.

Example: “PHP Error: ‘Line 42: Overflow in $cardChecksum'”

Optional – Further description

If you have more stuff to say about the bug than what’s listed above, don’t try to mash it into the fields above. Place it under the header “Additional Information” or similar. If you can find conditions the bug suspiciously doesn’t happen under, list them here. Don’t muddy the waters with ideas on how to fix the bug here, unless you’re absolutely certain you know what’s wrong. Your job is not to be the doctor and diagnose, but to be the patient and report symptoms.

Sloppy writing and conjecture isn’t encouraged anywhere in a bug report, but the one place you can get away with it is in the “Further description” field – provide as much information as you think is relevant. Developers would far prefer to have to read extra material than have to hunt you down to figure out just what your bug report meant.

Optional – Severity/Priority

The severity or priority of a bug indicates how damaging it is to the product. A properly set severity level is important, as often, bugs can either be fixed or not fixed based on a set schedule, and the severity level is used to identify which fixes make it in and which don’t. QA Testers are rarely the people who set severity levels of bugs, but I find they’re so often misunderstood that covering them here would probably help.

There are many schools of thought on how to properly categorize severity levels, but in the interests of brevity I’ll just cover one, a fairly common 5-level classification:

  • Severity level: Blocker Blocker – Blocks development or testing. Examples: Installer doesn’t work/application fails to launch.
  • Severity level: Critical Critical – Loss of data, crashes, public exposure of private data. Examples: Renaming an account deletes it, listing the credit cards on your account lists all credit cards on all accounts.
  • Severity level: Major Major – Major loss of function, or function loss where no work-around exists. Examples: Layering function in a graphics editor doesn’t work, or unable to change font in a text editor.
  • Severity level: Minor Minor – Minor loss of function, or function loss wher an easy work-around exists. Examples: “Close” button on tabs don’t work, but Menu->Close does, italics doesn’t work with little-used font.
  • Severity level: Trivial Trivial – Cosmetic problem that does not impede functionality. Examples: “Exit when finished” translates to “Please leave now” in Korean, where you sell five units a year, border around statistics doesn’t show up under certain browsers.

Keep in mind that severity doesn’t necessarily equate to priority, and these are typically separate fields in many bug tracking systems. A Critical bug that affects only one customer worth $200/month should definately have a lower priority than a Major bug that affects multiple customers worth multiple thousands.

Summary

The aim of the bug report is to make the failure you encountered as instantly obvious to the developer assigned to fix it as it is to you. Approach the whole thing scientifically. If I follow the same steps as you followed, I should get the same result, on which we can build (i.e. fix). You should aim to be specific and clear about what you’re reporting (never “it” or “the textbox”, use “the cursor” or “the address textbox” instead), concise in your summary and repro steps, and accurate.

With only a little more effort and attention paid to including the right information, you can turn a rushed, bad bug report into a thoughtful, good bug report. Your developers will thank you, your managers will thank you, heck, Amnesty International will probably thank you. Okay, well maybe not, but still, it’s a good idea.

Coding
Gulp.js – an AMAZING build system!
Coding
Rage-quit support for fish shell
Coding
Code faster with simple Sublime Text improvements
  • geoff

    geoffgeoff

    Author

    Do your testers have a predetermined set of test procedures to go through, and very little reason or opportunity to just play around with the system?

    I ask because often I will have a crash or interface screw up of some sort when just playing around, and it’s not immediately obvious what causes the problem. Your bug report instructions don’t seem to acknowledge that intermittant and hard-to-reproduce problems can occur. How would one report a crash that happens sometimes, but for which a specific series of steps to reproduce hasn’t or can’t be found?

    I suppose your assumptions are a product of your web-based (AFAIK) coding environment, in which things only tend to go wrong after a relatively few discrete user actions that makes it easier to narrow down the cause, but you might not want to present instructions with an implication that they are universally applicable if they aren’t…


  • Dan Hulton

    Dan HultonDan Hulton

    Author

    Obviously, there’s going to be times when you can’t reproduce a bug at first blush. However, it always helps to stick to the format, even if it’s “Steps to reproduce: Unknown”.

    Further testing and discovery by both the tester and developer can uncover them, but the point is to be as explicit as possible, and maintain a common format that people expect to see.

    And actually, my assumptions come from a lot of research and recommendations from other, smarter people than I, who are working on all manner of programs. The information I described above is ALWAYS necessary to fix a bug, and testers should do their utmost to be able to provide it before it gets to the devlopers.