Re: Testing work within a Sprint

Scrum for Team System - v2.x Discussions

Testing work within a Sprint

kbrown 03-25-2008, 10:10
I need advice from any veteran Scrum teams out there regarding best practices for testing with the Scrum TFS template.

Our developers set SBIs to "Ready For Test" and assign them to the tester as needed, but we're not sure where to take it from there. When a tester finds a problem with an SBI, where does this problem get recorded and how does it get communicated to the team?

We've entertained all sorts of ideas:

- Don't record it; verbal communication is good enough.

Even if the communication is _mainly_ verbal, it seems we're going to need to write something down one way or another. Either the tester will jot down notes to take to a developer (or to the team), or the developer will jot down a note indicating that he needs to look at the issue. If we need to write something, we'd prefer it to be in TFS so we can see it on the Sprint Backlog and estimate it.

- Record it in the SBI that the tester is testing.

This allows us to see the issue in TFS as part of the Sprint Backlog, but we're not sure how best to make use of it. Does the tester assign it back to a developer or leave it in the Sprint Backlog unassigned? What if a developer has it and the tester wants to add more to it (found another related issue while testing)? Maybe the tester should record all issues for an SBI before sending it back -- but that may negatively affect turnaround time. Also, it's not always clear which SBI a problem even pertains to.

- Create a new SBI for the problem.

This handles the case where a tester doesn't know which SBI a defect pertains to, but they still need to relate it to a PBI (which should be far easier). The problem with this approach is that we now have two SBIs for a single unit of work -- the original development task and now a defect on that task. Do both of them get set to unassigned? Do both of them get assigned directly to a developer? Which one do we estimate on? The redundancy involves makes this approach seem bad.

Some other questions we have:

1. How do you handle untestable SBIs?

It's often the case that a tester can't do anything meaningful with an SBI because they pertain to schema or configuration changes, issues for investigation, or other work that will not have a testable user-interface counterpart. Do testers just mark anything that they can't test as "Done"? Do developers skip "Ready for Test" for these and mark them "Done" (this one seems particularly dangerous).

2. During Sprint Planning, do your testers create SBIs in order to record the amount of time they think it will take to test the functionality implemented for various PBIs? This is what we're doing, since tester time was included in our PBI estimates, but I was curious if other teams handle this differently.

3. Do your testers work solely from the Acceptance Criteria contained within each PBI? We're trying this approach (in place of our typically thick test case documentation complete with procedures and screenshots) but it's very different from what we've done in the past and we'd like to hear if others have success with this.

Given that Scrum defines testing as a required element of the Sprint cycle, I'm sure these issues have been come across by teams that use the Scrum for TFS template. I appreciate any advice that you can give.

Re: Testing work within a Sprint

perb 03-28-2008, 7:51
1) Scrum Master or Tester moves SBIs to Done (or whatever feels practical), everyone can see what happens on the Dashboard anyway (especially on morning meetings).
2) No, we don't include testing in estimates or planning. But because bugs has to be fixed they affect the overall performance of the team when they show up as SBI causing a natural incorporation of time taken related to testing and bugfixing.
3) Yes, but we try to keep PBI as small as possible, splitting larger ones into smaller PBI. Testers also work in collaboration with the scrum master which can answer questions or make decisions on what is testable and not.

Most often we test the PBIs and not the individual SBIs, depending on the scope and type of functionality of course.

I wrote some more on how we handle the bugs in Scrum Dashboard here if you are interested:
http://www.codeplex.com/scrumdashboard/Thread/View.aspx?ThreadId=24217

Per Bjurström
EPiServer

Re: Testing work within a Sprint

kbrown 04-02-2008, 12:42
Thanks for your input. I'm in agreement with you on points 1 and 3 and considering the ramifications of 2.

More important for us at the moment is how exactly a tester is supposed to communicate problems found with software developed in-Sprint. To take a quote from the process guidance:

"The critical change is to treat a Sprint Task that hasn’t reached a stage of 'Done' as an incomplete task rather than as a bug, with the task status field used to track its incomplete state, e.g. 'In Progress' – being worked on by the developer, 'Ready for Test' - developed code ready for the QA to test and provide feedback."

It's these last _three_ words that is starting to cause real problems for our team -- "and provide feedback". How do testers provide feedback to developers when they find something wrong?

Prior to Scrum, our testers would create a defect report in Test Track Pro. This would include steps to reproduce and a description of the problem, and it could then be assigned to a developer by the project manager.

With the TFS template for Scrum, we're really not sure what to do. The testers don't like any of the solutions that have been dreamed up because they are all quite confusing.

Does _anyone_ out there actually record details about defects that are discovered in-Sprint? If so, where? If not, am I overlooking some essential Scrumness here? The only possible way I could imagine doing away with recording defect details would be if:

- The tester immediately finds a developer upon discovery of an In-Sprint defect.
- The developer immediately shelves current feature development and works on the bug with the tester present for details on how to reproduce.

Even then, what if there are no developers available? What if the steps to reproduce were so complicated that the tester now has to go scribble them down someplace in order to ensure that the steps aren't overlooked during the next test pass?

Kevin

Re: Testing work within a Sprint

DancesWithBamboo 04-05-2008, 3:14
This is such a timely post. I have just been working with the QA team for a couple weeks now to come up with a plean to all those questions. We have had had many good debates and some got heated as it is always hard to change people's approaches after they have used them so long. One extra big question that came up for us that you didn't mention was the QA work product, Our QA guys have always viewed the bug db as their list of work accomplished. If we took that away and no longer tracked in-sprint bugs, what is it they were producing?

Here is some of the thougts and decisions we hade so far:

Defining a "bug" was the first step becuase you would be amazed at how many different ideas there are even in a small gourp. We decided that a bug is an unexpected outsome in production software. This means that a defect in-sprint is not a bug. Bugs are things that the users find while working and they get recorded as PBI's to be scheduled for another sprint (possibly a concurrent emergency one).

One of the things that led to this definition was that all developers have defects in the first code they write. hence, why we unit test right away. At what point do you start recording those defects? If you take it to far you could say that after I compile, everything gets recorded. Obvious insanity would ensue. Hence we set the hard line at production.

I disagree with the other poster that testing is not estimated. Our QA guys are responsible for making testing estimates as SBIs right along with the dev task SBIs. Now one thing here that we are doing is saying that SBIs are not testable units by the QA team. SBIs are unit tested by devs and as such that testing is estimated in by the dev in their task. So QA is making SBIs for testing of PBIs (feature units). One test SBI may in fact cover all of the dev task SBIs for a given PBI. Testers estimate their time to write and execute their scripts. My question is how could you not estimate tester time? They are part of the SCRUM team and their time must be scheduled into the project, estimated, tracked, and evaluated as well.

If a defect is found on a PBI during a sprint, that is not a bug. It is unfinished development work. Just as if the developer was debugging his own app, that is really what the tester is doing (just more methodically). This is where I wish the "Ready For Test" state had been added to the PBI statuses, not the SBIs. Our only communication problem so far was how does a tester know when the feature is ready for them to test. What has worked so far is to cover that in the morning scrum and with nightly integration build release notes. reports.

Yes, the tester runs over (or usually IM) and the dev drops what they are doing to hear the story. Why do this? Because that is the moment at which the defect is cheapest to fix. At this point there are 3 outcomes. One, and most common, the defect is super easy and the dev fixes it right then and the tester knows it will be ready the next day. Two, the bug is more intense than that and won't be fixed for the next build. At this point it is put up on a tack board on a post it with the details and the name of the Dev who is reponsible for the atrocity. Or three, it is huge (> couple hours). Here it still goes on the board, but some SBI is going to get set back to In-Progress and the "Work Remainig" will be re-set to a > 0 value. This then shows up on the burndown as an upward blip in the nice downward trending graph. Explanations then ensue as to why such a large defect left the devs cube.

I see many advantages to this. As a dev, if you don't want to be bothered all the time; then test your own ***. As the severity of the defect increases, so does the public visibility of it. Long lists of bug-flinging are not generated that need to be constantly evaluated, closed out, looked at, filtered, and hated. At my last job, the "bug" became a long running conversation; more often then not performed by reassinging it back and forth with more comments. Lists of bugs are inventory; be LEAN; scrap the lists and reduce costs. Important user discovered actual "bugs" can be evaluated and scheduled in standard sprint planning sessions. Testers now look at their work product as being the suites of test cases they build to cover all of the code rather than bug lists. This has turned them from being "wasters" to producing valuable work materials. The TFS tool does not become a peer-to-peer communications platform. It is designed for planning, estimating, and status/trend reportingand so that is what we use it for.

OK, that was a long ramble. Looking forward to some other thoughts out there on this.

Re: Testing work within a Sprint

kbrown 04-05-2008, 2:18
Thanks for that great response. It looks like you're going down a path very similar to the one my team is on (at the moment). I wholeheartedly agree with you that I would have liked to see "Ready for Test" at the PBI level. I think it's not ideal according to Scrum, since we're basically supposed to work from the Sprint Backlog during a Sprint, but it would solve some of the problems we're having.

I like your lean approach. Early on, we had a lot of success with the tester just coming in to interrupt the developer when a bug was found. Details usually didn't need to be tracked anywhere and the problem still got resolved. However, it there was concern all around that this wouldn't scale well and I could see it becoming a problem.

So, we came up with a tentative solution that is along the lines of putting bugs up on the taskboard that you mentioned. We do this for all defects (currently) simply because people like hard and fast rules wherever possible -- thereby avoiding the question as to whether or not a problem needs to be reported in TFS or just communicated to a developer verbally. I'm not saying we'll stick with this, but here's what we're trying:

1) During Sprint Planning, the testers create "Test SBIs" to estimate time to create and run through test cases for the related PBI. I believe this is similar to what you're doing.

2) These "Test SBIs", like all the other SBIs, are initially unassigned.

3) When a developer begins work on an SBI, he/she grabs the "Test SBI" for the related PBI as well.

4) Each day, the developer changes his/her time estimates for the SBIs, _not_ including the "Test SBI".

5) The developer can check-in/build SBIs at any time -- but when _all_ SBIs for a given PBI are "Ready for Test", then the developer also submits the "Test SBI" as "Ready for Test".

6) This "Test SBI" showing up in the tester's queue is the trigger for testing. It indicates to the tester in an unambiguous way that the PBIs functionality is ready for testing. This essentially gives us the "Ready for Test" indicator at the PBI level.

7) The tester updates the Work Remaining estimates each day for any "Test SBI" he/she is working through.

8) If the tester finds something wrong, the details are recorded in the "Test SBI" and it is bounced back to the developer.

9) The developer _never_ touches the "Test SBI" Work Remaining estimate. This means that we don't estimate and plan for defects. The effect of this is that the burndown chart flattens (and our velocity decreases) as defect rates go up. I think this is alright since it makes sense to not consider defect fixing part of "productive" work (we shouldn't be happy when we spend 5 days fixing a defect -- it could have been 5 days working on features).

Ok, there's a long ramble to match yours! I hope it made sense. Initial feedback on this from the testers is that it's much better and more clear than what we were trying previously -- which entailed recording details in whatever SBI the tester could find that made sense and bouncing it back to the developer.

Feedback from developers so far has been that it's clunky and likely to not get followed very well. For this reason, I'll definitely give further consideration to having testers just communicate information straight to a developer for the majority of simple cases. I agree that it would be best to not try to make TFS a "peer-to-peer communications platform".

Thanks for the input.

Kevin

Re: Testing work within a Sprint

perb 04-07-2008, 10:11
Very interesting discusson, managing tasks for testing and bugs is not easy at all.

1) We have a offshore testing team and have tried having testers assigned in the Scrum teams but it isn't working, that's why we dont include testing estimates made the the testing team.

2) I totally agree that bugs found in a PBI during development should be classified as "unfinished development work", that's how we do it. It should always pay off to write high quality from the start, and fix any bugs as they appear. We normally create a "Bugs from previous sprints" PBI for every sprint to track bugs found after the previous sprint is closed (due to the fact that having all testing done within the sprint does not work for us, and new bugs are bound to be found when product management is starting to play around). Bugs found in a PBI in the current sprint are created directly in that PBI as tasks.

3) We currently don't use WIT Bug as PBIs, but rather as SBIs due to how Scrum v1.3 worked (so I will probably change in "Scrum Dashboard" that "Create bug" in a sprint is actually just a task with "Bug:" prefix. Much easier. And if you want to work with the real bugs they appear when adding PBIs to a sprints but you will have to create them yourself.)

4) We handle bugs in production code in a separate project and importing them into time-boxed PBIs using the Dashboard.

Per Bjurström
EPiServer

Re: Testing work within a Sprint

S C Potter 04-10-2008, 10:05
Awesome thread, and incredibly timely for our project. We've been struggling with the new Bug WIT for about a month now and at the same time I was reading this thread the four senior Dev and QA folks came up with that "we want to go over this" look. I told them I thought I knew why they were there and that I was sure the answer to all their questions was on my screen. I'd just read this, which is very insighful and helped out discussion a huge ammount:

DancesWithBamboo wrote:
Defining a "bug" was the first step...We decided that a bug is an unexpected outsome in production software. This means that a defect in-sprint is not a bug. ...One of the things that led to this definition was that all developers have defects in the first code they write. hence, why we unit test right away. At what point do you start recording those defects? If you take it to far you could say that after I compile, everything gets recorded. Obvious insanity would ensue. Hence we set the hard line at production.

We decided to modified the defiinition to "an unexpected outsome in test not easily tracable to one PBI or any unexpected outsome in production". There was a concern about how to capture a regression type defect in QA can't easily trace back to a specific PBI, and we thought this worked for that.

I hadn't gotten to this yet:

kbrown wrote:
I'm not saying we'll stick with this, but here's what we're trying: [9 steps]... I hope it made sense.

That makes sense to us, because in our discussion this is exactly what we came up with independantly. The whole team got a good laugh out of this. The dev feedback is good to know. Our typical practice is for QA to talk to / IM the developer before writing anything up, so we'll see how this goes. We only plan to write up items that QA or Dev feel they need documented.

In summary, for those looking for testing practices that others have found usable, our team has is generally in agreement with the parents referenced above, and seem to be having fair success with this. Now I'm going to head over to the feature request thread and sign up to the PBI "Ready To Test" state request.

-
Scott