Better Products through Science!

This past Saturday at Houston TechFest 2016 during an Agile development panel discussion, I was reminded of one Agile technique that we might not practice often enough. That is the use of the scientific method to help us create the right features for our customers.

Wikipedia says:

“The scientific method is an ongoing process, [in which we] come up with questions about the things we see or hear … and develop hypotheses [which can be] tested in various ways [with] experiments that gather empirical data. [The original hypotheses] may require refinement, alteration, expansion, or even rejection.”

That should sound familiar if you practice Agile and/or Scrum. It’s an iterative approach to learning and can be applied to how we meet the needs of our customers.

The Lean Startup is a book by Eric Ries which describes approaches new startups can use to build the right products for their customers. A central topic is the use of the scientific method to achieve validated learning.

Ries says in his book:

 “…We need the scientific method. In the Lean Startup model, every product, every feature, every marketing campaign – everything a startup does – is understood to be an experiment designed to achieve validated learning.”

Of course, we don’t all work at a startup. But we can apply some of the Lean Startup thinking to new products, projects, or features.

We use validated learning when we are uncertain about what to do or build. When we can show that we have discovered valuable truths about the needs of our customers by using empirical information, we have achieved validated learning. Ries says “It is the principal antidote to the lethal problem of achieving failure; successfully executing a plan that leads nowhere.”

Ries suggests we should add small pieces of functionality to our products then gauge our customers’ response. The pieces of functionality are experiments to test our hypotheses and gain quantifiable outcomes. We need to know if customers want the functionality or not. The empirical results of these tests help us achieve validated learning.

Again, this should sound familiar to anyone on a Scrum team using an iterative approach. We can design our iterations to help us understand what the customer wants and then get their feedback to make sure we are on the right track.

We also need to remember that our customers might surprise us. Failure to meet their needs is not bad when we work in short iterations. It just means we need to rethink our plans based on what we learned. Ries says, “This is one of the most important lessons of the scientific method: if you cannot fail, you cannot learn.”

How often do we work on a feature only to realize we’ve put so much time into it that we can’t afford to fail? Scrum allows and helps us create small pieces of functionality, so we should incrementally create features to determine if they are really needed. Use these increments as experiments to validate or refine our hypotheses.

Ries writes about an executive who asks his development teams to answer the following questions:

  1. Do consumers recognize that they have the problem you are trying to solve?
  2. If there was a solution, would they buy it?
  3. Would they buy it from us?
  4. Can we build a solution for that problem?

Many times we skip straight to the fourth question. Any assumed answers to the first three questions can result in an incorrect answer to the fourth, which leads to trouble.

This is the question that popped into my head during the panel discussion:

How much of what we are doing is based on customer feedback versus what the project plan tells us to do?

Let’s find ways to focus on customers and experiment to achieve validated learning.


Make Backlog Refinement More Efficient

Backlog refinement is necessary for any Scrum team. But our teams often get bogged down because our refinement sessions are inefficient or ineffective.

Do you find that you aren’t refining as many stories as you would like during refinement sessions?
Do you regularly struggle to have enough stories ready for sprints?
Do your stories lack good acceptance criteria?
Do you regularly miss key details which effect the success of your user stories?

In situations like this I follow one rule – If something isn’t working for the team, do something else. Find something that does work.

Backlog Refinement is a Pipeline

Backlog refinement acts like a pipeline to supply user stories that are ready for sprints. Stories enter the pipeline needing additional details. At the end of the pipeline they should meet your team’s Definition of Ready. In general, they should have a well understood title and value statement, have good acceptance criteria, be properly sized and estimated, and have tasks.

Teams which are new to Scrum may think of backlog refinement as something which is only done in refinement sessions. I know I did. However, my team became more efficient once we started thinking of backlog refinement as a continuous process.

Now, I’m not saying that we should spend all our time refining the backlog. I just want you to see that backlog refinement doesn’t need to be restricted to refinement sessions and that anyone on the team can work to improve story details at any time.

So how do we make the refinement pipeline more efficient?

Be Prepared

A big hindrance to effective backlog refinement is being unprepared for refinement sessions. Trying to refine a backlog that isn’t prioritized, trying to refine stories which have very few details, or creating new stories during refinement sessions is not efficient.

How can the Product Owner (PO) prepare?

Keeping the backlog in priority order is the responsibility of the PO. This allows anyone to review the most important stories at any time.

Teams are more efficient when the stories they are refining have a reasonable amount of detail. In my experience, it’s hard to create stories on the fly in a meeting. I believe any story presented for refinement should have a title, value statement, and acceptance criteria. This gives everyone the same starting point and reduces discussion due to different interpretations. Basically, the PO is anticipating and answering questions before they are asked.

The PO can also provide an agenda with a list of stories to cover in the refinement session. The agenda keeps the team focused on the important stories and also gives the development team the opportunity to review those stories beforehand.

How can the development team prepare?

The development team should do things which allow them to better understand and estimate the stories. As mentioned earlier, they can review the top priority stories beforehand. Come to refinement sessions prepared with questions or comments and add them to the stories for everyone to see.

Developers on some teams spend a few minutes each day reviewing the top priority stories. Again, the PO can help them make this effort productive by keeping the backlog prioritized.

I’m a big believer of referring to code to better estimate stories. Of course, we don’t want to get too deep into implementation details while refining stories. But, a general understanding of existing code and how it will be modified or used will help us create better user stories.

Be Focused

We all need to stay focused during our refinement sessions. I understand that can be hard if discussions get too detailed or the meeting drags on. My new phone gets more tempting as time goes on and my inbox starts to fill. However, it is important to be courteous to everyone in your refinement sessions. Put the phone down, the emails can wait, stay focused on story refinement.

How do we gain better focus?

The PO facilitates the sessions and keeps the discussions focused on refining the stories on the agenda. The PO should also time-box the refinement of each story and decide when to move on to the next one.

Story refinement is about getting answers to questions. Therefore, bring the things you need to answer those questions to your sessions. If you need to refer to documents, UI mock ups, or code, have them available. Bring your laptop so you can refer to them. Making assumptions without verification is a major cause of unsuccessful stories.

Don’t let your sessions run too long. It’s best that they are no longer that an hour because it’s hard to keep people focused for much longer. It’s even possible to have highly productive sessions in only 30 minutes.

Work between refinement Sessions

As I’ve said, it’s important to recognize that refinement doesn’t stop at refinement sessions. During any session, you will probably realize more work is needed to fully refinement some stories. Some of this work can be done prior to the next refinement session or next sprint planning.

What can we do between refinement sessions to improve user stories?

The PO should be doing the usual things – adjusting backlog priorities, improving story details, and adding or improving acceptance criteria. Essentially, they are improving stories based on the discussions we had during previous refinement sessions.

The development team can also improve stories between sessions. Many times during refinement sessions, we raise questions which require research or to review code to find the answers. If that can’t be done in the refinement session, assign someone to do this work before the next session. First, record the questions in the story. Then once someone finds the answers, also add them to the story. This makes the information available to everyone, we won’t forget it, and it’s ready for the next session.

The development team can also add tasks to stories between refinement sessions. I’ve found it can be hard to add tasks as a team because people think about tasks differently. Sometimes it’s more efficient for one person to add the tasks and then validate with the team later in a refinement session.

The development team will usually need to help the PO with creating and refining technical user stories. This might include writing the initial acceptance criteria. This work is also best done before the refinement sessions.


As we refine the backlog there are a couple of important things to remember. First, only the PO can change priorities in the backlog. Members of the development team might act on their behalf, but the PO must be involved in any priority changes. Second, the PO must be involved in any changes to acceptance criteria.

Good Enough

The key to good user stories is that they have enough details so we can successfully complete them in a sprint. We aren’t trying to write the perfect story because we’d be in never-ending refinement sessions. We do just enough story refinement so our stories are good enough.

Code Reviews

The article “Bad Software Architecture is a People Problem” in the September 2016 issue of Communications of the ACM touches on several of the development practices that we’ve talked about lately.

For example:
Bug Fixing
Coding Standards
Code Reviews

I’m glad to see code reviews mentioned because I think they can help solve some of the problems we face.

We benefit in multiple ways from code reviews which help us:
Double check our work before mistakes make it into the code base.
Verify design decisions and coding standards.
Transfer knowledge between developers.
Train new developers on team practices.
Learn from each other’s suggestions.

I particularly liked this statement:
“One of the best ways of bridging knowledge gaps on a team is to encourage sharing among team members. When other members review and give feedback, they learn the code, too.”

Another big reason for code reviews is to verify unit tests and automated acceptance tests. We should all be writing unit tests and acceptance tests for new code and when fixing defects. And it’s always helpful to have multiple people review these tests because in some cases it can be easy to get them wrong. This is especially true for complex business logic.

One of the worst things you can have in your solutions is unit tests that give false positives. In other words, tests that indicate your production code satisfies requirements or business rules when actually it doesn’t. Use code reviews to verify the logic of the tests and to verify that all the necessary tests have been written.

Your team should have a code review process for code commits. Some tools, like VS/TFS, automate the code review process. If your tools don’t, use email or IM tools. Decide which people on your team should review changes in the various functional areas of your product. Then include the responsible people in a review request before you commit to those areas. When you get a request, complete them in a timely manner – don’t let them sit.

Keep this in mind – Code reviews reduce defects.