April 24, 2020
Building Systems with Feedback
Feedback Benefits
- Increases system stability
- Improves performance over time
- Provides a method of determining compensation
- Improves morale
The Feedback Triangle
Goals -> Execution -> Result Determination
When you are setting up processes you need to keep in mind this triangle. The more independence between goals, execution, and result determination the better. If the same person/entity is responsible for creating goals or executing or determining results you are likely to get bias in your feedback. Sometimes the bias is worth the lower overhead, but if you are designing larger systems care should be taken to break them up.
All Work Product Should be Reviewed
- Not just code, UI/UX, design, copy, even agile stories need to be reviewed
- Knowing all work product is reviewed is good for moral
Invest in Process
Your company isn’t just your products, it’s how you make your products. Make sure you work on the process as you work on the product. When VCs invest, they invest more in the people than they do the idea. That’s because the process is more important to success than the actual product.
Training Your Intuition
Decisions need to be made fast, analysis takes a lot of time. Most decisions have to be made with intuition, not data.
But there are different types of decisions. How long should you do analysis before you make a decision?
Some executives divide decisions into two types: reversible and non-reversible. Where reversible decisions should be made quickly. But this is a bit too simplistic to be useful. The reality is that the amount of time that should be spent on deciding a course of action should depend on a number of factors:
- The possible negative impact of getting it wrong
- The reduction in risk that analysis could solve
- The cost associated with doing nothing while analysis is done
Reversible decisions would normally have a very small #1. But a product manager should also pay attention to #2 and #3. Very often a decision is not reversible, but there is really no way of determining the correct course of action. If this is the case, then a decision should be made quickly because analysis won’t make it any better. Finally, many don’t factor in the cost of doing nothing. Sometimes waiting doesn’t hurt you too much, but very often not doing anything is the worst decision. It is often good to think of doing nothing as one of your options. This is the default, and it has its own costs and benefits associated with it.
When you are making a decision, you should also think about mitigating possible damage from it, even making it reversible. If you are able to do this, then that will often lead you down one possible path.
If you can, make sure you can unwind the changes.
Producing Statistically Significant Results
Periods and Averaging
Taking enough data is important, and selecting the right periods to compare can reduce the noise in your data. Generally, you want to find periodicity in your data and remove it. For most businesses, this means using weekly/quarterly/yearly periods. You really never should use monthly because that will just add noise. Then you need to remove trends. It’s possible to look at daily numbers if you understand how your numbers fluctuate throughout the week. You need to be particularly aware of seasonal changes and how they affect your business.
Control groups
Things change. Using a control group can make your data much more accurate. Here are some ways I have found effective for software:
- Using one section of your website against another
- Releasing to a subset of users (by state/country)
- Releasing to a random subset
External Baselines
Using a control group can be too difficult to do all the time. Here are a couple of ways that you can baseline your features without all the effort. The main way is to use your competitors as a baseline. The information may not be easily available. You may need to use a proxy. I have found that using internet traffic can be an excellent proxy for software and internet companies.
Scheduling for Independence
If you have two or more features that affect a result significant independence can be achieved by separating them in time. It is not always that we need to do this. Because this can affect the scheduling significantly we should be prudent in its application. Often companies will have hundreds of small changes happening in a sprint, and it is not possible to separate each of them. It’s also difficult to assess how much separation in time is necessary. We want to have features released in a sprint that affect different metrics if we can.
Checklists
Pre-Release:
- Do you have enough baseline analytical data (sometimes you need to put in more analytics before you release your feature)?
- Is this feature’s results correlated to another feature that is being released, was just released, or will soon be released?
- Do you have the ability to roll back the feature?
Postmortem
When a product or feature succeeds, there is typically significant thought put into why it succeeded, but if it fails most companies move on quickly without giving sufficient thought to why it failed or what they could have done differently. I recommend a postmortem checklist. Feedback is essential.
On Success:
- How were your estimates? Cost, schedule, return?
- What would you have done differently, if you had to do it again?
On Failure:
It’s important to understand why it failed. And to make sure you keep good records for the next guy.
Did you see the risk of it failing before starting the feature?
If so, could you take steps to mitigate that risk?
Was it a problem with the idea or a problem with execution?
If you aren’t certain, why aren’t you certain?
What would you have done differently, if you had to do it again?
Do you know how good your estimates were?
How were your estimates? Cost, schedule, return?
Have you made the same or a similar mistake before?
Was there a feature that you should have made instead?
When did you know it was going to fail?
Have you asked the users of the feature what they would change?
Could you have had better analytics in place before the release?
Was the coordination between the technical release of the product and sales and marketing what it should have been?
Written by Scott creator of Valtrace.
