Prerequisites to get to an experimentation culture — mindset (1/4)
When trying to scale up experimentation, companies often find that the company culture is the biggest obstacle. Shared behaviors, beliefs, and values make getting to an experimentation culture difficult or impossible.
Common challenges are related to the mindset in the organization, the organization’s structure, the leadership model, how decisions are made, what behavior gets rewarded, and how things get done.
In this article, and the following three, I try to summarize the prerequisites to get to an experimentation culture by zooming in on these topics using a wide variety of resources.
Let’s start with the mindsets within an organization.
Three mindsets to get to an experimentation culture
When it comes to the mindset within an organization, three mindsets are essential to get to an experimentation culture; an experimentation mindset, a learning mindset, and a growth mindset.
Let’s start with an experimentation mindset. There are many explanations of what an experimentation mindset is. For me, an experimentation mindset is thinking like a scientist. This means being curious, questioning assumptions, creating testable hypotheses, running experiments that produce evidence, and analyzing cause and effect. An experimentation mindset helps us learn about our customers, reducing risk and improving outcomes.
At the highest level of experimentation maturity, the whole organization has this mindset and approach.
However, a lot of companies are not there yet. One of the biggest problems is that only a small percentage of ideas turn out to be winning experiments. That means most ideas do not improve anything. As many companies are output-driven, experimentation might feel like a waste of time, slowing down the whole operation.
Let’s say the average win rate of the experiments is 25%. Now, imagine an organization solely focussing on output, and you want everyone to experiment so that 75% of all output is discarded. This is quite a mismatch. The company wants a high output, but 75% is discarded.
If a company aims at outcomes, meaning positive changes for the customers and business, implementing only winning or non-losing ideas is complete logic.
As many ideas are discarded, an experimentation mindset does require humility. It is hard to maintain a big ego when anyone at the company can look at the data and see all the big ideas that ultimately resulted in a losing experiment.
Therefore, another requirement for an experimentation culture is the acceptance of failure. In other words, everyone needs a learning mindset.
With a learning mindset, you do not view losing or inconclusive tests as failures. Instead, you can learn a lot from every experiment about what works and does not work for different user groups. Combining learnings from experiments helps obtain a complete picture of the types of experiences that do and do not resonate, leading to new hypotheses, new experiments, and ultimately winning tests.
Thus, turning every experiment into knowledge is essential to success. Inconclusive and losing tests can not be avoided, so think of it as a critical part of moving forward. Instead of dreading a losing experiment, negative experiment results should be celebrated as saving the customers and business from harmful releases. Many innovations have emerged from failures. Thus failing experiments are inevitable and valuable.
This is only possible if failures are accepted. If losses are punished in any way, like in personal assessments, it is better not to have proof of results. Thus, people do not run experiments.
Also, if managers overemphasize winning instead of improving decision-making and mitigating risk, it may encourage employees to focus on familiar solutions, short-term gains, or game outcomes. For example, I have seen an organization that wanted their employees to experiment but overemphasized winning. This resulted in employees determining the KPI of their experiments only after the test was done. When you pick the right KPI after your experiment, you will almost always have a winning variant.
Unfortunately, learning from failure can be challenging to manage. Failure can lead to embarrassment and expose essential gaps in knowledge. This makes it challenging to obtain this mindset.
(You could argue that a learning mindset is part of an experimentation mindset. But then again, you could also argue how much inconclusive and rejected hypotheses are valued in science.)
Finally, a growth mindset is essential to realize a change in the organization. You might already be familiar with Carol Dweck’s book Mindset, in which she distinguishes two types of mindsets; a fixed mindset and a growth mindset.
People with a fixed mindset believe their intelligence, talents, and personalities are fixed traits that cannot grow. They believe they are born with a certain level of ability or skill. When it comes to change, they feel things have to be a certain way and get frustrated with failure, as they see this as their shortcomings. Therefore, they do not like being challenged and do not like new things.
Someone with a growth mindset believes their basic abilities can be developed through dedication and hard work-brains and talent are just the starting point. They value change and see failures and challenges as a way to grow.
To get to an experimentation culture, change in the organization is necessary. Those with a fixed mindset will quickly struggle and resist the change. If an important stakeholder has a fixed mindset, try to understand them. Often people who appear to have a fixed mindset have reasons for their attitude. By understanding them, you could create a win-win situation.
For instance, someone with a fixed mindset is under a lot of time pressure. By making sure he does not get extra work (better yet, if the change helps him save time), he might be more open to the change. Or someone could be worried about achieving his goals. If you show experimentation helps him accomplish them, he will be more open to working together.
Change the mindset step by step
In short, an experimentation mindset is required to get to an experimentation culture. Everyone needs to think like a scientist. Furthermore, failed experiments will happen, so it is crucial to be resilient against it and see it as a learning opportunity, thus requiring a learning mindset. And finally, for change to succeed, people need to have a growth mindset.
You can get to the best situation step by step through change management. But changing the mindset of an entire organization does take time.
- Experimentation Works — Stefan Thomke
- Growing Happy Clients — Daphne Tideman
- Experimentation Program Management — Ben Labay (CXL course)
- Top Challenges from the first Practical Online Controlled Experiments Summit (paper)
- Mindset — Carol Dweck
- Adapt — Tim Harford
- Failing forward — John Maxwell
Photo credit: Dialysis Technician Salary