A fundamental shift in the way businesses make decisions is taking place. Traditionally, running experiments and abiding by the scientific method has been the domain of scientists, but managers across industries have been adopting cultures of experimentation to inform their decisions at an unprecedented scale.
At Microsoft’s Bing, for example, approximately 80 percent of product changes are initially tested as controlled experiments. eBay now saves many millions of dollars per year in advertising thanks to a simple in-house study, which revealed that buying paid search ads wasn’t lucrative for them. Google conducts experiments to see which interview questions are most likely to correspond with on-the-job performance, and then it heavily weights candidates’ answers to those questions.
A combination of the availability of massive amounts of consumer data and improved computing power, plus the fact that it’s easier and cheaper to randomise participants than ever before, has created the perfect conditions for the rise of what’s being called the “experimental revolution” in business. As businesses increasingly demand actionable data to inform their decisions by leveraging experiments, MBA programs need to do more to equip future leaders with the necessary skills.
The case for experimentation
The value of experiments in non-scientific organisations is quite high. Instead of calling in managers to solve every puzzle or dispute large and small (Should we make the background yellow or blue? Should we improve basic functionality or add new features? Are staff properly supported and incentivised to provide rapid responses?), teams can run experiments and measure outcomes of interest and, armed with new data, decide for themselves, or at least put forward a proposal grounded in relevant information. The data also provide tangible deliverables to show to stakeholders to demonstrate progress and accountability.
Experiments spur innovation. They can provide proof of concept and a degree of confidence in new ideas before taking bigger risks and scaling up. When done well, with data collected and interpreted objectively, experiments can provide a corrective for faulty intuition, inaccurate assumptions, or overconfidence. The scientific method (which powers experiments) is the gold standard of tools to combat bias and answer questions objectively.
But as more and more companies are embracing a culture of experimentation, they face a major challenge: talent. Experiments are difficult to do well. Some challenges include special statistical knowledge, clear problem definition, and interpretation of the results. And it’s not enough to have the skillset. Experiments should ideally be done iteratively, building on prior knowledge and working toward deeper understanding of the question at hand. There are also the issues of managers’ preparedness to override their intuition when data disagree with it, and their ability to navigate hierarchy and bureaucracy to implement changes based on the experiments’ outcomes.
Some companies seem to be hiring small armies of PhDs to meet these competency challenges. (Amazon, for example, employs more than 100 PhD economists.) This isn’t surprising, given that PhDs receive years of training — and that the shrinking tenure-track market in academia has created a glut of PhDs. Other companies are developing employees in-house, training them in narrow, industry-specific methodologies.
For example, General Mills recently hired for their innovator incubator group, called g-works, advertising for employees who are “using entrepreneurial skills and an experimental mindset” in what they called a “test and learn environment, with rapid experimentation to validate or invalidate assumptions.” Other companies — including Fidelity, LinkedIn, and Aetna — have hired consultants to conduct experiments, among them Irrational Labs, co-founded by Duke University’s Dan Ariely and the behavioral economist Kristen Berman.
The bottom line is that companies today are looking for the person who can do it all — design and run experiments, be numerically literate to interpret the results, utilise their interpersonal savvy to implement data-driven change, and be inspiring and visionary enough to lead a culture of experimentation. These skills, which are at the core of what business students should know when they graduate, demand training in experimentation and the scientific method. But business schools aren’t putting that training at the core of their curricula. What can they do differently?
Implementing Experimentation in Business Schools
Some business schools do teach experimentation to a limited extent. Professors who teach marketing courses, for example, often cover basic forms of A/B testing (e.g., analysing whether an advertisement versus a public service announcement leads to more product purchases and whether the ad campaign was a good investment). Some professors also teach “Booking.com,” a case by Thomke and Beyersdorfer about a travel-accommodations company that has an exemplary culture of experimentation, empowering employees to run thousands of experiments per day. In addition to culture, the case teaches some nuts and bolts of experiments, including terminology (e.g., p-value, null hypothesis, Type 1 Error). But this is just one assignment and one case — and the teaching notes to “Booking.com” even warn instructors that most business-school students “will have little or no experience with online experimentation.”
One of us (Elizabeth) devotes time to experimentation in a class called Managing and Leading in Organizations in the PMBA program at the David Eccles School of Business, at the University of Utah: Each year in the class, she asks students to break into groups and each design an experiment, which they then discuss as a class. Students taking part in this exercise learn valuable lessons about experimentation, but the class itself isn’t devoted to the use of experiments in managerial decisions. There is one notable course that does this, taught at Harvard Business School by Michael Luca, but it isn’t offered every year. Teaching true experimentation is neither typical nor widespread in business schools.
Doing more wouldn’t be hard. Through their university libraries, MBA students already have access to a full published literature that they could be using to construct and test ideas. They typically have Qualtrics accounts or similar, which allow them to create surveys, and they can access statistical packages to analyse data. If they were taught the basics in a business research methods class, they could partner with companies while still in school to get experience with experiments in the field while also providing a benefit to the company.
An MBA-level course in experimentation should include foundational aspects of the scientific method, which can later be adapted for the specifics of a given firm or industry. The basics include:
l How to form testable, falsifiable research questions and build on what has been done already
l How to operationalise constructs and define tasks
l How to think about random assignment to conditions and threats to the validity of the experiment
l Why a transparent workflow can allow healthy scrutiny and help protect the experiment’s integrity
How to do inferential statistics
Overall, we believe there are three main actions that business schools should take in order to best position students to realise their full potential as leaders of the experimental revolution. First, educate students on how to mine academic literature efficiently and build on insights from prior experimental work, then solidify this as one of many resources they turn to when generating value. Second, teach the scientific method and how to conduct effective experiments in the business world using up-to-date best practices. Finally, give students an opportunity to practice these methods while they’re in school, so they’ll be more likely to apply them once they leave the classroom and enter the workplace.
But some questions remain: How integrated into the MBA curriculum can or should experimental research methods be? Is a single class enough? Should a research apprenticeship at a company or with an academic researcher be required? These are best answered by the faculty within each school, keeping in mind the school’s mission, existing curriculum, student body, and industry/employer partners.
For centuries, in all sorts of professional arenas, we’ve relied on experimentation and the scientific method to generate reliable and insightful outcomes. But all too often in business today, as we try to cope with the onslaught of change, we rely on costly and wild intuition-based decision making, and it’s costing us. It’s time for us — with the help of MBA programmes — to get back to the basics. — Harvard Business Review.
