Why Do We Need More Experimental Design in the Classroom?
Experimentation in business refers to the systematic approach of testing new ideas, strategies, or products by implementing controlled trials and analyzing the outcomes to make data-driven decisions (Davenport, 2009). It offers several advantages for managers working in multinational corporations (MNCs). First, it helps them understand causal relationships (Zellmer-Bruhn, Caligiuri, & Thomas, 2016), where, by isolating and testing specific factors, managers can gain clearer insights into the impacts of various strategies and actions. Therefore, many companies like Booking.com and Amazon use monthly thousands of experiments to measure the effects of their (potential) decisions, from small changes on their websites to larger strategic changes (Thomke, 2020). As Jeff Bezos now famously said, Amazon’s success “is a function of how many experiments we do per year, per month, per week, per day” (Thomke, 2020: 51). Furthermore, experiments help to mitigate biases by random assignment, ensuring that any differences in outcomes are due to the manipulation of the independent variable rather than external factors (Reeb, Sakakibara, & Mahmood, 2012). Last but not least, they are a useful method to quickly test and iterate on innovative ideas, products, or services, before scaling them up. This approach reduces the risk of large-scale failures and ensures that strategies are viable before full implementation, while at the same time allowing the company to stay ahead of competitors by rapidly refining and optimizing their offerings.
The realization of the power of experimentation in MNCs has been embraced by a growing cohort of managers, leading to a notable increase in the frequency of experimentation within diverse companies and industries (Thomke, 2020). Simultaneously, this surge has emphasized a growing demand for junior and middle managers equipped with a data-driven mindset and the expertise to design and execute business experiments effectively.
However, regardless of the advantages of experimental design, its increasing adoption in companies, and the growing employment demand for managers with such skills, a significant gap remains in business school curricula (Tenney, Costa, & Watson, 2021). There are two main reasons for this. One is that academics in the international business (IB) field rarely use experimental design in their research. As a matter of fact, less than 1% of all empirical articles published in the IB most prominent journal, the Journal of International Business Studies, used experiments as a research method (Zellmer-Bruhn et al., 2016). This clearly shows that in the IB domain, there are not many experts on experimental methods who could teach students how to implement them in companies. Second, the recent trend in business schools to offer fewer methods seminars and limited opportunities for academic (re)training due to financial constraints (Aguinis, Cummings, Ramani, & Cummings, 2020) does not promise a bright future in this regard.
As a consequence, there is a mismatch between the growing companies’ need for employees with experimental skills and the lack of supply by business schools. To bridge this gap, the innovative master’s-level course, “The Power of Experimentation in Multinationals,” equips students with a crucial research method that enhances their future management careers. This course guides students on how to conduct experiments within companies and identify where the greatest potential of experimental design lies, ultimately providing students and their future employers with strategies to leverage systematic experimentation for addressing complex challenges.
How to Integrate Experimental Design in the Classroom?
The course takes a comprehensive learning approach by incorporating various teaching methods, such as lectures, in-class and homework case analyses, an experimental lab visit, participation in experiments, and a team research experimental project. This diverse set of methods (presented in detail in Table 1) aims to achieve a holistic learning experience for students through several phases with different aims.
The course begins with a compelling introduction to the importance of experiments in business, designed to capture students’ attention and emphasize the significance of understanding experimental methods in corporate settings. It features vivid examples of critical decision errors by experienced managers, such as Steve Ballmer’s infamous dismissal of the iPhone’s potential during his tenure as CEO of Microsoft (Satell, 2016) and Google CEO Sundar Pichai’s unsuccessful attempt to launch the censored search engine Dragonfly in China (Keshav Pailwar, 2019). A business professional guest then wraps up the session, sharing a personal journey from skepticism to the daily use of experiments in strategic development.
Following this introduction, the course delves into the principles of designing effective experiments. We explore what makes a good experiment, how to run experimental tests in online and offline business models, and how to experiment with new digital tools. After the lecture on different types of experiments, their advantages and drawbacks, we analyze two experiments conducted in a truck company (Khamitov et al., 2015), both of which are imperfect in different ways, and finish with students presenting an experimental design of already published IB or strategy papers (which they further elaborate on for homework).
Students then learn how some of the most successful MNCs, such as Booking.com, employ experimental practices and learn about high-impact experiments done in this company (Thomke & Beyersdorfer, 2018). Even though such companies run thousands of controlled experiments annually, engaging millions of users, we also discuss how similar approaches can be applied in small companies, startups, and everyday life, making the concepts more relatable to students.
A unique feature of the course is the participation in an experiment specially designed for each semester. These experiments vary to maintain an element of surprise and prevent information from previous semesters from leaking. Past topics include cultural impacts on AI reliance, investment choices, and career decisions, among others. Through group discussions, students attempt to identify the hypotheses tested, analyze methodological choices, and discuss their strengths and weaknesses. This diverse exchange of perspectives enriches their understanding and prepares them for their final research project.
This prompts a group discussion where students brainstorm various topics they would like to explore through experimental design and engage in a premortem exercise (Blokdyk, 2021), imagining their experiment’s failure and identifying potential causes. Each topic receives personalized feedback, with a strong emphasis on peer feedback and collaborative discussion to tackle challenges. Once final topics are chosen, students reconsider the ethical implications of their experiments, sparking broader discussion on evolving ethical standards over time, illustrated by an example of Facebook’s ethically questionable experiment on nearly 700,000 users (Albergotti, 2014). The conversation then covers the necessary steps for registering an experiment for ethical approval.
Moreover, the course includes a visit to four experimental labs at the university, offering students hands-on experience with experimental equipment and procedures, and broadening their understanding of what type of experiments can be done in which type of labs. This practical engagement deepens students’ grasp of theoretical concepts and demonstrates their real-world applications.
Finally, after gaining both theoretical and practical knowledge, students design their own experiments through Design Sprints (Kowitz, Knapp, & Zeratsky, 2016), where 4-5 students with various backgrounds come together to brainstorm, prototype, and test a new idea and experimental design. Collaborative problem-solving is fostered through peer interaction, with groups presenting their projects and gaining detailed feedback from classmates and the instructor. Additional feedback is also available for students who choose to further develop and implement their experiments, whether as a master’s thesis or in a professional setting such as an internship.
The chosen teaching methods foster active engagement, practical application of theoretical knowledge, and development of critical thinking -, decision-making-, and collaborative skills. These are essential for designing business experiments, which often require diverse perspectives and expertise.
Conclusion and Implications for Educators, (Future) Managers, and Informed Research
This course prepares students for management careers where experimental approaches to problem-solving and innovation are essential. By the course end, students will have acquired the necessary skills to independently conduct business experiments, identify business segments where experimentation is valuable, and foster experimental culture within organizations. Equipped to tackle complex challenges, they will use data-driven insights to refine strategies and drive continuous improvement in their companies and industries.
At the same time, this proactive educational approach aligns closely with real-world industry needs, ensuring that graduates are not only theoretically proficient but also practically adept. Consequently, managers benefit from a streamlined recruitment process, as they can recruit individuals who possess the specific skills and hands-on experience required to meet organizational demands when it comes to identifying causal inferences and analyzing complex data patterns. This ensures that new hires can immediately contribute to projects involving experimental design, and the interpretation of empirical results, thereby enhancing the organization’s capacity for rigorous and innovative problem-solving.
Furthermore, for educators, this course shows how to avoid monotony and student fears when dealing with a research methods topic, as it unveils an engaging strategy for learning research methods with confidence. Using methods listed in Table 1, the course sets an example of how to turn complex concepts and research designs into an enjoyable learning experience. This course’s approaches are also adaptable for teaching other research methods topics, encompassing both quantitative and qualitative research designs.
Finally, classroom-based experimentation not only familiarizes students with experimental methods but also builds a strong foundation for causal research skills, empowering them to tackle complex business phenomena with rigorous, evidence-based approaches. By embedding hands-on experimentation in the curriculum, some of these students who later pursue a PhD in IB can bring these essential skills into the field, ultimately helping to close the gap of the lack of experiments in IB.
In conclusion, the impact of this teaching innovation extends far beyond the classroom. By preparing students for careers in consulting and multinational corporations, the course promotes the adoption of causal thinking and advanced problem-solving approaches, laying the foundation for a culture of continuous improvement across companies and industries. Additionally, it serves as a model for educators, illustrating how a methods course can be delivered in a dynamic and innovative way, ultimately enriching informed research in IB.
About the Author
Jelena Cerar is an Assistant Professor at WU Vienna. She gained her PhD from WU Vienna and was a visiting scholar at Copenhagen Business School and the Institute for AI Research and Development of Serbia. Her specializations are strategic decision-making in MNEs (focusing on AI and cognition) and quantitative research methods in international business. Her research has been published in highly ranked journals, such as the Journal of International Business Studies, Journal of Management Studies, and Journal of International Management.