Stella® Architect Powers Management Experimentation at MIT
Hazhir Rahmandad
“In the last decade, online behavioral experiments have become more common,” says
Hazhir Rahmandad, Associate Professor of System Dynamics, Sloan School of Management,
Massachusetts Institute of Technology. “We used to have to recruit people and bring
them into a lab but, using services like Amazon Mechanical Turk and Stella, we can
quickly find subjects who participate in the experiment at home. It’s less costly,
more efficient and I’ll be leveraging the technology more and more.”
Rahmandad found his way to Stella three years ago after trying to create an online
experiment using another platform. “My graduate student worked on the project for
six months but it still wasn’t finished when he graduated,” says Rahmandad. “I came
across Stella Architect and finished the project in two days.” Now, Rahmandad and
his colleagues have completed two experiments that study managerial decision-making
and created a model that describes COVID-19 pandemic response across 92 countries.
Working with Michael Shayne Gary, PhD, Associate Professor, Australian School of
Business, University of New South Wales, Rahmandad used Stella Architect to study
how managers apply, or don’t apply, learning to strategic decision-making. “We
created a model of a service operation and presented participants with two choices
to manage this simulated operation: how much to pay their employees and how to
structure the work,” says Rahmandad. “The system includes two promising, but
incompatible, strategies.”
The first strategy calls for hiring people into low paying, narrowly defined positions.
Training takes less time, there is little room for confusion or mistakes, but employee
dissatisfaction and turnover is high. The second strategy hires high quality employees
into higher paid positions that engage them in more complex tasks. Training takes longer
but employee satisfaction is high and turnover is relatively low. These mimic low-roads
and high-roads strategies in many service operations.
“We wondered why we don’t see more instances of businesses using the high-roads strategy,”
says Rahmandad. “Our hypothesis was that delays between adopting a strategy and realizing
the full benefit or downside of that action influenced what gets adopted.”
Amazon Mechanical Turk, an online outsourcing marketplace that connects employers to individuals
who can complete repetitive tasks or, in this case, subjects to experimenters, was used to
recruit non-expert study participants. Subjects with managerial experience were recruited
through Qualtrics, an online experience management platform.
Using a Stella Architect simulation of the business, published to the
isee Exchange™, both groups of subjects attempted to find the combination
of work structure and compensation that maximized profits through four training
rounds of 24 quarters each. They received feedback through a simulator
dashboard that tracks quarterly metrics for key performance indicators.
Subjects were able to track the performance of their strategy decisions.
Published in Organization Science in December 2020 (“Delays Impair Learning and Can
Drive Convergence to Inefficient Strategies”), the study showed that delays and uncertainty
influence strategic decision-making for both non-experts and experienced managers. “We noted
that the longer the delay in seeing benefit from the chosen strategy, the more likely
subjects were to move towards a different strategy with faster payback,” says Rahmandad.
“In this study, we used the simulation environment to look at managerial learning in the
presence of feedback delays. Business simulations could also compress the time needed to
learn about the impact of alternative strategies, which otherwise may take years to unfold,
helping leaders identify and internalize strategies that pay off over the longer term.”
The second experiment, conducted with colleagues Jerker Denrell, PhD, University of
Warwick and Drazen Prelec, Phd, Massachusetts Institute of Technology, also leveraged
a Stella model on the isee Exchange and Amazon Mechanical Turk, this time to investigate
learning in simpler dynamic tasks, including in settings with multi-player markets.
Similar to the first study, Rahmandad and his colleagues sought to determine why most
decision makers have trouble making near-term decisions that benefit long-term performance.
“We were able to build a simple model and simulation based on a task called Harvard Game,
recruit subjects, and start collecting data in days,” says Rahmandad. Using the online,
multiplayer simulation, subjects repeatedly chose between two options (buttons). A payback,
or profit, was determined by the combination of choices they made and the choices that other
players were making.
“We know it’s harder to learn to make best choices when the task is dynamic,” says Rahmandad.
“We found that the challenge persisted even when everything about the task structure was
fully known and participants were given plenty of hints. In fact, many continued to learn
the exact wrong lesson.”
This interface provided subjects with feedback measuring the profitability of their prior 10 choices.
By varying the type of information provided to subjects, the study showed that
unprofitable strategies could not be attributed to game complexity, ambiguities,
or learning difficulty. Rather, Rahmandad and his colleagues found that the biggest
stumbling block was the ability to think systematically and understand how trade-offs
influence outcomes over time. Their experiment and findings were published in
Strategic Management, October 2020
(“What Makes Dynamic Strategic Problems Difficult? Evidence from an experimental study”).
“I’ll continue to leverage the work we’ve done with Stella Architect and online subject
pools to understand dynamic learning and decision-making,” says Rahmandad. “Through the
experiments we’ve conducted so far, we have strived to build one-click solutions that
move us from recruiting subjects to running the experiment to processing data in hours.
That provides huge advantages in trying alternative ideas and coming up with better
research designs in a matter of days.”
The ability to quickly generate and run experiments and simulations enables Rahmandad
to investigate and disseminate current issues quickly. In July 2020, he and MIT
colleagues TY Lim, a PhD student and John Sterman, PhD, Forrester Professor of
Management Science Director, System Dynamics Group Chair, published “Behavioral
Dynamics of COVID-19: Estimating Under-Reporting, Multiple Waves, and Adherence
Fatigue Across 92 Nations” on the Social Science Research Network.
“We calibrated our model using VenSim and then moved it to Stella Architect
with a user interface,” says Rahmandad. “We wanted to share broadly how
changes in behavior in response to risk of COVID-19 and different vaccination
rollouts create the disease trajectories in the coming months.”
Overview of COVID-19 behavior model
The model provides a global view of a global crisis, including essential biological and
behavioral mechanisms and feedback loops. It explains the significant differences in
per-capita cases and death rates across countries and offers projections for both measures.
Not surprisingly, collected data indicates that diminishing adherence to COVID-19 precautions
added significantly to cumulative cases and that improvements in adherence would reduce
both infection and death rates with limited additional economic costs.
The simulator is posted on the isee Exchange at:
https://exchange.iseesystems.com/public/mitsdl/covidglobal/.
“isee systems’ support has been really helpful through all these experiments and
simulations, even adding requested features in a matter of weeks,” says Rahmandad.
“Whenever I need to run an online experiment or simulation, I expect to start with
Stella Architect.”