Kauffman Knowledge Challenge and Designing Applicable Entrepreneurship Experiments

For the 2018 Kauffman Knowledge Challenge, Kauffman is seeking to fund projects in four specific areas:

1. Technology and the new nature of entrepreneurship
   
2. Barriers to entrepreneurship
   
3. Culture and entrepreneurship
   
4. Causal research on entrepreneurship interventions at scale
   

While looking into Special Interest Areas in more details here, on the first item “Technology and the new nature of entrepreneurship” we found these two sample topics to be very relevant and aligned with our mission in scaling entrepreneurship and innovation:

“What does digitalization mean for entrepreneurs and for the organizations that support them? What can be done to achieve inclusion? How can entrepreneurship support organizations leverage digitization for their programs?”

Related to this, few weeks back at Global Entrepreneurship Congress, Istanbul, we did a speech about “Digital Side of Startup Ecosystem Development”. The presentation materials of this speech, include many valuable learnings and insights that can help to contribute to projects under this area.

And

“What kind of education and training programs fit a future where people will need entrepreneurial skills to work?”

In our recent blog post “Entrepreneurship Education: Educating to jobs vs. Educating job creators.” we wrote extensively about our views related to this topic in general, as well as concrete initiatives that we are currently working on.

Naturally the points covered in this post, would also be big positive contributors for Special Interest Area 2: Barriers to entrepreneurship as well.

In both of these topics, if the related solutions we have described would already be broadly available, those would bring enormous help also in enabling Causal research (specific area 4), especially to tackle the known challenges Causal Research Studies related to Randomized Controlled Trials (RCTs). As well, as contributing to all sample topics regards to questions about “how do we measure...?”.

The fourth area itself, causal research, is naturally connected to all three other areas in many other ways as well.

Causal Research Studies
Below, we also want to contribute to the creation of causal research studies in the entrepreneurship space in general as well.

Kauffman “is interested in multi-site causal research the tests specific interventions designed to overcome barriers to entrepreneurship.” This research should be focused on the United States and utilize randomized controlled trials (RCTs) across multiple sites.

Having identified that Randomized Control Trials (RCTs) can have limited generalizability, which limits their usefulness in informing program and/or policy design. As such, the Foundation is interested in projects that exploit rigorous causal research across multiple ecosystems which, by nature of being systems, comprise many moving parts.

We will use this post to explore general RCT design, so that you can more effectively propose experiments to Kauffman.

About Randomized Controlled Trials
Aiming to reduce bias, randomized controlled trials randomly place study participants in a treatment or control group. Using a case example, the treatment group would receive an intervention designed to overcome barriers to entrepreneurship. The control group would receive a placebo intervention, meaning that they would receive a false intervention that would be designed to not accomplish anything. Randomization should be done after participants have been screened and selected for the experiment, but before the intervention is administered.

Based on research by Anthony J. Viera, MD, MPH and Shrikant I. Bangdiwala, PhD, RCTs are most effective when conducted in a single-blind or double-blind manner, also known as masking. In a single-blind experiment, the research participants do not know whether or not they are being given the treatment. This means, neither the research participants nor the experimenters know who is receiving the treatment in a double-blind experiment. Both of these methods can be used to eliminate bias.

Challenges with RCTs in Entrepreneurship Policy Context
Related to Kauffman Knowledge Challenge, as an example model for conducting randomized controlled trials across multiple sites is described as using entrepreneurs in one region as the control group and entrepreneurs in another region as the treatment group. But as already identified, this may give limited results. Since there will always be many differentiating factors between any two regions (i.e. different cultures, demographics, financial opportunities, industries, etc.), confounding, a situation where an experimenter cannot reasonably eliminate alternative explanations for an observation, can easily occur.

We agree that better results could be achieved where treatment and control group are located in the same region. And in addition, it make sense to replicate the study across multiple regions.

Another issue to take note of is the Hawthorne Effect, also known as the observer effect. When entrepreneurs know that they are being observed, they are likely to behave differently than they otherwise would. This limits the applicability of RCTs. Two ways to mitigate the Hawthorne Effect are; a) performing discrete observations and b) making the period of study as long as possible.

Additionally, Jeffrey Hammer identified three problems with RCTs in development economics for the Brookings Institution:

1. “There is a systematic bias toward analysis of private goods as opposed to public goods.”
   
2. “The objectives ascribed to peoples’ actions are defined very narrowly and their specific constraints often ignored.”
   
3. “The jump from positive analysis to normative policy is stunning.”
   

It it good to be aware of RCTs known limitations in and development of economics and entrepreneurship policy. If they are not conducted properly and applied too readily, they can potentially have even adverse effects on policy design.

Enhancing Your Experimental Design for RCTs
By being aware of these challenges of RCTs, we hope that you can be better equipped to design proposals for the Kauffman Knowledge Challenge as well.

If you would like to leverage any of our materials in your white paper, additional ideas on experimental design or utilize or expertise, tools, resources etc. in your actual project if materialize, please reach out to Conor Flynn, our Operations Manager who is based in the United States. He is happy to help you to mitigate bias and confounding so that you can provide stronger data for informing future policy decisions and entrepreneurship interventions.