Committing to a randomized evaluation: The scariest and smartest thing we ever did
My name is Shane DeRolf.
I am the founder and CEO of Big Word Club, a digital learning company committed to improving the vocabularies of every kid in America.
It all started about 25 years ago when I quit my job to pursue a dream.
That dream was to create great media that helped kids think, learn and grow.
Here’s what those 25 years felt like…
Along the way, I’ve learned a lot.
About myself.
About life.
About business.
And though the dream has not changed, its focus has evolved.
It’s evolved from creating great media for kids to creating great media for kids that addresses and solves a large, unmet problem.
Closing the Word Gap
According to a landmark study by Hart and Risley in 1995, children’s vocabulary skills are linked to their economic backgrounds. According to Hart and Risley, children from lower socioeconomic homes may hear as many as thirty million fewer words than children from higher socioeconomic homes by the time they are three years old. This is commonly called the “word gap.” Because kids learn by hearing, they don’t learn what they don’t hear. This translates into a gap of 400-700 words in the vocabularies of low-income kids and their more privileged classmates.
Because kids who start behind tend to stay behind, we decided to try and create a vocabulary intervention that…
• Kids loved.
• Was fast, fun, and easy to implement by both teacher and parent.
• Was easily scalable and inexpensive.
• Improved the vocabularies of preschool and kindergarten students.
And that’s exactly what we did with Big Word Club. (Watch a short video.)
Problem was, nobody believed us.
Turns out anecdotal evidence doesn’t carry much water in the education market. Despite the fact that we successfully launched a pilot in 1,000+ preschool and kindergarten classrooms to rave reviews and converted 20 percent of our pilot teachers into paid subscribers, the truth was we lacked the scientific evidence we needed to provide school districts with the confidence to buy Big Word Club for their students.
The Moment of Truth
The minute we realized that if Big Word Club was going to be successful and help the millions of kids we envision helping, we knew we needed to get the evidence that proved it works.
As luck, life, or serendipity would have it, it was about this time that I was introduced to J-PAL and made my pitch to Vincent Quan, a Senior Policy Manager at J-PAL North America. It went something like this:
“Every year, millions of kids from lower socioeconomic homes are entering preschool and kindergarten knowing hundreds fewer words than their more privileged classmates. Kids who start behind tend to stay behind and these kids never reach their potential. We believe Big Word Club can help and we’d like your help to prove it.”
Fortunately, J-PAL agreed that the promise of Big Word Club was worth the cost of funding a randomized evaluation and assembled a world-class team from the University of Chicago and University of Toronto to implement the evaluation.
The Scary Part
Though we could not have been more excited than to be part of a randomized evaluation funded by a leading research center at MIT and implemented by rock stars of academia, we also knew that if the results were not good, we would close the company and many years of hard work and millions of investment dollars would be lost.
So it was a scary time.
It was also the smartest thing we ever did as a company.
Over the last eighteen months, the Big Word Club program has been the focus of a randomized evaluation to measure its impact on preschool and kindergarten students' vocabularies. The evaluation was implemented by Professors Ariel Kalil and Susan E. Mayer, along with the invaluable assistance of project manager Michelle Michelini of BIP Lab at the University of Chicago and Professor Phil Oreopoulos of the University of Toronto. The research team also helped us design a customized vocabulary assessment based on the Peabody Picture Vocabulary Test-IV.
As the founder and CEO of a company undergoing a randomized controlled trial, those eighteen months were some of the most difficult of my career—not because of what we had to do but because of what we could not do.
As a company, we felt that because we were committed to a randomized evaluation, we needed to wait until the results came in before we could, in good conscience, aggressively sell Big Word Club in the market or fundraise.
Fortunately, when the results came in, it was good news and full of exciting observations. Below are some of the highlights:
• In percentile terms, results show that a child at the 50th percentile would experience up to a 13 percent gain on average, jumping to the 63rd percentile after having access to Big Word Club after 17 weeks.
• A child at the 80th percentile would experience up to a 7 percent gain on average, jumping to the 87th percentile after having access to Big Word Club after 17 weeks.
• Unlike other vocabulary interventions that are expensive and require significant teacher training, Big Word Club is low-cost and requires no teacher training.
What We Learned
• We learned what we can do to improve the effectiveness of our program when developing future content (this was invaluable and unexpected).
• We learned how to work better with incredibly smart people that think differently than we do.
• We learned how to be more patient.
What’s Next for Big Word Club
Now that we are truly an evidence-based intervention, we can market Big Word Club with confidence to school districts, elementary schools, daycare centers, Head Start programs, teachers and parents. And that’s exactly what we plan to do!
We are also exploring strategic partnerships and investment opportunities with individuals, companies, and organizations that can help Big Word Club do more good faster.
By improving a child’s vocabulary, we improve his or her chances for a successful and happy life and that, to us, is a job worth doing.
