Dozens of elementary-aged kids impale gummy bears on toothpicks and twist their creations together with licorice while chatting away in Spanish with their parents and siblings and noshing on pizza. It’s not a craft – the kids are learning about polymer models at a night of “Cena y Ciencias,” or Supper and Science, a free monthly Spanish-language initiative in Urbana, Illinois, to introduce K-5 students to science and scientists.
Across the country, scores of similar initiatives aim to bring more people, particularly underrepresented populations, into science, technology, engineering and mathematics, aka STEM. Targeting groups from kindergartners through early-career faculty, these recruitment, retention, and advancement programs are more important than ever as our nation works to remain globally competitive and faces thorny issues at home related to infrastructure, inequity, health, and the environment.
“We need scientists and people who are versed in mathematics and engineering to solve some of these really important problems,” says Dr. Ayesha Boyce of the Department of Educational Research Methodology. “That means tapping the next generation of talent.”
But how do we know that these STEM initiatives, buoyed by millions of dollars of funding from the government, are working?
That’s where Boyce and her department colleague Dr. Aileen Reid come in: they direct the STEM Program Evaluation Laboratory, also known as SPEL.
“The thing that people don’t know is that when all these initiatives are funded, someone’s standing alongside all these scientists evaluating their work,” Reid says. “That’s what we do. You never see or hear about what we’re doing. But we are right on the frontlines and on the frontiers with them.”
In the three years since its launch, SPEL has worked with over a dozen STEM initiatives, representing approximately $90 million in funding, to evaluate their effectiveness. That includes the new $12.5 million NSF-funded parent institute of Cena y Ciencias.
Reid and Boyce hope SPEL will serve as a model at UNCG and across the nation. Their research and mentorship structure is unique in their field. So is the makeup of their team.
“Oftentimes, frankly, students don’t see Black women getting million-dollar grants, or leading these big research initiatives, particularly in our field,” Boyce says. “It’s important for them to see that representation.”
Making a name in the “Wild West”
Evaluation is a new discipline. The American Evaluation Association was only founded in 1985.
As such, researchers and students in the interdisciplinary field come from a variety of areas including sociology, communications, psychology, statistics, health, and education. Private firms and government entities including the Department of Education also carry out program evaluation.
“Over the last two decades, the National Science Foundation and the National Institutes of Health began to require evaluation for all funded projects, but there’s no accreditation process for evaluators,” says Reid.
“We’re really at a point where everything is kind of like the Wild West – everyone’s doing something different,” Boyce adds.
SPEL traces its roots to fall 2015, when Boyce and Reid joined the Department of Educational Research Methodology as an assistant professor and graduate student, respectively. Under the auspices of the Office of Assessment, Evaluation, and Research Services, which she co-directs, Boyce created the lab in 2018.
The following year, Reid graduated and joined SPEL. They are both co-directors, and within their field, the researchers say, having two faculty running a laboratory is rare. The lab now includes another faculty member, Dr. Tiffany Tovey, as well as twelve graduate students and six undergraduates. That’s not all that distinguishes SPEL.
“We’re one of the only places in the nation where you are learning how to do evaluation with a social justice orientation,” says Boyce.
That orientation leads lab members to focus on culturally responsive evaluation, keeping the cultures of project participants in mind as they examine project methods and measure project impacts.
For example, researchers might assess a project’s anti-deficit orientation – a way of thinking about underrepresented groups.
“Instead of saying, ‘Oh, we need to help these poor people, what’s wrong with them?’ we say, ‘What’s wrong with our own culture so that certain people can’t succeed?’” Reid explains. “And we ask what strengths those groups bring versus saying, ‘Here’s all the things wrong with them.’”
Pushing for growth
The researchers express a deep sense of ownership and pride in the projects their lab helps advance.
A recent example is the $10 million NSF-funded STEM PUSH network, which connects and advances pre-college programs that increase the preparation of ethnically and racially underrepresented students for college STEM programs. PUSH also seeks to expand the ways college admissions offices assess potential students, diversifying beyond standardized test scores and grades.
With the shifts in admissions requirements due to the pandemic, it’s a perfect time to push for a reassessment of traditional metrics, says Reid. “We’re in a kairos – a critical moment – for admissions.”
Other large clients include the Purdue University-based Network for Computational Nanotechnology, which has been in operation for 20 years, and the Midwest Big Data Innovation Hub connecting 12 states and six research universities.
“When we started off we had smallish projects that had between like $50,000 and $100,000 over the life of the projects,” Boyce says. “The really exciting aspect is that, as more time has gone on, we have more opportunities for large-budget projects.”
Closest to home is the prestigious $1 million NSF ADVANCE grant recently won by a team of physical and social science faculty at UNCG. The project, which includes Boyce as a co-principal investigator and Reid as senior personnel, seeks to promote gender equity at the University.
Their aim, the researchers say, is to change culture, climate, and policies at UNCG – and not just within STEM fields.
The Next Generation
This summer SPEL ran 13 active projects, each involving at least two graduate researchers and one undergraduate. Since the lab’s launch, 20 students have worked there, developing their research skills by conducting interviews, focus groups, document reviews, and surveys.
“Through SPEL, we are able to mentor the master’s and PhD students as faculty members,” Reid says. “And then they in turn get experience, not just in evaluation research and evaluation practice, but in mentoring undergraduate students.”
Those undergraduates are another feather in the SPEL cap.
Last spring, Reid won the University’s Thomas Undergraduate Research Mentor Award, a feat all the more remarkable because the Educational Research Methodology department has no undergraduates.
In fact, there are no undergraduate programs in the young field of evaluation. Hence Reid and Boyce focus on mentorship of undergraduates from other fields. Some will hopefully move on to graduate programs in evaluation.
For those who don’t, says Reid, it’s still “a big win, when they can begin to make those connections with the work that they’re doing and transfer it to their own field – it brings evaluative thinking into another area.”
The lab offers a high-impact experience. In addition to learning evaluation skills, students create data visualizations, generate reports, and present to funders and at conferences.
All lab members take part in weekly three-hour lab meetings, which include an hour of professional development as well as project check-ins. Reflecting the SPEL’s values of social justice, anti-racism, and inclusivity, the group discussed “How to Be an Antiracist,” by Ibram X. Kendi, for some of their professional development hours.
With their emphasis on diversity and inclusion both in their work and within their lab, Reid and Boyce embody the values they espouse.
“We have students from all walks of life,” Boyce says proudly. “I mean ethnicity and race, but I also mean in terms of age, background experience, having children and not having children – they are in all kinds of places in their lives.”
That makes a difference to the work, says Reid. “I think we are transforming the landscape of STEM program evaluation because of who we are and how we’re doing it.”