Conflating outcomes and impact – a case study from my experience

The importance and difficulty of measuring impact  was a common thread throughout this week’s reading. As Ebrahim and Rangan point out, many organizations confuse measuring outputs and outcomes with measuring their impact (Ebrahim and Rangan 2014). My experience with a socially driven business prior to starting my graduate studies highlights this key tension and illustrates just how difficult impact can be to nail down.

At 2U, the large education technology company at which I worked prior to graduate school, “outcomes matter.” This mantra, written on walls across headquarters and ubiquitous in speeches by the CEO, permeates everything the company does. 2U partners with world-leading nonprofit colleges and universities to build high quality online graduate degree programs. As an online program manager (OPM), 2U handles everything a university needs to bring a given graduate degree program online. This includes marketing the program, recruiting students, hosting the technology, managing operations, providing post-enrollment support, and securing local internships/clinical placements. This allows individuals to earn a high-quality, well-known, accredited graduate degree without having to move or quit their job.

Each year, 2U releases an annual Impact Report to showcase and quantify its impact. However, the content of this report highlights the central tension of conflating outcomes with impact. The “logic model” referenced by Ebrahim and Rangan demonstrates the key differentiators between an organization’s outputs, outcomes, and impacts. Outputs are immediate results. For 2U outputs would include number of students graduated or number of graduates passing licensing exams. Outcomes are medium-to-long-term results. For 2U, outcomes would include increased income or improved job satisfaction. Impact, or effects on root causes and sustained significant change, for 2U would include sustained decreases in the real cost of attending graduate school or improvements to efficiency in job markets for which graduate degrees are required.

The 2U Impact Report contains a multitude of data on outputs and outcomes, but is illustrative of the challenge to define and measure impact. For instance, he report states that 2U has enrolled over 21,000 students in its programs as of 2016. In terms of outcomes, students in the online MBA program at UNC Chapel Hill report an average salary increase of $33,000 from the time of enrollment to graduation and 97% of graduates from Georgetown University’s Family Nurse Practitioner program passed their board examinations. Evidence of outcomes, on the other hand, is less convincing as the report simply pairs outcomes data with economic trends in an attempt to highlight impact.

Despite these truly impressive outputs and outcomes, the title of the report is essentially a misnomer. However, as more cohorts progress through 2U’s programs and the model further proves out 2U can begin to shift its focus to impact as defined here. This will likely require engaging with third parties, such as universities or social impact organizations, to clearly define impact and quantify 2U’s contribution.