Edition 10: ‘How do we know it works?’

🔥 The New Stuff: We’re back!🧠 The Deep Stuff: One big dollop of Impact measuring puzzlement

Two weeks ago we disappeared. We promised you an article on Impact Measurement, and did we deliver? Nope. We went on holiday. That’s why the silence.

So here we are to make amends. Back with a deep deep deep dive into why figuring out what works (in the impact space) has been so f*cking hard for the Aid industry. In a stunt that’s enraged our marketing team, threatening millions of dollars of advertising revenue we don’t have, we’re going to link you to the whole thing off site at mirror.xyz.

The full length article is here, but for those to lazy to read the whole thing here’s a synopsis, damn you.

This is a change in tone for us. It’s a bit more scholarly (PUKE!), but stick with us. We’ll be back next week:

Did ‘the good’ we tried to do, do any good? On the heels of much disparagement, this question drove not-for-profits, governments and institutions for much of the last half-century. Now, enter an explosion of DAO activity to the idea of funding public goods. So, how can Web3 learn from past Impact mistakes, and measure effectively? Let’s find out.

Over the last 40 years Impact changed dramatically. From the Bretton Woods System, to massive international charities, the Millennium Development Goals, CSR, SDGs, and now ESG. Money has shifted from cause to campaign, often aided by political trends: saviourism (inter-war years to 1950s), reconstruction and anti-communism (1945 to 1980s), famine in Africa (1980s), HIV/AIDS (1990s to early 2000s), to climate change prevention and adaptation (now).

Blockchain technology arrives as an obvious candidate to democratise global Impact. Powered by potential for hyper-local, transparent and frictionless governance, smart contracts offer the possibility of programmable funding for projects. Also, blockchains create means by which we can fairly price undervalued public goods, meaning communities can be supported to do more good things for the planet, and companies can be incentivised to stop doing the opposite.

Yet any excitement around improvements to philanthropy’s substrate should not obscure Impact’s complexity. Improved tooling won’t guarantee improved outcomes, and some Web3 projects know this, tapping into older expertise and seemingly unfashionable industry knowledge.

When we think about Web3 and impact, we think about three tendencies that development seems to be leaning towards:

1. Better Coordination: Impact DAOs

Most people will know Web3 as crypto, NFTs, and DeFi. But Web3’s first wave of democratisation towards governance came in the form of DAOs. A DAO is an organisation without central authority, governed from the bottom-up by a decentralised body of users who make decisions on proposals. Ideally on-chain but as a minimum public, the number of DAOs is exploding, a phenomenon we explored earlier this year.

DAOs that focus on activities which were once the domain of Aid and philanthropy have been called ‘Impact DAOs’; the accepted definition being ‘a DAO that creates net positive externalities to the ecosystem around it’ As we’ve said before, Gitcoin has published a book on Impact DAOs; a great resource for further reading).

2. Better Funding: Quadratic Funding

Web3 tech provides an improved substrate for decision mechanics. Quadratic Funding, based upon quadratic voting, is a method of matching-fund allocation that when properly discounted, rewards projects with the highest number of contributors, rather than those with the most contributed. The effect of this is greater plurality, and wider distribution of power, away from those with the deepest pockets.

3. Better Proof: Impact Certificates

‘Doing good’ confers social status; it’s a major driver behind philanthropy, and sees its latest iteration in the now widely used concept of ‘purpose’. Yet, an absentee has been proof that you have contributed to ‘doing good’. Web3 could allow for collections of ‘Impact Certificates’. These certificates would be owned by you, (some) non-transferrable, and displayed in wallets to denote history, or right to participate in other Impact activities. This creates the possibility of certificates being traded, leading to Impact Markets. A thriving marketplace for impact certificates (provable impact) would provide an improved incentive mechanism to fund more people to ‘do good’. Impact Certificates are in their infancy, with projects like Funding the Commons stress-testing them.

But, all these solutions are going to encounter very old problems with measuring impact at three levels: meta, macro and micro.

At the Meta level, we are trying to understand problems as they relate to themselves. Another way of saying this is to say, we are looking at the fundamental level of the challenge, beyond superficial characteristics, to the basic rules of the game.

Back in 2017, one of us wrote a book chapter and a thesis on one way in which power asymmetries manifest themselves in ‘development’ interventions (in that research, the kind of interventions that use sport). Although the work was specific to one type of ‘for good’ work by non-profits, the research drew from a 50+ year canon of Development Studies, Political Economy, and Sociology.

We learned that these projects could rely on tenuous conceptual frameworks that sought to sustain these power asymmetries via rendering them ‘knowable’. They achieved this by artificially condensing a huge variety of human experience into microscopic instances where the non-profit spent time with the individuals they’re meant to serve: all the complex factors that led to a change in a community were, at times attributed almost exclusively to an agencies own work.

Crypto, conversely, has huge potential to democratise stakeholders and involve them as equity partners within a project. In so doing, they can create new, potentially fairer power dynamics.

Web3 avenues for further discovery or widened application at the meta level:

• Plurality

• Bottom Up approach

• Self Sovereign identity

• QF

• SBTs/POAPs

It’s one thing to ask, ‘what worked?’ It’s quite another to ask the question ‘what do we measure, to understand if something worked or not?’ This is the Unit of Analysis problem at the macro level; what to measure; what to pay for?

Let’s imagine improving access to water in a community in Arizona. We have the building contractor drilling the well. We have the government sanitation department who are tasked with regulating it, and the community who hopefully use it, and maintain it. What do we measure, and thus pay for, if we want to know we are improving access to water? The awarding of the well contract, the boring of the hole, the completion of the project, or the community adopting it? What if we decide on any one of these, but no one actually ends up using the well (this happened, elsewhere btw, many times)?

The Unit of Analysis problem is a very real and likely inheritance Web3 will need to solve for. It’s a problem you can see well understood by contributors in the first 19 minutes here.

Web3 avenues for further discovery or widened application at the macro level:

• Vesting

• Retroactive funding

• Oracles

• Most proximate

• SBTs/POAPs

Once something that we wanted happens, we then have the problem of knowing what caused it. This is the micro problem of attribution. Did the thing that we did result in the good stuff we wanted?

When you begin with an intention, and a positive change happens shortly afterwards, it can be tempting to think you made it happen. ‘It’s too much of a coincidence for this not to be down to what we did, right?’.

Wrong, often, unfortunately. Causality is extremely difficult to prove, even in seemingly, the most nailed-on examples. This is why in medicine, researchers are cautious about attributing effects to causes. Instead, more commonly, ‘associations’ are made (‘where X happens, we often see Y also happening).

And the problem of attribution becomes even more punctuated in interventions designed for behaviour change. Socially, the problem of attribution is reducible to bias. The University of British Columbia breaks this bias down into two forms in the context of organisational change and management: ‘fundamental attribution error’, and ‘self-serving bias’.

For Impact DAOs, the challenge to overcome is to ensure we are incentivising outcomes that weren’t on their way anyway. If a community plants mangroves, and these sequester more carbon, has the planting happened because of an incentive from a DAO, or is it because of a resurgence in particular forms of community knowledge, or a governmental awareness campaign, or all?

One potential answer is it doesn’t matter. If we are funding outcomes that are going to happen anyway, at worst we may only be risking the likelihood of those outcomes happening elsewhere; even perhaps sustaining the positive outcome for longer. In the frame of this response, the outcome is the most important thing, not the intention, nor even the object of the funding. But if we are to evaluate what we did, and if it worked, we won’t know with any certainty unless we can attribute effect to cause. And this may problematise the accuracy of funding in the future, and risk higher opportunity costs.

Web 3 avenues for further discovery or widened application at the micro level:

• Stakeholder approach

• Plurality

• SBTs/POAPs

We hope you enjoyed this digest. Our humble attempt to condense some knowledge from the older Impact space. It’s by no means substantive, or complete. Merely an aperture to 3 different problem areas at three different levels, which may provoke other discussion to improve Impact DAO effectiveness in the future. We hope though that there is some value-add in the building of regen’s institutional memory, so that smarter Web3 folks can communicate and build with these problems in mind.