Over the last decade and a half there's been a major push by economists to do rigorous research on poverty — basically to run experiments to figure out which solutions actually work.
But putting a halt to those that come up short is easier said than done.
Now one prominent charity is trying to set an example. It's called No Lean Season, and it's an offshoot of a larger charity called Evidence Action. No Lean Season's goal is to help rural farmhands in Bangladesh during what's called the lean season — a roughly three-month period every winter when the fields lie empty and there's no work.
And No Lean Season has decided to stop raising money until it figures out why its program, which was once enormously successful, is no longer producing results.
No Lean Season is one of the most prominent ventures from Evidence Action, whose mission is to look for the most promising experiments, then scale them up massively.
"Our goal is to reach the lives of millions, or tens of millions of people," says Karen Levy of Evidence Action, who for the last 20 years has worked on programs to help the world's poorest people.
And as it scales up, Evidence Action makes a vow. At every stage it will run tests to see if its programs are still working. "If we find that they're not having impact," says Levy, "then we'll shut it down."
Too often, says Levy, the kind of charitable initiatives that get funded are "good proposals. Not necessarily good, impactful programs. You can keep an ineffective program funded for years if you're good at fund-raising."
That's because, in contrast to the business world, the people being served by a charity — the customers, if you will — aren't the ones paying. Instead the bill goes to donors such as foundations and governments. That means they often base their decisions on whether a program sounds good.
"You can keep an ineffective program funded for years if you're good at fund-raising," says Levy.
The idea for No Lean Season came from Ahmed Mushfiq Mobarak, a development economist at Yale. During Bangladesh's lean season he says, "up to 45 percent of people report that they regularly miss meals."
Instead of three meals they'll cut down to two — and in some cases from two meals a day to one. It's particularly hard on children. The resulting malnutrition can lead to stunting, which, in turn, causes long-term cognitive problems.
Traditional aid such as food distribution or food-for-work programs has proved too expensive to reach all of these farmhands. But back in 2008, Mobarak had an idea: Instead of trying to bring jobs to where the farmhands are living, why not bring the farmhands to where the jobs are?
Specifically, Mobarak's proposal was to offer the farmhands a tiny loan — around $20 — to buy a bus ticket to the closest urban area where, for the duration of the lean season, they could get a job, like pulling a rickshaw.
Mobarak's early tests were so intriguing Evidence Action decided to team up with him to expand on the idea. The charities first round of loans, in 2014, targeted 3,500 households.
Levy was in the D.C. office when the results came in. "We were a bunch of geeks jumping up and down in the conference room," she recalls, laughing.
The loan had roughly doubled the number of people who decided to try their luck in the city from 34 percent to 77 percent. Most importantly, those who went earned enough to give their families an additional meal each day.
Results of that magnitude are rare in the aid world, says Levy. So Evidence Action went bigger: rolling out the program under the name No Lean Season and reaching out to 20,000 households. Those results were so impressive the charity was able to raise $11 million dollars to scale up to 160,000 households in late 2017.
You can read NPR's report on the rollout here.
Then, this past fall, the results from that 2017 round came in. "It was very sad. I was very disappointed," says Levy.
This time around the loan offer had not induced anyone who wasn't already planning to go to the city to make the trip.
Mobarak suspects there was insufficient communication between Evidence Action and the organization it contracted with to go village-to-village making loan offers — a microcredit group called RDRS Bangladesh that has a long history of working in rural communities and a large enough staff to carry out Evidence Action's loan program on a massive scale.
The contract consisted of two steps: First RDRS would survey rural households to come up with a pool of eligible farmhands. Then it would go back to those in the pool and offer them the loan.
The trouble, suggests Mobarak, is that when the contract with Evidence Action was drawn up, RDRS estimated it would be able to come up with a pool of 100,000 people. Based on that estimate, Evidence Action calculated the number of people who would likely end up agreeing to take the loan. RDRS then used that calculation to set targets for their individual loan officers.
In fact, it turned out that RDRS was far more effective at the first step than anticipated — they came up with about 160,000 eligible people. But they did not adjust accordingly the number of loans each staffer was encouraged to make.
Mobarak's hypothesis is that the staffers stopped offering loans after they had reached their targets. They set their goals too low, and so "the people who are going to stand in line first are going to be the people who are going to go anyway," says Mobarak. They are the most eager, after all.
Evidence Action is now in the midst of one final round of loans to see if they can verify that this was the issue and can fix it. Mobarak, for instance, personally traveled to Bangladesh to meet with RDRS field staff.
But Evidence Action has also announced that if they can't confirm the source of the problem and fix it, they will not seek new funding for No Lean Season.
That's an unusual move, says Catherine Hollander. She's with GiveWell — an organization that researches charities to come up with an annual recommended shortlist for donors. In the fall of 2017 Evidence Action's program made the list. For 2018 the charity effectively said, Don't even consider us.
But Hollander says it was also a smart move: "It makes us trust them more."
Evidence Action's Levy says she hopes their example will inspire other charities to own up to failure.
"It's sad when something that you thought might actually solve a problem turns out not to. But it's much sadder to waste resources."
Copyright 2019 NPR. To see more, visit https://www.npr.org.
