This is a guest blog by Vanitha Sivarajan. Her background includes over 10 years of biodiversity conservation and participatory water resource management with local communities, non-governmental organizations, governmental agencies, and the private sector. She has worked both domestically and internationally, with a focus on Latin America and India. Currently Vanitha is an Environmental Consultant who provides NGO clients in the water and climate sectors with a variety of programmatic services.
Improve International is working to promote accountability from groups who build water and sanitation systems in developing countries. One way for those groups to demonstrate accountability (and to learn from their successes and mistakes) is to have an independent evaluation done. Like an independent financial audit, independent program evaluations are an objective way to determine the ability of a water and sanitation organization to successfully deliver safe water and convenient toilets for their constituents. The difference is that rather than looking through the financial files, a program evaluation requires experts to visit all or a sample of the assisted communities.
Out of the 500+ water organizations identified, we could only find 37 independent evaluations by web searches.
This information can be useful for donors, supporters, the communities they are serving, the governments of the communities they are serving, the general public, etc.
I offered to help Improve International find out how many water and sanitation organizations have had independent evaluations. Improve International challenged me to first develop a list of at least 100 water and sanitation organizations, and to find at least 20 independent evaluations. First, I wanted to make sure I knew what I was looking for.
What is an independent evaluation?
Vanitha interviews community members for an evaluation of a water project for the DHAN Foundation in Tamil Nadu, India
According to the World Bank, the definition of an independent evaluation is: “If independent evaluation is to be impartial, its findings, analyses, and conclusions must be free from bias. This means that [the evaluator] must be independent from line management at all stages of the process, including planning of work programs, formulation of terms of reference, staffing of evaluation teams, and clearance of reports.” I used this as a guide in identifying which evaluations were independent.
How many WASH organizations are there?
I began my quest by compiling a list of organizations that are implementing, supporting, and/or funding water, sanitation, and hygiene projects in developing countries. I quickly found that not all water and sanitation organizations are created equal. When I think of water organizations, traditional ones such as CARE, World Vision, Catholic Relief Services, etc. immediately come to mind. However, in the last decade or so, there has been an exponential increase in water organizations, from traditional NGOs, to startup companies, to social enterprises, to bottled water companies that support global water projects with their profits. I couldn’t find one go-to directory for all the organizations that fit these categories. However, I got a good start with World Bank’s Water and Sanitation Program, Water for the Ages, filtersfast, WASHfunders.org, Twitter lists, and Google searches. Susan at Improve International added in several that she found as well. By the time we were done, we came up with a list of more than 500 water and sanitation organizations! This list will, in the near future, be included on WASHfunders.org.
How do we know who’s doing good work?
Now that I had a list of organizations, it was time to look for independent evaluations. Looking to the organizations’ websites themselves proved fruitless, as it was rare to find actual independent evaluations listed on their site. While searching, I realized that sometimes independent evaluations were called “external” evaluations or “third-party” evaluations. Out of the 500+ water organizations identified, we could only find 37 independent evaluations by web searches.
I did not find a site that listed organizations that had either a) done independent evaluations, b) rated them using standardized indicators, or c) provided easy and clear information on how to determine the efficiency or success of a water and sanitation organization (or any international aid or development organization for that matter).
Philanthropedia’s Ranked Nonprofits: International Water, Sanitation, and Hygiene site came the closest with opinion-based rankings, while the Better Business Bureau’s 20 Standards for Charity Accountability and Charity Navigator’s Accountability and Transparency Rating provided overall rankings based on the organizations’ self reported information.
My search resulted in some interesting finds:
- The oldest independent evaluations I found were several for UNICEF dated 1991-1995 by various organizations such as the Swedish International Development Cooperation Agency and the Regional Center for Development Management & Research. UNICEF also evaluates other organizations such as PlayPumps International in 2007.
- The American Red Cross, which I don’t usually think of as a water organization, has multiple evaluations of water systems they built in Central America after Hurricane Mitch that were done by the Centers for Disease Control between the years 2001-2009.
- There were several other evaluations of water and sanitation projects in response to disaster relief such as tsunamis or earthquakes. For example, a group called the Active Learning Network for Accountability and Performance in Humanitarian Action did a Mid-Term Independent Evaluation Report of Save The Children’s Tsunami Response Programme in 2008 that encompasses multiple aspects besides water and sanitation such as construction, health, education, livelihoods, child protection, etc.
- Multilateral banks such as the World Bank, Asian Development Bank, and Inter-American Development Bank have independent evaluation departments that serve as an unbiased separate arm that evaluates the Bank’s program work. However, the evaluation documents that I found were long and complex and didn’t highlight its water and sanitation components as they primarily aimed to provide comprehensive programmatic information.
What I didn’t find
The wide discrepancy of the ratio of easily found independent evaluations to the total number of water organizations out there indicates that these are an important, but overlooked, aspect of an organization’s monitoring and evaluation program, not to mention its fundraising program. Imagine if I was a donor trying to find objective information on which water organizations had done effective and sustainable work. It was a challenge to find just a comprehensive list of water organizations, not to mention a list of independent evaluations.
Other aspects of this research that were difficult include:
- Easily identifying the stage in the project that the evaluation was done. It would be ideal to do an evaluation mid-project, post-project, and then another at least once 10-50 years later, all using the same indicators.
- Knowing what is done with the evaluations after they are read: do organizations address challenges surfaced in the report and then re-evaluate later?
- Finding evaluations with a completely unbiased or uninvolved team. Some evaluations that claimed to be done independently included a staff member of the organization it was evaluating.
- Finding highlighted mentions of failed aspects of the project/program. Who follows up after these types of negative evaluation findings? What and how can other organizations learn from these?
- Who funds the independent evaluations? Are they included in organizational budgets for monitoring & evaluation?
Improve International and I hope that publishing this list of organizations will prompt water and sanitation organizations to share their independent evaluations. We imagine that they might be in paper version, or on someone’s hard drive. We’d like to get the information out there so we can all learn from it. Are there common challenges we all face? Is there someone who’s figured out how to promote sustainable hand washing behaviors? It’s not too late – let us know if your organization has had (or has done) an independent evaluation – email the link or the document to email@example.com