My friend Bekele Abaire from CRS Ethiopia said this to another colleague who complained that a conference presentation on sanitation & hygiene monitoring was too basic.
As I sat in a seminar about Menstrual Hygiene Management in Schools at Emory University earlier this week, I kept thinking about my trip to Vietnam to visit CARE’s projects. As we drove all over the beautiful country, my fellow travelers and I marveled at the flocks of school girls walking or biking along the side of the road in their pristine white school uniforms (called ao dai, see photos here). We thought it would be hard to keep the uniforms clean because of the dirt on the side of the road, but we didn’t think about problems they might have managing their periods.
Most women know the fear and embarrassment of visible blood stains on their clothes. For many girls in developing countries, the implications of not being able to deal with “the curse” are much worse. As a fellow female, I’m ashamed that I haven’t focused on how important it is for girls to be able to better manage their periods so that they can stay in school and pay attention while there.
Fortunately, other people are bringing more clarity and the power of evidence to this issue. Bethany Caruso and her team at the Center for Global Safe Water at Emory University collaborated with UNICEF to look at the challenges girls face in menstrual hygiene management in three countries (see an overview of the program here) In the Philippines, Bolivia, and Rwanda girls face similar challenges:
- Fear, shame and teasing from boys
- Lack of information
- Lack of access to pads (many girls use rags)
- Inadequate or no school toilets
All of these can lead to girls skipping school or being distracted while in class, unplanned pregnancies, and infections.
There are a variety of unrelated factors that lead to these challenges, but the water, sanitation, and hygiene sector could help by making sure that girls’ toilet facilities are private and girl-friendly (which might mean they are far from boys’ toilets, have locking doors, more space, mirrors and sinks inside, and contain covered trash cans for bloody toilet paper or incinerators for used pads). Training of teachers and girls on how to use the facilities is also important.
“Gender equity” has been a buzz-phrase in recent years, but as Bethany said, “instead of equity of inputs, we need to think about equity of outcomes.” To achieve better health and educational outcomes for girls, the water, sanitation and hygiene sector must consider its role in addressing these important findings.
By Susan Davis, Executive Director, Improve International
Last week I went to the Sustainable WASH Forum and Donor Dialogues in DC. A theme of the conversations was roles and responsibilities, especially the roles of governments. One interesting debate was about who should be responsible for monitoring. Some said that governments should be solely responsible. There are some governments who are leading the way on this, but I and others believe that this doesn’t mean that development organizations shouldn’t also be accountable for their own work. If an organization visits water and toilet systems for years after they are built, they can learn from their successes and failures and make their future work better.
Since many organizations only do monitoring & evaluation (M&E) during development programs (see my thoughts after the Learn MandE conference), I think we need to use a new term like “services monitoring” to refer to the need for a way of confirming that water and sanitation services are still available to people.
Why is service monitoring important?
|783 million people without access to improved source of water[i]3 billion without access to safe water[ii]4 billion without access to safe, permanent, in home water[iii]|
|2.5 billion people without adequate sanitation[i]4.1 billion lack access to improved sanitation[iv]|
|water and sanitation systems that fail within a few years of construction[v]|
Less than 5%
|water systems that are visited at least once after they are built|
Less than 1%
|water systems and toilets that are monitored regularly for the long-term after they are built|
Long-term service monitoring is critical for the ongoing improvement of implementing organization practice and understanding, as well as donor policies. Beyond helping individual organizations learn from their experience, services monitoring could reveal geographical or sectoral trends. What if each year, USAID, other government aid agencies, development banks, and major foundations pooled a portion of their funds for water & sanitation projects ? These funds could be used to ensure service monitoring for all (or a sample of) previous water and sanitation systems funded by those donors in a country or region.
With this information, they could identify region-wide problems and solutions. For example, declining amounts of water available from spring fed systems in a geographic region could point to a need for investing in water source protection and installation of household water meters to reduce leaks and wastage.
A way forward
To remove some of the barriers to ongoing service monitoring, we recommend a way forward below.
- A percentage of funds (perhaps 3-5%) of each donor’s funding for water, sanitation and hygiene programs is contributed to a pool for service monitoring each year.
- The funds could be used to monitor a sample of past programs funded by the donors. For example, those 5, 10, and 15 years old. That way we get the learning now and can use it to change programs moving forward.
- Keep the monitoring indicators very basic and in line with government monitoring protocols, where present.
- Development organizations should be responsible for ensuring that service monitoring happens, but doesn’t have to use their own staff. For example, where governments have a robust system of national monitoring, the organization could pull recent relevant government data.
- Engage an independent auditor to verify a sample of results.
As more service monitoring data become available and accessible, we’ll get past the statistics to specifics, leading to learning, and more effective performance. Thus, people in developing countries will have a better chance at reaping the life-changing benefits of safe water for life.
by Susan Davis, Executive Director, Improve International
In an earlier blog I wrote about the confusion on how we should estimate how many people lack access to water. A recent article by the UNC Water Institute (full disclosure: I could only read the abstract without paying for the article) suggests some rather shitty numbers:
We estimate that in 2010, 40% of the global population (2.8 billion people) used improved sanitation, as opposed to the estimate of 62% (4.3 billion people) from the WHO/UNICEF Joint Monitoring Programme (JMP), and that 4.1 billion people lacked access to an improved sanitation facility.
The abstract also shares this sunny yet horrifying cartoon that shows the direct line from untreated sewerage (poop) to people’s drinking water:
So, this is yet another reason to rethink how we frame the global development goals. You, yes you, can vote on the changes that would make the most difference to you in the world. If you think of this graphic and remember that in a way we are all drinking the same water, you might want to include “access to water & sanitation” in your priorities.
Yesterday I chaired a panel at the Philadelphia Global Water Initiative conference on performance indicators. Our panel (“Perspectives & Experiences from National & International Organizations” had the honor of being interrupted by the Mayor of Philadelphia, Michael Nutter (a “green city” rock star). Below I’ve shared my talking points for introducing the panel in blog form.
Indicator: A thing, esp. a trend or fact, that indicates the state or level of something
Monitor: Observe and check the progress or quality of (something) over a period of time; keep under systematic review.
We are intimately familiar with performance indicators and monitoring them in our personal lives (body weight, baby length, miles per gallon) and in world in general (unemployment rates, housing starts, Dow Jones Index). So why do we need to monitor performance indicators in international humanitarian and development work?
- Doing good work: Well, simply because we want to know if we’re doing what we think we’re doing. When monitored during a program, data on these indicators can help us make mid-course corrections. When monitored after program completion, they can help us change the way we design future programs.
- Reporting to donors: Some of our donors want to know if their money accomplished what we said we’d accomplish.
- Advocacy for issue: And it’s helpful for advocacy efforts to be able to say to the general public that our organization is moving the needle on the big social problem we’re trying to solve; advocacy to governments where we work and their role in scaling and sustaining service delivery; and advocacy to our peer organizations on why it’s important to work in coordination, etc.
There are many organizations who have some variation of a monitoring program. Monitoring status of indicators is just part of a cycle, however. We need to also evaluate, learn, and reform our work based on what we’ve learned.
It may be actually a good sign, although it perhaps should have been discussed earlier in the history of development, that many questions are being debated in the water & sanitation sector:
- What indicators should we measure? Outputs or outcomes? Quantitative or qualitative? For example, the WASH Cost initiative is telling us we need to start considering costs. UNICEF is looking hard at equity of water and sanitation access.
- What should be mandatory (and measured the same way across the board) and which should be optional (recognizing that there are scarce financial resources)? There is an effort underway to identify common core indicators called WASH Monitoring & Evaluation initiative
- Should we / how do we consider cross-sectoral indicators (agriculture, health, economic, environment)
- Should we / how can we include customer (aka beneficiary) goals and satisfaction?
- Do we need to be able to dis-aggregate the data by gender, socio-economic status, age, ethnicity, or other factors?
- What tools (surveys or assessments) should we use?
- Who should collect the information, how much (every single water point?) and how often (every day, every month, every year)? And for how long?
- Who should pay for collecting information?
- Who should “own” the information? Is it ethical to make this information public if it contains financial and/or health information?
- How do we know the information are accurate? And does access to safe water mean the same thing as use of safe water?
- Who is responsible for doing something about the information we get (for example, if we see failures in water / sanitation projects years after implementation, who should do something about that?)
Okay, I would love to tell you that yesterday’s conference provided all the answers, but I can’t. [There were some good relevant resources highlighted in last week's TweetChat on water, sanitation & hygiene evaluation.] However, it’s only by knowing what we don’t know that we can start learning. What’s your question about performance indicators? Better yet, what’s your answer?
Last week I had the opportunity to speak to students in Dr. Christine Moe‘s water and sanitation class at Emory University. The theme was “10 un-FAQs”; or 10 questions that development professionals should ask themselves more frequently before embarking on water and sanitation projects. I chose this because after 13 years in international development, I have a lot of questions. I’m going to list them here and elaborate on each in future blogs.
un-FAQ 1: Why do people want a safe water project?
un FAQ 2: What happened before you got there?
un FAQ 3: Are you the best organization to help?
un FAQ 4: Is the technology you are planning to use appropriate?
un FAQ 5: How much will it really cost?
un FAQ 6: Who else is working in this area?
un FAQ 7: Who should pay for it?
un FAQ 8: Could your money be better spent on something besides a project?
un FAQ 9: What impacts can you expect?
un FAQ 10: What happens after you leave?
By Susan Davis, Executive Director, Improve International
My dad used to say, as we would watch some politician or sport star apologizing for something on TV, “He’s not sorry he did it. He’s sorry he got caught.”
I was reminded of this when I was with my friend Katie in the car. We had passed a police car on the side of the road. She tapped the brakes and I laughed. “It’s such a gut response, isn’t it?” She agreed with a smile and said her other instinct, oddly, is to turn down the radio.
There’s something about knowing you might get caught that makes you behave better (except on “reality” shows). As a young driver, I used to get a big speeding ticket each year. It only took about three $150 tickets for me to realize that it was a sort of tax on being stupid. Now, on the rare occasions that I drive a car, I stick closer to the speed limit. The presence of those nifty little cameras at stoplights helps me to make smarter decisions about what to do at yellow lights, too.
What if we always drove like there were police cars and cameras around?
There aren’t regulations that apply to effectiveness of water supply systems in developing countries. Right now the customers (beneficiaries) of these systems and donors who support them must rely on the good intentions of organizations that build them. With failure rates remaining high, it’s obvious that good intentions don’t always result in sustainable projects. Monitoring and evaluating a water project after it is finished is a form of self-regulation, but few implementing organizations perform these tasks well, if at all. Reasons include the difficulty and expense to visit remote communities with bad roads after project completion.
But would regulation be enough to make water projects last? It would require some enforcement. Sometimes regulation, or self-regulation, isn’t enough; you need peer pressure too. For example, some people may not care very much about laws for child seats in cars and they may not believe they are important for child safety. But maybe they use child seats to avoid embarrassment. They do not want family or friends or even parking lot attendants to think they are bad parents. These drivers imagine people will want to stone you if you even THINK about driving around with your child in your lap. (Even if you are trying to escape the paparazzi.)
Unlike the case of child seats, there is little pressure from peer organizations to perform well and there are few other types of incentives to achieve and verify effectiveness of water supply systems. This may be one of the main reasons that some organizations and donors fail to monitor and evaluate water systems or even visit them regularly. The few donors who can afford to visit often see just the best projects. I know; I remember the conversations with field staff when setting up a few of these visits. This is understandable; it’s hard enough to raise money without having to explain why projects sometimes go bad, even if we can still learn from those.
But what if we designed each water or toilet project like anyone might pop by? What if we knew a BBC reporter, our dads, our donors, or the donors’ great-great-grandkids might drive by the community and stop for a glass of water and a quick pee?
Of course, even if we do not expect anyone to visit, we should be most concerned about the opinions of the customers: the people who live in the community and need to use the water system and toilets every day, for generations. They are the ones who will be the most sorry if we don’t get caught doing work that doesn’t last.
By Susan Davis, Executive Director, Improve International
[NOTE: As of March 2013, the Accountability Forum is now called the Water for Life Rating]
I’m very excited to be supporting the first Accountability Forum, and not just for professional reasons. It feels serendipitous that my first project with Improve International will involve the Honduran water and sanitation organization COCEPRADIL (Comité Central Pro-Agua y Desarrollo Integral de Lempira). My very first visit to a developing country was Honduras, and the first water projects I ever helped to evaluate were built by COCEPRADIL. Twelve years later, one of my favorite stories is about the time we had to hike up a hill for three hours to get to one of the most remote communities in Lempira. My American colleagues and I were very slow hikers compared to the Hondurans who were used to zipping up and down the steep hills. The COCEPRADIL staff insisted we ride burros back down the hill so that we could get back to the hotel before dark.
So, back to the Accountability Forum. It is an innovative approach to address a sticky problem the water and sanitation world has been talking about for decades: the high project failure rate. Marla Smith-Nilson, Executive Director of Water1st International, who developed the concept for the Accountability Forum, explained her eureka moment. In November 2009, she attended Joe Cook’s presentation at an American Water Works Association Conference. Joe, an associate professor of public affairs at the University of Washington, suggested accountability clubs as a possible way to ensure sustainability of water and sanitation projects. Marla said, “Kirk [Anderson, her co-worker] and I thought it was a fantastic idea and ran with it.”
To develop the idea, Marla and Kirk met with Joe and his colleague, Mary Kay Gugerty. Mary Kay co-wrote a thought-provoking paper on voluntary self-regulation by non-profit organizations. They strengthened the concept by including third party evaluators. In the summer of 2010, Water1st obtained funding for launching the Forum. Marla announced the Accountability Forum at the October 12, 2010 Sustainable WASH meeting in DC. I was at that meeting representing Water For People, and I knew they would want to be part of it. Coincidentally, Water For People had begun its own series of Accountability Summits to present monitoring data from its water and sanitation projects to supporters and peers in the US.
And now, the Accountability Forum is really happening! The Honduran organization COCEPRADIL has courageously agreed to be the first organization evaluated by peers and third parties in the first Accountability Forum evaluation visit. Central American and other representatives of Water For People, Living Water International, Save the Children, and A Childs Right are planning to join as members and observers, and we are still encouraging others to join.
The dates are set: December 12 -16, 2011. Improve International will facilitate the process by building consensus among the participants on the evaluation criteria, recruiting appropriate candidates for independent evaluation roles, and helping COCEPRADIL to prepare their project portfolio.
This pilot will inform future Accountability Forums in different countries. The goal is to evaluate organizations on the actual sustainability of the projects they are implementing. We hope these types of events will create an environment where monitoring and evaluation is rewarded. Donors would clearly benefit from information provided by peers and a reputable, independent source.
On top of the professional thrill of being part of this experiment, I have to admit I’m hoping to get another burro ride.
If you are interested in joining the Accountability Forum in Honduras or future countries, email me a photo of your favorite burro at email@example.com
By Susan Davis, Executive Director, Improve International
Have you ever given to an international charity? How did you make the choice? International humanitarian work can be an important tool for improving people’s lives in developing countries. While many of us see it as a moral obligation to help people in need, good intentions are not enough. We need to have evidence of what works – and what will continue to work – to improve this vital support.
For example, an estimated 30% to 50% of water systems built by humanitarian organizations in developing countries have broken. This doesn’t mean you shouldn’t continue to give to international causes; rather, it emphasizes the importance of your understanding as a donor how to find the best organizations and types of projects to support.
Improve International will help you “give smarter”. Improve International will evaluate humanitarian projects. Our ultimate goal is to improve the chances that humanitarian efforts will improve people’s lives over long periods of time.
Our first area of focus is drinking water and toilet projects. We will arrange for independent experts to evaluate the effectiveness of these projects. Whenever possible we will engage local experts to build local capacity for monitoring and evaluation.
We will collect and analyze the results of on-site evaluations to determine patterns of what works and what does not work. We will also collaborate with other organizations to promote learning and sharing across humanitarian organizations, donors, and the governments of the developing countries so that they can look for opportunities for innovation and make better decisions about how use their funds.
Who will benefit from these independent evaluations?
Beneficiaries – The people that donors and humanitarian organizations are trying to help will benefit the most by having better quality support.
Donors like you - People want to make good decisions about their philanthropy, but often don’t have enough information. Currently, most donors must rely only on success stories provided by the humanitarian organizations.
Humanitarian organizations – Rather than trying to point out mistakes, we aim to help humanitarian organizations not only demonstrate their success and impact but also to learn from what didn’t work so they can innovate and improve.
How is Improve International different? While rating agencies like Charity Navigator and Guidestar provide valuable information, they are based on data reported by the humanitarian organizations themselves. Improve International is different because we will look at the actual projects in the developing countries.