5ssp

MONITORING SUSTAINABILITY AND SECTOR PERFORMANCE

T5 UNICEF-SUDA2014-XX758-Noorani

 

line5

5ssp1 Joint Sector Reviews

Level: Country and subnational

 

 

Public sector performance monitoring is part of a global thrust towards improved accountability, transparency, results-based management and evidence-based decision and policy making. It is an important element of public sector reform and aims to answer key questions such as:

  • Is the delivery of public services improving?
  • How does performance compare to similar public sector bodies?
  • Are performance targets being met?
  • Should more or fewer resources be allocated to certain areas or specific public sector bodies?
  • How sustainable are the operations of public sector bodies?

As noted in the background paper for the Monitoring Sustainable WASH Service Delivery Symposium (April 2013), sub-sector performance monitoring requires a common monitoring framework that combines data from various sources at different levels, including asset inventories, service delivery characteristics, budgets and finance.

Sector performance monitoring is by definition a country-led process in which a dedicated country entity takes the lead in coordinating regular updates, engages civil society and other stakeholders in analysis and sense making, makes performance reviews accessible and feeds into global monitoring processes.

Uganda is a good example of joint performance monitoring where the government, through its Ministry of Water and Environment (MWE), takes the lead with the involvement of civil society organizations, local government and development partners, who contribute to data collection and the joint analysis and reflection.

Interesting summaries and lessons from Uganda’s water and sanitation sector performance monitoring and review are available at www.mfdr.org/sourcebook/6-5Uganda-AssessingPerformance.pdf and www.lboro.ac.uk/well/resources/Publications/Briefing%20Notes/WELL%20BN71%206pages%20No%20Crops.pdf.

 

Summary of Uganda’s Joint Sector Review success factors and lessons

  • Incentives for improved national sector monitoring and evaluation (M&E) have been enhanced in Uganda because the government is genuinely interested in reviewing sector performance.
  • A sector monitoring and evaluation group that includes representatives from major stakeholders is key to coordinating performance M&E.
  • There is national sector performance monitoring and evaluation in Uganda.
  • A gradual process of the transfer of responsibility for data analysis and performance reporting from consultants to sector government agencies has worked well.
  • A set of 'golden' performance indicators for water and sanitation enables tracking of performance against a manageable number of key indicators to inform key decisions.
  • National household survey data has been invaluable as a means of independent assessment and it enables the review of service levels by income group.
  • The use of a variety of data sources provides a more comprehensive and balanced picture.
  • Production of good quality annual performance reports enables key M&E information to be used for the government's annual planning and budgeting.

(Source: Briefing Note 7.1 ‘National sector performance monitoring and evaluation in water and sanitation in Uganda’, WEDC, www.lboro.ac.uk/well/resources/Publications/Briefing%20Notes/BN%207%201%20Sector%20performance.)

 

Summary of how Uganda’s Performance Measurement Framework embodies the principles of Management for Development Results

  • At all phases – from strategic planning through implementation to completion and beyond – the dialogue focuses on results for partner countries, development agencies, and other stakeholders.
  • A sub-group of partners has been specifically established to oversee the implementation of the Performance Measurement Framework.
  • Dialogue with stakeholders takes place through regular consultation and formal sector stakeholder meetings, such as the Joint Sector Review.
  • Actual programming, monitoring, and evaluation activities are aligned with the agreed expected results.
  • The Performance Measurement Framework has been specifically designed around the goals of the water and sanitation sector.
  • The results reporting system is kept as simple, cost-effective, and user-friendly as possible.
  • An annual Sector Report is produced that uses existing data sources wherever possible.
  • The framework manages for results by arranging resources to achieve outcomes.
  • The Performance Measurement Framework has been specifically established to address Management for Results. Thus the golden indicators have been developed to focus on sector outcomes and impacts rather than the traditional outputs of water points constructed.
  • Results information is used for management learning and decision making, as well as for reporting and accountability.
  • The Sector Report has become the presentation at the annual Joint Sector Review, which is the main policy-level decision-making body for all sector stakeholders.
  • The Performance Measurement Framework plays a key role in all the main planning tools and documents related to the role of the water and sanitation sector in poverty eradication.

(Source: Pinfold, John, ‘Uganda: Assessing Performance of the Water and Sanitation Sector’, MfDR Principles in Action: Sourcebook on Emerging Good Practices, pp. 95–100.)

 

Sustainability checks (SCs)

Concerns about the sustainability of sanitation are shifting. In the past, household sanitation programmes offering subsidized facilities were often plagued by demand-related issues that resulted in abandoned or under-used toilets. Newer Community Approaches to Total Sanitation (CATS), in which households build their own toilets as part of community-wide campaigns to eliminate open defecation, are successfully overcoming these issues.

Sustainability problems persist even in successful CATS programmes, for example in the ‘depth’ of behaviour change, the longevity of self-built latrines and the capacity of local markets to meet demand for new or upgraded facilities.

To better understand and address sustainability problems, UNICEF introduced the use of sustainability checks into WASH projects funded by the Government of Netherlands in six countries in East and Southern Africa. By the end of 2012, 14 sustainability checks were complete or underway in these countries.

Sustainability is a multi-faceted issue, and solutions must be holistic in nature, encompassing not only technology choice or community management arrangements, but the entire range of factors that affect sustainability. These include the policy context, management and institutional arrangements, financial issues, community and social aspects, technologies and supply chains.

Most importantly, issues of sustainability must be mainstreamed into the design and execution of WASH programmes.

 

Table: Indicator groups used in sustainability checks: Mozambique example

Indicator Group
Indicators (selected)
Technical
Water point functionality and use, breakdown frequency and repair time, knowledge of spare parts location and cost, distance to spare parts
Social
Water committee meeting regularity, clarity of committee roles and responsibilities, proportion of women, routine maintenance capacity, availability of local artisans for repairs
Financial
Household contributions, tariff and financial management, financial records
Institutional
Water point database existence and functionality, frequency of database updating
ODF status
Evidence of open defecation, latrine infrastructure assessment, existence of handwashing station with water and soap or ash

Source: See ‘Sustainability Checks the UNICEF Experience in ESAR Case Study’, available here

 

Sanitation criteria – Mozambique example

  • Latrines should have a durable and easily cleanable slab.
  • There should be a well-fitting lid to cover the latrine/toilet.
  • The latrine should provide privacy (with a door or curve).
  • A handwashing system should be in place.
  • Soap or ash for handwashing should be present.
  • The backyard should be clean.

ODF status is maintained if:

  • 100 per cent of inspected households have a latrine;
  • 100 per cent of latrines have a slab;
  • 100 per cent of households have a handwashing system;
  • 100 per cent of handwashing systems have soap or ash;
  • there is no visible faeces in the environment;
  • 0 per cent of subjects report open defecation by themselves or others in the last 3 days.

 

Questions used for sustainability check in Madagascar

There are three sections to the questionnaire:

  • Had the community been declared ODF? If so, who was the triggering agency, when was the triggering undertaken, and was it ODF now?
  • Were households in the community rising up the sanitation ladder?
  • Had the community received sensitisation to HWWS?

The visual inspection also has three elements:

  • Was the community visibly practising ODF (i.e. did every household visited by the enumerator have and/or use a latrine)?
  • What was the condition of the latrines visited by the enumerator?
  • What proportion of households visited by the enumerator was practising HWWS?

Questions for WASH committee and CLTS leaders

1.    What is the population of this community?
2.    How many households are there?
3a.   Has this community been declared ODF?
3b.   If yes to question 3a, when was the community declared ODF?
3c.   Who made the declaration that the village was ODF?
4.    Who was the triggering agency?
5.    Was a prize given for the award of ODF?
6.    If yes to question 5, was the prize i) cash or ii) other?
7.    Has the community adopted a regulation declaring that all who live and visit there must not practice open defecation?
8.    Does the WASH committee/community police monitor adherence to this regulation?
9.    Does the community build latrines for those unable to do so in order to ensure ODF?
10.   Is the community ODF now?
11.   How many latrines are there in the village?
12.   How many latrines are shared by more than one family?
13.   Since the community was declared ODF, how many latrines have collapsed?
14.   Of that number, how many have been rebuilt and are being used again?
15.   Since the community was first declared ODF, how many households have made replacements to the latrines (not after collapse)?
16.   What replacements have been made?
16a.  Replacements of same quality
16b.  Better latrine sanplats
16c.  Better superstructures
17.   Is there a seller/are there sellers of sanitary wares in the locality?
18.   Has the community had sensitisation on handwashing with soap?
19.   Who was the triggering agency?
20.   When did this take place?

ODF confirmation: Questions to fill in after a visual inspection around the community

21.   What proportion of the homes you visited have functioning and used latrines? (Refer particularly to the guidance below.)
22.   What is your assessment of the quality of the sanplats and superstructures, especially to last periods of rough weather etc. Will they last?
23.   What proportion of houses had very clean toilets?
24.   What proportion of houses had a handwashing point within around ten paces of the latrine?
25.   What proportion of houses had water available at that handwashing point (e.g. a functioning tippy-tappy; you can get a household member to indicate how to use it to confirm their use)?
26.   What proportion of houses had soap or other cleaning agent present at the handwashing point which is clearly being used?

 


Monitoring ODF sustainability

Experiences and challenges

The CATS approach for eliminating open defecation has expanded rapidly in ESAR in recent years, and is now widely practiced in many countries in the region. The Directorate-General for International Cooperation, Ministry of Foreign Affairs, the Netherlands (DGIS)-UNICEF projects have been modified to include CATS and, by 2011, five of the six countries had incorporated open defecation (OD) monitoring into their sustainability checks.

In an ideal scenario, monitoring the sustainability of efforts to eliminate OD in communities would use the same comprehensive toolset used for the certification of ODF status. These include techniques such as visiting all previous known OD sites, conducting transect walks with community members, rigorously checking footpaths and other signs to determine if latrines are actually being used, and extensive community consultation. However, due to time and resource constraints, the annual sustainability checks use more limited monitoring protocols.

These protocols vary from country to country. Some OD monitoring includes a wide observation radius (including in nearby fields), some limit observation to the immediate area surrounding household latrines, and some use focus group discussions in addition to observations. In Mozambique, for example, a focus group of nine people selected at random in each survey community are asked whether or not they defecate in the open and whether or not they’ve seen anyone else do it. Since handwashing with soap or ash is a criterion for ODF certification in some countries, the sustainability checks are also beginning to incorporate tools to assess this (using the proxy indicator of the existence of washing stations with the presence of soap/ash).

A key challenge for ODF sustainability monitoring through the sustainability checks is finding tools nuanced and flexible enough to assess defecation and hygiene practices and behaviour change patterns within the quick in-and-out visits typical of sustainability checks. This involves a careful mix of proxy observation indicators and community consultation tools, and the fielding of teams that can effectively use these tools. In addition, there is a need to move towards the kind of predictive indicators that are being used for water supply sustainability monitoring, such as the existence of local sanitation markets, supply chains and the reach of marketing campaigns (Source: Downs, K., ‘The Sustainability Check: A rural water supply and sanitation monitoring tool’, presentation at the University of North Carolina Water and Health Conference, 2012).

In spite of these challenges, the sustainability checks are already contributing to the body of knowledge of the elimination of OD in the region. In both Zambia and Mozambique, the sustainability checks have chronicled some relapse in ODF communities, while the latest Malawi sustainability check is documenting the impact of sanitation promotion in CATS communities that have been triggered but not yet certified.

 

Key learning points from sustainability check experience so far

Multivariate monitoring:

Findings so far highlight the importance of moving beyond technical criteria to addressing the complex causes of poor sustainability by monitoring key indicators in the institutional, social, financial and behavioural spheres.

Special tools for ODF sustainability assessment:

New methodologies may be needed for monitoring ODF sustainability that strike a balance between the comprehensive ODF assessment toolkits used during certification exercises and the more limited set of tools that are used in the context of the sustainability checks.

Predictive vulnerability monitoring:

Predictive monitoring indicators that pinpoint vulnerabilities likely to have an impact on long-term sustainability are an essential component of the sustainability check toolkit. The use of such indicators allows the sustainability checks to predict future problems at the early stages of the project cycle, when infrastructure is still new.

Balancing continuity with the need to modify monitoring toolkits:

Because the sustainability checks cover a long period of time (e.g. 8 years in Mozambique), a balance must be achieved between the need to modify sustainability check indicators and methodologies – reflecting improved monitoring practices, changes in project parameters or changes in the programming context – with the need to maintain multi-year comparability.

Multiple tools for assessing sustainability:

Sustainability checks are the centrepiece of the sustainability monitoring effort, but they are not the only tools. Evaluations and other components of the project-monitoring package also yield important information on sustainability, and the sustainability checks are complemented by special studies on specific sustainability issues, such as the drilling practices study in Malawi.

Sustainability accountability:

As the sustainability check initiative grows there is a need to refine sustainability accountability frameworks. Since UNICEF implements all projects jointly with government partners within the formal country programme agreement, responsibility for the sustainability of outputs is shared and accountability mechanisms must take this into account. Sharing accountability with national counterparts in this way will also help to promote the institutionalization of sustainability within government policies and planning.

 

5ssp2 Assessing service levels

Level: Service-provider and community

 

 

Service-level assessments are aimed at defining and measuring the actual service levels received by users. The ‘sanitation service level tool’ aggregates and benchmarks sanitation based on service levels rather than technology-related or behaviour-related indicators and takes the full delivery chain into account from containment to disposal of faecal waste.

The tool was developed to assess the disaggregated life cycle costs of net service levels received. Although the tool uses a ladder metaphor to illustrate net service levels received and applies similar terminology, service levels do not correspond to the JMP sanitation technology access ladder.

Service levels assessment parameters that are used are:

  • accessibility (number of toilets per household and distance of toilets from household);
  • use (are the latrines used?);
  • reliability (household maintenance and availability of pit-emptying services and support);
  • environmental protection (toilets constructed at least 15 m from water sources, safe disposal of latrines contents and safe re-use).

The sanitation service-level ladder covers excreta and urine management and comprises four levels, two of which represent different types of acceptable service and two represent a limited or below-standard service, which does not meet basic norms and does not properly merit the description of a service. The two levels of acceptable service are described as ’basic service’ and ‘improved service’.

1. Basic service:

At this level, all households have reasonable access to at least one safe, relatively robust, private sanitation facility, available handwashing facilities, relatively weak desludging and other long-term maintenance provisions, and non-problematic environmental impact or safe disposal of sludge. This is typical of most acceptable rural and peri-urban sanitation services.

2. Improved service:

At this level, all users have easy access at all times to a convenient, private, safe, robust sanitation facility which seals against flies and bad odours, has nearby handwashing facilities, where minimal effort is required for desludging and long-term maintenance, and there is re-use as well as safe by-products and non-problematic environmental impacts.

 

Table: The sanitation service-level ladder

 
Accessibility
Use
Reliability
Environmental protection
Improved service
Each family dwelling has one or more toilets in the compound
 
Easy access for all family dwellings
Facilities used by all household members
Regular or routine operation and maintenance (O&M) service (including pit emptying), requiring minimal effort
 
Evidence of care and cleaning of toilet
Non problematic environmental impact
 
Safe disposal and re-use of safe by-products
Basic service
Cement or impermeable slab at national norm distance from households (per household or shared)
Facilities used by some household members
Unrealiable O&M (including pit emptying) requiring high level of user effort
 
Evidence of care and cleaning of toilet
Non problematic environmental impact
 
Safe disposal
Limited service
Platform without impermeable slab separating faeces from users
No or insufficient use
No O&M (e.g. pit emptying) taking place and no evidence of cleaning or care for the toilet
Significant environmental pollution, increasing with increased population density
No service
No separation between user and faeces (e.g. open defecation)

Source: Potter, Alana, and Amah Klutse, ‘Assessing Sanitation Service Levels’, WASHCost Working Paper No 3, IRC, September 2010