Developing Spiritual Impact Metrics
A thorough, iterative process was undertaken by ICM to develop core impact metrics for spiritual formation
by caremin
Abstract
As a data-driven organization, ICM upholds the core values of effectiveness and innovation in everything it does. To measure impact, ICM conducts regular monitoring and evaluation processes during every program period, keeping track of the organization’s success indicators and ensuring that the organization continues to deliver positive change towards the communities being served. This monitoring and evaluation methodology continues to evolve as ICM seeks to innovate and improve on its processes.
This paper highlights the iterative review process conducted by ICM to identify and develop core impact metrics for the organization’s spiritual formation initiatives.
Overview
- Topic(s): Spiritual Formation Indices, Spiritual Metrics Evaluation
- Location: Philippines
- Date: 2013 ~ ongoing
Context
ICM’s vision is to see ultrapoor families released from all forms of poverty, and one of the key facets is spiritual poverty. ICM recognizes that living in extreme poverty is often characterized by a lack of hope, which is why the organization aims to transform the spiritual lives of the ultrapoor and allow them to experience renewed hope for their future. Through the 15-week core Transform program, ICM partners with pastors to provide a Biblically-based values curriculum, allowing participants to discover who God is, how much He loves His people, and what He graciously offers to those who have faith in Him.
ICM places importance in cultivating the spiritual formation of ultrapoor families. Because of ICM’s desire to continuously grow and steward the families God has entrusted to the organization, ICM invests in implementing and developing monitoring and evaluation tools used to assess Transform’s spiritual values component.
Since 2011, over 140 questions have been employed to better understand where people are in their faith journeys and whether attending Transform has made a difference in their relationship with God and their understanding of salvation. In 2014, a scoping review of measures designed to assess spiritual health and maturity was conducted. In this review, Engel’s scale was identified as a promising tool and also recommended by other organizations. Engel’s scale provided a framework to develop specific questions for the Philippine setting. During this time ICM also partnered with Back to the Bible to develop questions on spiritual disciplines and Bible engagement. These questions were then trialed in ICM’s baseline and endline participant surveys from 2014 to 2018.
In 2018, ICM was introduced to the Kingdom Impact Framework developed by EIDO research, to help the organization measure spiritual impact. A short version of this tool was tested in subsequent surveys to give the organization a snapshot of the participants’ spiritual formation before and after the program.
After undergoing several different iterations of spiritual health metrics, a thorough review of all spiritual measures used in the past decade was conducted, initially in 2019 and then completed in 2021, to reach a concise module of spiritual health questions that address ICM’s core measurement needs.
Design and Results
Design
The process of the spiritual metrics review were as follows:
- Gather anecdotal enumerator feedback on the usability, translation quality, and acceptability of the survey questions and constructs being evaluated;
- Refresh ICM’s spiritual values theory of change (ToC) with the implementation team and review all spiritual metrics used against the latest theory of change;
Identify patterns of change across the batch-by-batch pre and post survey results; - Assess all available survey data for high-value questions or metrics; and
- Review analysis and discuss the results with key ICM leaders to finalize core spiritual impact measures.
ICM Research in AidEx East Africa Webinar Series
ICM’s very own Research Associate Daryn Joy O. Go will be one of the speakers at AidEx 2021 East Africa Webinar Series! Daryn will be part of a panel discussion “Monitoring Evaluation, Accountability, Learning (MEAL), and Faith: Beyond Western Approaches”.
Freakonomics Rebroadcasts Interview with ICM’s David Sutherland with Dean Karlan and James Choi
Poverty alleviation research is a huge part of ICM’s work with people living in ultra-poverty. We are committed to the rigorous measurement of our programs.
Should NGOs Work with Religious Leaders?
A Realist Evaluation on the Role of Church Leaders in Relief Operations with NGOs published in International Journal of Disaster Risk Reduction.
ICM’s Research Published in Economist and New York Times!
ICM is pleased to announce that the six-month results of a large scale randomized controlled trial on ICM’s Transform program were released on February 19th as a National Bureau of Economic Research working paper.
Results
A total of 128 questions were examined over this period spanning from questions about the participants’ (1) understanding of God’s grace and salvations, (2) beliefs about prayer and the Bible, (3) beliefs about who God is and His sovereignty and love, (4) perceptions and attitudes towards the state of their religiosity and relationship with God, and (5) church engagement.
From these, questions that were scarcely used, questions that were generating inconclusive and inconsistent results, and questions that were difficult to understand based on enumerator feedback were removed from the roster of survey items being closely considered. Additionally, upon reviewing ICM’s spiritual values ToC with the implementing team and strategic leaders, a second round of questionnaire culling was done to generate a short list of questions that were aligned with the ToC.
The key outcomes identified through this ToC process are the following:
- To understand salvation by grace,
- To increase church engagement, and
- To increase the practice of spiritual disciplines.
From 128 questions, a final list of 27 questions were identified to be most informative in measuring spiritual formation and aligned with the updated ToC.
Upon narrowing the selection down to 27 questions, a third round of thorough assessments was employed, involving batch-by-batch comparisons from 2011 to 2019 Transform data and short reports in order to further dissect the patterns of change and each question’s ability to measure the core spiritual outcomes identified from the ToC review process. In this review stage, 11 high-value questions were identified by key ICM strategy leaders due to their alignment with ICM’s core spiritual values constructs and their ability to elicit consistent responses and generate consistent pre to post trends across Transform batches. These 11 questions showed the most potential in serving as ICM’s core spiritual impact measures[1]:
| Success Measure / Core Outcome | Construct | Question/Indicator | Indicator Type |
|---|---|---|---|
| Participants understand salvation by grace | Understand Salvation by Grace | Why do you feel that you are going to heaven? | Primary |
| Are you confident that you are going to heaven after death? | Validation question | ||
| There are many beliefs about what happens after death. Which of these statements is closest to your own belief? | Secondary | ||
| Eternal salvation is possible through grace, not works. | Validation question | ||
| Increased church engagement | Expressed allegiance to Christ through ICM church engagement | What type of church do you belong to? | Primary |
| Participants start to attend a Protestant church | What type of church do you belong to? | Primary | |
| Participants start to attend any church | Do you belong to a church (Sunday service)?
Do you belong to religious meetings (Small groups, bible study, choir, etc.)? (outside of Sunday service) |
Secondary | |
| Participants start to engage with the local evangelical church | How active are you in this church (Sunday service)?
How active are you in this religious meeting (Small groups, bible study, choir, etc.)? |
Secondary | |
| Increased practice of spiritual disciplines | Increased prayer habits | In the past 7 days, how many DAYS did you pray? | Primary |
| Increased Bible engagement | In the past 7 days, how many DAYS did you read or listen to the Bible? | Primary |