Difficulties Demonstrating the Impact of OAA Programming and Glimmers of Hope

Abstract:

The positive impacts of Older Americans Act (OAA) programs are often accepted as fact. If a home-bound older adult receives a home-delivered meal, or family caregivers attend support groups, or a tribal elder uses a wheelchair from a lending closet, then these program participants are better off. But is it enough to accept that these services are beneficial? In supporting the aims of the OAA, there is often pressure at local, state, and federal levels to “show me the data.” This article reviews pressures and opportunities as the aging network demonstrates the value and impact of the OAA.

Keywords:

evaluation, performance metrics, Older Americans Act, aging network

State Units on Aging, Area Agencies on Aging (AAAs), and local community-based organizations provide “a comprehensive array of community-based, long-term care services adequate to appropriately sustain older people in their communities and in their homes, including support to family members and other persons providing voluntary care to older individuals needing long-term care services” (OAA, 2020). This lofty objective of delivering home- and community-based services (HCBS) has for decades been addressed by the aging network. During the same period of time, it has been accepted that the delivery of Older Americans Act (OAA) services has been beneficial for participants, and there is evidence that OAA services help older adults continue to live at home and not in a nursing home (ACL, 2019).

Section 206 of the OAA explicitly outlines the requirement to “measure and evaluate the impact of all programs authorized by this Act, their effectiveness in achieving stated goals in general, and in relation to their cost, their impact on related programs, their effectiveness in targeting for services under this Act unserved older individuals with greatest economic need (including low-income minority individuals and older individuals residing in rural areas) and unserved older individuals with greatest social need (including low-income minority individuals and older individuals residing in rural areas), and their structure and mechanisms for delivery of services, including, where appropriate, comparisons with appropriate control groups composed of persons who have not participated in such programs” (OAA, 2020, pg. 35).

Even with this OAA requirement, there are pressures related to data and evaluation of OAA programs that involve the network at all levels—from consumers, local providers, state agencies, and federal policymakers. The types of pressures span the practical to the evaluative to the political. Despite these pressures, there are also new opportunities emerging at the federal level that will elevate the aging network’s ability to demonstrate the value of its programs and systems.

Pressure Pushing Down on Me, Pressing Down on You

OAA program managers have ongoing requests about the services being provided, and many of these requests rely upon data. Requests may come from an AAA director who wants to share how wide-reaching OAA services are in a specific county or portion of a state. At the federal-level, justifications for future funding and information on program performance rely upon summary information from across the country and are required annually (ACL, n.d.). The challenges of obtaining data and conducting evaluations of OAA programs encompass practical, evaluative, and political considerations.

The Practical

Two key questions when it comes to practical issues about data and evaluating programs are:

• how will the data be collected?
• how will the results be shared?

Asking clients about their experience with a case manager, counting the number of rides provided in a rural county, or submitting a state’s annual summary report of services and expenses—these are all essential tasks needed for reporting on the impact of OAA programs, and ideally these tasks are completed in a consistent way each year and by everyone involved.

The National Survey of Older Americans Act Participants (NSOAAP) is an excellent example of the practical challenges of collecting data and sharing the results. The NSOAAP is collected specifically “to obtain performance outcome measurement information” (AGing, Independence, and Disability, n.d.), which informs how well OAA programs are addressing client outcomes, delivering services to at-risk populations, and improving programs and services efficiency (ACL, n.d.).

Information from local AAAs is needed before the NSOAAP questions are even asked of clients. Data provided by the clients then is used in reporting OAA performance outcomes. This is a complicated process. First, a group of AAAs are selected based upon budget size and geographic location—these different layers or details are needed because the NSOAAP is nationally representative—and the information is used to ensure adequate representation among large and small AAAs across the country.

Second, within the selected AAAs, a group of clients is further chosen by service type. In this case the NSOAAP focuses on OAA clients who use home-delivered meals, congregate meals, homemaker services, transportation, case management, or caregiver services (NSOAAP, n.d.). After completing these steps to come up with a list of clients invited to respond to the survey, new challenges arise.

The NSOAAP is collected by an interviewer calling the service client and completing an approximately 30-minute interview. But, clients change phone numbers and cannot be reached. Clients screen calls and may not answer. Clients may have limited cell phone minutes and not be willing to use those minutes on an interview. Clients may have hearing problems and have a difficult time completing the interview. With great patience on the part of the client and diligence on the part of the interview team, these challenges can be overcome; and across several months, enough people will complete the survey to result in nationally representative data that tells us how OAA services are performing.

Once the NSOAAP data is collected the results are used in reports to Congress, such as ACL’s Justification of Estimates for Appropriations Committees (i.e., the Congressional Justification; ACL, n.d.), which accompanies the President’s budget every year. Because of the NSOAAP and the publicly available Congressional Justification, we know these details about OAA transportation clients and services:

• 46 percent of OAA-funded transportation participants are mobility impaired (i.e., they do not own a car, or they do not drive and are not near public transportation (ACL, n.d.).
• 71 percent of transportation participants have at least one chronic condition, which could limit their ability to navigate safely (ACL, n.d.).
• 96 percent of transportation participants take daily medications, with more than 16 percent taking 10 to 25 medications daily (ACL, n.d.).
• 53 percent of transportation participants rely upon OAA services “for the majority of their transportation needs and would otherwise be homebound” (ACL, n.d.; p. 51).

More fun facts can be found by reading the Congressional Justification (ACL, n.d.), but this raises the question: How many people know these annual, 300-page summary reports are available? For those interested, the data (and customized tables to help one look at details by demographic characteristics such as age group or race) is available online.

ACL’s AGing, Independence, and Disability (AGID) Program Data Portal holds a trove of performance results, surveys, and other data files related to ACL programs, including the OAA programs. AGID is a practical resource for students, researchers, policymakers, and staff across the aging network who are looking for evidence and information on OAA programs’ impact. As with collecting the NSOAAP, sharing data results requires patience, diligence, and some practical acumen.

The Evaluative

At present there are some limitations to the data available to determine the impact of OAA programs and services. While Section 206 requires evaluating OAA programs’ effectiveness in “targeting for services under this Act,” (OAA, 2020; p. 35), one needs to consider how targeting is assessed. Section 206 outlines some examples of who should be targeted for OAA services:

“unserved older individuals with greatest economic need (including low-income minority individuals and older individuals residing in rural areas) and unserved older individuals with greatest social need (including low-income minority individuals and older individuals residing in rural areas)” (OAA, 2020; p. 35).

Using older individuals with low income as an example, Census data can be used to determine what percentage of the population older than age 60 (the age for OAA services) have low incomes. Then data from OAA State Program Reports can be compared. If the percentage of OAA clients with low income exceeds the percentage of all U.S. older adults with low income, then ACL can indicate that it has met (or exceeded) the goal of targeting the underserved as defined by older adults with low income (see p. 47, ACL, n.d.).

This comparison process for determining whether targets are met works in some instances; however, there is a lack of national and OAA data on underserved populations such as LGBTQ+ older adults. Using the LGBTQ+ older adult population as an example, the national Census cannot provide a definitive estimate of how many adults older than age 60 identify as LGBTQ+ (SAGE, n.d.). Although the 2020 Census introduced a question asking about whether an individual is part of a “same sex couple,” this still does not adequately address sexual orientation or gender identity (Morgan et al., 2020).

In 2013, the National Health Interview Survey (NHIS) included a question on sexual orientation revealing that 96.6 percent of the ages 18 and older population identified as straight (Dahlhamer et al., 2014). The NHIS report authors note that because the percentage of people identifying as sexual minorities was small, “there is a heightened risk of generating unreliable estimates when exploring the health of the sexual minority population. Researchers and NHIS data users should be careful not to draw conclusions from statistically unreliable estimates based on small sample sizes” (p. 12–13).

As a follow-up, additional analyses were performed combining data from the 2013 and 2014 NHIS and looking specifically at the U.S. population of people ages 65 and older. This additional analysis suggested that less than 1 percent of the U.S. population ages 65 and older is a sexual minority (i.e., lesbian, gay, bisexual; Dragon et al., 2017).

Without a solid percentage of how many U.S. older adults identify as sexual or gender minorities, it is difficult to know if OAA services are reaching the LGBTQ+ population. The NSOAAP has included a question about sexual orientation of OAA clients since 2014. However, similar to the concerns with the NHIS and a small group of people identifying as a sexual minority (i.e., lesbian, gay, bisexual), no data on sexual orientation of OAA clients has been made publicly available (AGID, n.d.).

For several years, ACL has worked with other federal agencies to address the issue of how best to collect information on sexual orientation and gender identity across the lifespan (Interagency Working Group on Sexual Orientation and Gender Identity [IWG], 2016; Morgan et al., 2020; National Academies of Sciences, Engineering, and Medicine, n.d.), but decisions about how survey questions should be asked or which sexual orientation and gender identity categories should be used take time and feedback from researchers, and most importantly, from people represented in these categories.

The Political

With the emphasis on evaluation in Section 206 of the OAA, it is no surprise that the Act also indicates how much money should be funded for evaluation. What is a bit surprising is the amount of funding allocated. Section 206 (h) indicates: “(h) From the total amount appropriated for each fiscal year to carry out title III, the Secretary may use such sums as may be necessary, but not to exceed ½ of 1 percent of such amount, for purposes of conducting evaluations under this section” (OAA, 2020; p. 36).

That amount of ½ of 1 percent is 0.50 percent of what is appropriated to Title III (note: COVID-19 supplemental funds supporting Title III programs did not increase the funds available for evaluation). In FY2021 and FY2022, ACL submitted proposals to increase the evaluation funds by removing the language “½ of,” which would double the funds available for OAA program evaluation. Accepting this proposal would increase the maximum allowed for evaluation from approximately $7.78 million to approximately $15.47 million for FY2022 (ACL, n.d.).

A Change Will Do You Good … We Hope

Even while recognizing the pressures of demonstrating the value and impact of OAA services, mechanisms are taking hold which may change the methods and resources for collecting data and evaluating OAA programs. The efforts to increase funding for evaluation activities is one example of possible promising change. Another exciting opportunity is also being established.

Section 201(g)1 of the 2020 OAA Reauthorization establishes a Research, Demonstration, and Evaluation Center for the Aging Network (OAA, 2020). For the network and researchers alike, the purposes and promise of the Center should address some of the challenges of demonstrating the value of OAA services and also open new opportunities to support the network and OAA participants. The purpose of the Center is three-fold:

• to coordinate research, evaluation, demonstration projects, dissemination, and related activities carried out under the OAA;
• to provide assessment of OAA-authorized programs and interventions;
• and to increase the available information on evidence-based programs and interventions available to the aging network in an effort to support existing programs and to further develop new evidence-based programs and interventions (OAA, 2020).

The Center is expected to carry out specific activities in support of the aging network and OAA service. Many of these activities build upon current evaluations and performance measurement activities of ACL’s Office of Performance and Evaluation. For example, the Center is charged with “research on and assessment of the relationship between programs and interventions under this Act and the health outcomes, social determinants of health, quality of life, and independence of individuals served under this Act” (OAA, 2020, p. 17).

The explicit requirement to assess important outcomes of OAA services advances the need for ACL to partner with other federal agencies for data linkages (e.g., the Department of Housing and Urban Development, Centers for Medicare and Medicaid Services [CMS], Centers for Disease Control and Prevention). Connecting OAA services with outcomes such as quality of life and independence also reinforces the importance of person-centered care, which by definition is reliant upon what matters most to the person receiving services.

The Center also will conduct “demonstration projects that support the objectives of this Act, including activities to bring effective demonstration projects to scale with a prioritization of projects that address the needs of underserved populations, and promote partnerships among aging services, community-based organizations, and Medicare and Medicaid providers, plans, and health (including public health) systems” (OAA, 2020, p. 17).

Implementation and evaluation of demonstration projects to test and evaluate new models of OAA services allows the network to create new approaches and scale up what works best in the community. Conducting demonstration projects will require ACL to allocate resources to establish an infrastructure that will ensure robust implementation of service models and rigorous evaluation. The network will be best served if a solid infrastructure is in place, because this will maximize the possibility of success. The Center for Medicare and Medicaid Innovation (CMMI) within CMS is one model ACL may mimic as it aims to test effective and innovative approaches to OAA services (Aging and Disability Business Institute, 2021).

The last two activities of the Center involve disseminating research findings and providing technical assistance (OAA, 2020). These two activities will provide the network with the resources and assistance needed to assure that quality data is consistently available for reporting on the value of OAA services. In addition, a targeted activity for outreach and dissemination of research results will increase the use of evidence for decision making by policymakers, program staff, and providers (Evidence-Based Policymaking Collaborative [EBPC], 2016).

Despite the ongoing pressures to “show me the data,” there is hope that the aging network and ACL are in a prime position to take advantage of an increased interest and better resources to reassess how the impact and value of OAA services are demonstrated. Balancing the need for sufficient data with the burden on clients and providers will be an ongoing challenge; however, it is an exciting time to be a friend of ACL data.

With a federal emphasis on evidence-based decision making (EBPC, 2016) and the establishment of a Research, Demonstration, and Evaluation Center for the Aging Network (OAA, 2020), ACL and the aging network can work together to collect the evidence that demonstrates how community- and home-based services benefit older adults, caregivers, and the larger health and social services system.

Author Note: Much of the insight for this article is based upon the author’s previous role as a social science analyst at ACL, overseeing evaluation and data collection of OAA programs, and her current work at RTI International where she continues to support ACL’s research and evaluation of OAA programs. Points of view or opinions expressed here do not necessarily represent official ACL policy.

Heather L. Menne, PhD, is a senior health policy researcher at RTI International.

References

Administration for Community Living. (2019). Aging in Place: DO Older Americans Act Title III Services Reach Those Most Likely to Enter Nursing Homes? Retrieved from https://agid.acl.gov/Resources/DataOutputs/OAA_Brief_1_12-4-19_508.pdf.

ACL. (n.d.) Fiscal Year 2022, Justification of Estimates for Appropriations Committees. Washington, DC: U.S. Department of Health and Human Services. Retrieved from https://acl.gov/sites/default/files/about-acl/2021-06/FY%202022%20ACL%20Justification%20of%20Estimates%20for%20Appropriations%20Committees_Final.pdf.

Aging and Disability Business Institute. 2021. Policy Spotlight: Aging Network Opportunities in the New Older Americans Act Research, Demonstration and Evaluation Center. Retrieved from https://www.aginganddisabilitybusinessinstitute.org/wp-content/uploads/2021/05/Policy-Spotlight-OAA-Research-FINAL-508.pdf.

AGing, Independence, and Disability. n.d. Data Files, National Participants Survey (NPS). Retrieved from https://agid.acl.gov/datafiles/NPS/Files.aspx?year=2019&serviceid=8.

Dahlhamer, J.M., Galinsky, A.M., Joestl, S.S., & Ward, B.W. (2014). “Sexual Orientation in the 2013 National Health Interview Survey: A Quality Assessment.” Vital Health Statistics, 2(169). Retrieved from https://www.cdc.gov/nchs/data/series/sr_02/sr02_169.pdf.

Dragon, C.N., Laffan, A.M., Erdem, E., Cahill, S.R., Kenefick, D., & Ye, J., et al. (2017). “Health Indicators for Older Sexual Minorities: National Health Interview Survey.” 2013–2014. LGBT Health, 4(6), 398–403. http://doi.org/10.1089/lgbt.2016.0203

Evidence-Based Policymaking Collaborative. (2016). “Principles of Evidence-Based Policymaking.” https://www.urban.org/sites/default/files/publication/99739/principles_of_evidence-based_policymaking.pdf. Retrieved June 28, 2021.

Interagency Working Group on Sexual Orientation and Gender Identity. (2016). Current Measures of Sexual Orientation and Gender Identity in Federal Surveys. Washington, DC: Federal Committee on Statistical Methodology. Retrieved from https://fcsm.sites.usa.gov/files/2014/04/WorkingGroupPaper1_CurrentMeasures_08‐16.pdf.

Morgan, R.E., Dragon, C., Daus, G., Holzberg, J., Kaplan, R., Menne, H., et al. (2020). Updates on Terminology of Sexual Orientation and Gender Identity Survey Measures. FCSM 20–03. Washington, DC: Federal Committee on Statistical Methodology. Retrieved from https://nces.ed.gov/FCSM/pdf/FCSM_SOGI_Terminology_FY20_Report_FINAL.pdf.

National Academies of Sciences, Engineering, and Medicine. n.d. Measuring Sex, Gender Identity, and Sexual Orientation for the National Institute of Health. Retrieved from https://www.nationalacademies.org/our-work/measuring-sex-gender-identity-and-sexual-orientation-for-the-national-institutes-of-health.

National Survey of Older Americans Act Participants. n.d. 13th Annual Survey Documentation. Retrieved from https://agid.acl.gov/DataFiles/Documents/NPS/2018SurveyDocumentation.pdf.

Older Americans Act of 1965, 42 U.S.C. § 3001 et seq. (2020). Retrieved from https://acl.gov/sites/default/files/about-acl/2020-04/Older%20Americans%20Act%20Of%201965%20as%20amended%20by%20Public%20Law%20116-131%20on%203-25-2020.pdf.

SAGE. n.d. The facts on LGBT aging [infographic].Retrieved from https://www.sageusa.org/wp- content/uploads/2018/05/sageusa-the-facts-on-lgbt-aging.pdf.