Appendix B. Detailed Research Domain Framework Definitions and Classification Rules

This appendix provides the following supplementary information on the research domain framework and environmental scan and portfolio analysis results:

• Detailed definitions and classification rules used to operationalize the research domain framework for the scan analysis of federally funded HSR and PCR.
• Grant activity codes from the NIH RePORTER database related to research infrastructure that were excluded from the scan.
• Scan analysis results by funding agency, including confidence intervals.
• Scan analysis results by year.

Detailed Definitions of Research Domains and Categories

Chapter 2 presented the research domain framework that the study developed to identify and differentiate HSR and PCR projects, including brief descriptions of the research domains and categories that comprise the framework. Here we provide detailed definitions of these domains and categories, which include four domains of health care “outputs,” four domains of health care “inputs,” five categories of HSR and PCR-related MTD, and five HSR and PCR subcategories of interest.

Health Care Output Domains for HSR and PCR

The output domains are as follows:

• Quality of Care. This includes studies that assess care process as well as intermediate or definitive outcomes of care. Examples of care process include timely provision of preventive services, accurate diagnosis of health conditions, provision of recommended care, and patient safety-related practices to prevent unsafe procedures and medical errors. Examples of intermediate outcomes include clinical health measures (e.g., adequate control of blood pressure, blood sugar, and asthma symptoms), patient safety related events (e.g., surgical errors, healthcare-associated infections), and patient experience and satisfaction. Examples of definitive outcomes include patient functioning, health-related quality of life, and mortality.
• Cost and Utilization. Cost and Utilization research examines health care cost or resource utilization. This includes studies that examine health service use as an intermediate measure in order to better understand some aspect of health service cost or resource utilization. 
• Access to Care. Access to care research examines barriers and facilitators to patients receiving needed health services. This includes studies measuring availability of care (health facilities, clinicians, medicines, equipment, having access to insurance, being able to make appointments, see a provider in a timely manner, etc.), patient attitudes and behaviors related to accessing care, and the cost of access. It would include evaluations of efforts to improve patient access in additional to observational studies of access. 
• Equity. Studies that examine whether any health care outputs (e.g., Quality of Care, Cost and Utilization, Access to Care, Personal Preferences and Behaviors [when an output]) differ among subsets of the population as well as studies that are conducted in the context of disparities. Studies whose thrust is to improve the health of an already identified disparity population (such as prisoners) are included, even if comparison is not made to more-advantaged groups (e.g., nonprisoners). Studies examining disparities based on existing health problems are included only if those disparities are exacerbated by social disadvantage and are not directly a result of the health problem itself (e.g., a project that examines quality of care for hypertension among those with mental illness). 

Health Care Input Domains for HSR and PCR

The input domains are as follows:

• Organization of Care. This includes studies on the structures and routines of care and the process of how they change or improve, from the composition and functioning of care teams to the composition and dynamics of health care delivery systems and markets, including both social (e.g., workforce, clinician, and staff experience) and material assets (e.g., health technologies, physical facilities).
• Financing of Care. This includes studies of systems that determine how health care professionals and health care organizations are paid to deliver care.
• Social Factors. This includes studies on the social, economic, and community determinants of health and health care.
• Personal Preferences and Behaviors. This includes studies of health-related patient behaviors (e.g., adherence to therapy, exercise, or smoking) and patient knowledge, attitudes, and understanding of health and health care.

Research Methods and Tool Development for HSR and PCR

The MTD projects are considered more “developmental” than research focused. Even though these projects may not directly examine either an input or output of health services, they develop methods and tools that are designed to be used as part of HSR and PCR projects or are part of health service delivery (with the exception of out-of-scope projects—see table entry on “Non-HSR/PCR Clinical Trials, Preclinical Research, and Non-HSR/PCR Epidemiology” below). However, some projects can be both HSR and/or PCR as well as MTD, if they have dual aims of both carrying out research as well as developing a particular method or tool for use in future research or health services delivery.

• HIT Applications and Tools. Many subcategories exist. Examples include the design and testing of technologies such as telehealth platforms, mobile applications, electronic prescribing, computerized clinical decision support systems, and various EHR functionalities to improve the quality, safety, or efficiency care.
• Model Development and Validation. This category includes psychometric instruments such as screening tools and health-related quality of life measures, quality measures, case mix adjustment models, risk-prediction and risk-stratification models, and machine learning and natural-language processing models that relate to health service delivery or primary care. Many research studies involve creating measures, surveys, or models for use in their analysis—these would count as a model development and validation only if they are intended for use beyond the current study, whether in future research and evaluation (e.g., patient safety measures) or incorporation in health service delivery (e.g., a screening tool or a risk-assessment algorithm to inform clinician decisionmaking). 
• Toolkit Development. Toolkits document interventions or tools and provide strategies to facilitate the uptake and implementation of interventions, initiatives, or best practices. Toolkits contain at least some informational, training, or operational materials useful to health care delivery organizations to incorporate and sustain evidence-based or promising practices into health service delivery (Hempel et al., 2019). Creating, designing, adapting, or updating materials for use in a toolkit will be considered Toolkit Development. 
• Evidence Review and Synthesis. This category includes systematic reviews and narrative reviews regarding any health services or PCR topic. Some project abstracts mention carrying out an initial literature or evidence review as part of a scoping analysis to inform project research design; these would count as an evidence review and synthesis study only if the evidence review is a primary project output rather than solely being an initial design step in the project. All evidence review and synthesis projects will be counted as research projects. 
• Simulation Modeling. Studies that use simulation models (i.e., models that use simulated inputs) to examine cost-effectiveness, decisionmaking, and other aspects of health services delivery. This category does not include studies that employ mathematical models that solely use inputs derived from real-world observations, though it does include models that use both simulated and observed input data, such as many agent-based models, Markov Chain models, Monte Carlo models, or other models that use probabilities as inputs.

Research Subcategories of Interest for HSR and PCR

The research subcategories of interest for HSR and PCR are as follows:

• Aging. This includes research pertaining to the aging process as it manifests in older people. We do not have a firm age cutoff at which “older people” begins; 65 is often used, 50 is sometimes used, as are other numbers. The aging area of interest includes research about syndromes primarily impacting older persons (often called “geriatric syndromes”), such as urinary incontinence, falls, and sarcopenia, even if the research includes some persons who are not older. The aging area of interest also includes research about the health and health care of older persons as well as research about the practice patterns of geriatricians, or about hospital wards that focus on older patients, such as geriatric psychiatry wards. We will not include research about conditions that become more common with age—with prominent examples being cardiovascular disease, cancer, and end-of-life care—unless the topic of study is specifically how those conditions are managed in older patients.
• Definitive Health Outcomes. This category includes death/mortality, other definitive outcomes, and health-related quality of life and related concepts of self-assessment of health (e.g., self-reported pain level). 
  • Definitive outcomes represent permanent or at least long-term harm to the individual and include examples like death, heart attack, amputation, broken bones.
  • The category of health-related quality of life includes concepts such as health-related quality of life, ability to function independently, mobility, and functional status.
  • Intermediate outcomes are not included in this concept of “definitive health outcomes.” Intermediate outcomes include measures related to the incidence of chronic conditions—such as high blood pressure, diabetes, or presence of cancer—even when these intermediate outcomes are known to be strong risk factors or even direct causes of definitive outcomes such as pain, loss of function, or death.
• Patient Safety. This category focuses on research that examines the causes, predictors, and prevalence of accidental or preventable injuries produced by health care as well as research that evaluates interventions meant to reduce harm produced by health care. Important topics include understanding the causes and prevention of medical errors, developing and testing systems to learn from errors and near-errors, and creating a culture that supports these activities (a “safety culture”). An example of a safety culture would occur when a health care professional feels empowered to speak up when he/she notices a risk of harm. This category also includes HSR or PCR that focuses on treatment of injury or disease produced by health care (e.g., studies on how to best treat Methicillin-resistant Staphylococcus aureus infection in the intensive care unit).
• Pediatrics. This category includes research related to the physical, mental, and social health of children from birth to young adulthood (age 18 or younger for the purposes of this project) and encompassing a broad spectrum of health services ranging from preventive health care to the diagnosis and treatment of acute and chronic diseases. Research about pediatricians, including primary care and specialty pediatrics, as well as children’s hospitals, is included. Research on family practice is only included if it pertains to how they manage the care of children. Research on maternal-fetal health is only included if it at least partly focuses on the child’s outcomes—not just those of the mother.
• Prevention. For our purposes, research on prevention includes evaluation of activities intended to prevent rather than treat disease. This includes cancer screening, management of asymptomatic risk factors including hypertension, diabetes, and hyperlipidemia; and general efforts to promote a healthy lifestyle including adequate sleep, adequate exercise, healthy diet, and refraining from unhealthy substance use. Prevention also includes prevention of injury, such as a health professional providing counseling about seat belt use, gun safety, or sunscreen use. Prevention also includes patient safety studies that examine the factors underlying accidental or preventable injuries or diseases produced by health services (e.g., hospital-acquired infections, diagnostic errors, addiction to drugs prescribed by a health provider).

We also examined one additional HSR and PCR area of interest, multimorbidity, but do not report results for this category due to methodological challenges. The definition we used for this concept was:

• Multimorbidity. This category includes research that explicitly addresses how best to care for patients who have more than one chronic medical condition, or describes the care delivered to such patients. This concept is synonymous with “multiple chronic conditions.”

We ran into several challenges in categorizing multimorbidity studies, including the difficulty in separating which chronic conditions are distinct from each other as well as the difficulty in distinguishing between studies that actually examined comorbid conditions versus those that accounted for co-occurring conditions solely as a control or confounding variable. As a result, the levels of agreement (kappa) between multiple reviewers, as well as between reviewers and our machine learning algorithm, were consistently poor for this category (Appendix A).

Additional Classification Rules for Manual Reviews

Coding Logic Workflow

1. Determine if the project is research (i.e., investigates a research question using scientific methods):
   • If it is research
     − go to Step B
   • If it is not research due to being a conference grant, professional development grant, or a center grant without research
     − END REVIEW (leave all other domains blank)
   • If it is not research due to some other reason (e.g., grant for health service provision, or for MTD, without any research component)
     − go to step C only. After step C, end review
2. Determine if the project is clearly in one of the following out-of-scope research categories: non-HSR/PCR clinical trial, preclinical research, and non-HSR/PCR epidemiology (column K)
   • If it is clearly in one of these categories, END REVIEW
3. Determine if the project is in any MTD or methods of interest categories.
4. Determine if the project is PCR, covers one or more main research domains, or covers one or more of the other areas of interest.
5. Determine if the project is HSR (see below). 

Adjudicating HSR

Projects that are determined to be research, but that are not excluded as clearly out-of-scope, are reviewed to see if they meet the criteria for each of the HSR and PCR domains. They are then reviewed to see if they meet the following criteria for HSR overall:

1. Any project that measures Organization of Care.
2. Any project that includes one or more HSR domains as a study input and one or more HSR domains as a study output:
   1. The Access to Care output domain is not used to determine whether or not a project is HSR, since this domain was developed later than the other domains and implementation of the HSR adjudication criteria, and was generally coded to projects together with at least one other output domain, such as Quality of Care or Equity, that would ensure the projects’ inclusion in the scan as HSR.
   2. While the research domain framework designates four domains as HSR inputs and four domains as HSR outputs, in some studies the following domains may be treated as either inputs, outputs, or both, depending on the study design and analysis.
      1. Quality of Care.
      2. Cost and Utilization.
      3. Organization of Care. 
      4. Personal Preferences and Behaviors.

Procedural Rules

Project descriptions do not always provide sufficient information to determine whether a study clearly covers a particular domain or not. In these cases, the reviewer should make reasonable assumptions, based on the information that is provided in the abstract, to determine whether the study probably does or does not cover a particular domain. For example, a study that otherwise does not touch on personal preferences and behavior, and simply mentions measuring “individual-level factors,” would not count as a Personal Preferences and Behaviors study. But another study, that simply mentions measuring “individual-level HIV risk factors,” would be counted as a Personal Preferences and Behaviors study, since examining HIV risk factors probably includes examining personal behaviors.

Some of our domain definitions make distinctions between studies that examine health service delivery and those that do not. Health service delivery, in our use of the term, is delivery of services primarily intended to improve health or the social factors affecting health by any of a wide range of health providers in a wide range of settings. Some services, when they are primarily focused on health, can count as health services even when delivered in less traditional health settings or by nonprofessional providers. This includes services such as substance abuse counseling delivered by peers in community health settings such as a church, schoolteachers referring students to their school’s mental health counselor, or health services delivered by family caregivers at home. 

While federal health agencies sometimes fund studies of other activities that affect human health—such as the government imposing a new regulation on car safety, or the opening of a supermarket offering healthy produce, or municipal water quality monitoring—we do not treat these types of activities as health services according to our definition.

Family caregivers are a special case in that they are generally considered as both part of the system of care and as part of the patient-family unit that is targeted by care. So, a study that examines the impact of a health care intervention on caregiver burden, stress, and behaviors would count as both Systems of Care as well as Quality of Care and Personal Preferences and Behaviors.

Some projects include variables for purposes of controlling for effects or matching cases (e.g., patient age and race/ethnicity to statistically control for patient demographics or to derive propensity score matching). Although these variables may be related to concepts in a research domain (e.g., Aging for age, or Equity and Social Factors for race/ethnicity), the report would not be considered to address these research domains unless the report substantively addresses the variables, i.e., examines differences in determinants, effects or outcomes of the variables (e.g., used as independent, dependent, co-variate, or mediating variables).

Regarding Table B.1, the main rules for the HSR domains (the ones we referred to most often) are found above in Section 2. The rules in this table are in addition to those rules, and represent rules for more unusual cases, with a goal of recording the full range of all coding decisions made by the study team for the environmental scan and portfolio analysis.

Table B.1. Additional Rules and Examples for Specific Domains and Categories

Return to Contents