The physical therapy efficacy index and chart: a stimulus for value-based healthcare using real-world data | BMC Health Services Research

This study was structured as an explanatory sequential mixed methods design.

Data collection

The data utilized for investigating the feasibility of a PE-_Index (Physical therapy Efficacy Index) and PE-_Chart (Patient Efficacy Chart) were extracted from the Dutch National Data Registry (LDK) of the Association for Quality in Physical Therapy (SKF), which contains real-world data provided via electronic health records. These electronic health records included patient and episode characteristics, such as the course, duration, and number of treatments, as well as the results of tests to measure and monitor the severity of the problems.

To construct the PE-_Index and PE-_Chart, Two Patient Reported Outcome Measures (PROMs) were employed, namely the Numeric Pain Rating Scale (NPRS) and Patient-Specific Functioning Scale (PSFS). The NPRS measures pain intensity on a scale from 0 (no pain) to 10 (worst imaginable pain) [5], while the PSFS assesses the ability to complete an activity at a level experienced prior to injury or change in functional status using an 11-point scale [6]. These PROMs have good measurement properties, are frequently used in clinical practice and are recommended in the Dutch guidelines for physical therapy for musculoskeletal disorders [7].

The data were collected by physical therapists during standard care through their electronic health records and classified using a four-digit coding system based on body location and pathology [8] (table A1 in the appendix). Physical therapists were free to select their own treatment methods, as the data were derived from routinely collected records. In the Netherlands, physical therapy for these complaints generally includes exercises, mobilization or manipulation, and advice, sometimes supplemented by passive modalities such as dry needling or massage.

Under Dutch law, the use of electronic health records for research purposes is permitted under certain conditions. When these conditions are met, neither obtaining informed consent from patients nor approval by a medical ethics committee is required for observational studies containing no directly identifiable data (art. 24 GDPR Implementation Act jo art. 9.2 sub j GDPR; Dutch Civil Law, Article 7:458).

Development of the physical therapy efficacy index and chart

The PE-_Index and PE-_Chart were developed and tested through a four-step process that included (1) conceptualization, (2) development of the tool, (3) discriminative value between practices, and (4) calculation of individual practice scores and charts.

Conceptualization

First, a general conceptualization of the value-based health care principle of Porter & Teisberg (2006) was formulated as an efficacy index. In this index, the health outcomes are divided by the cost of the episode. In this division, health outcomes were defined as meaningful patient-reported outcome measures (Eq. 1).

Development

Second, a physical therapy-specific efficacy index (PE-_Index) was formulated for patient episodes of a musculoskeletal nature. Musculoskeletal disorders comprise a diverse range of conditions (appendix 1). The outcome measures were defined as the difference in PROMs (NPRS and PSFS) between baseline and at the end of every individual episode. The episode costs were considered to consist of two parts, namely, the financial costs (episode price) and the costs expressed in the burden for the patient (duration of the treatment episode). The treatment episode price was defined as the number of treatments (N treatments) instead of the actual treatment cost since the price of all treatments was approximately equal. The treatment episode duration was defined as the total duration of the trajectory per two weeks (intake-end).

The constants C1, C2, C3, and C4 are added to ensure that the PE-_Index ranges between 0 and 1, and the outcome is multiplied by 100 for easier interpretation (Eq. 2). The PE-_Index formula is as follows:

where:

• ▪ ∆ NPRS: the change in the Numeric Pain Rating Scale (NPRS) score from baseline to the end of the episode.

• ▪ ∆ PSFS: the change in Patient-Specific Functioning Scale (PSFS) score from baseline to the end of the episode.

• ▪ N treatments: the number of treatments received during the episode.

• ▪ Episode duration: the total duration of the treatment episode, per two weeks, rounded up.

• ▪ C1: the summation of the maximal outcomes of ∆ pain reduction and ∆ function improvement (set to + 20 to ensure a positive numerator).

• ▪ C2: the summation of C1 and the maximum outcome of ∆ pain reduction and ∆ function improvement (set to 40 to ensure a PE-_Index score between 0 and 1).

• ▪ C3: set to 3, based on the minimal value of N treatments.

• ▪ C4: set to 1 based on the minimal value of episode duration.

Discriminative value

In the next step, the discriminative value of the PE-_Index between physical therapy practices was evaluated using linear mixed models. This value was compared to the discriminative value of the four components of the PE-_Index, namely, the delta NPRS, delta PSFS, number of treatments and episode duration. Additionally, the effect of adding explanatory variables was assessed. In total, ten linear mixed models were created, consisting of five intercept-only models and five adjusted models. The physical therapy practices were added as a random effect.

In the adjusted models, the following variables were included as main effects: age, sex, socioeconomic status, chronicity, classification code, number of treatments, and episode duration (in weeks). The patient’s socioeconomic status was calculated using their postal code [9]. Chronicity was expressed as the duration of the complaint before the patient sought help. The classification code is an administrative coding of the location of the complaint, e.g., low back pain, shoulder pain, etc. The continuous characteristics (age, baseline NPRS and PSFS) were converted to z scores. The characteristics were tested for collinearity by calculating Pearson’s correlation coefficient. If the correlation coefficient exceeded 0.8, only one of the characteristics was included in the analysis.

Episodes were included if (1) the disorder was classified as musculoskeletal, (2) the trajectory started after January 2018 and ended between January 2020 and January 2021, or (3) the episode consisted of at least three treatments. Episodes were excluded when sex, age, postal code or the NPRS or PSFS measurements at the start or at the end of the episode were missing. Moreover, episodes were excluded in the case of a new treatment for a recurrent condition in the same practice. Finally, outliers in the number of treatments or episode duration were identified and excluded by converting the outcomes of these variables into z scores, where a score greater than three or less than three was marked as an outlier [10].

The outcome of interest of the linear mixed models was the amount of variance caused by the random effect and thus the variance in outcome caused by the practices. This amount was estimated with the intraclass correlation coefficient (ICC), which can be calculated by dividing the variance of the random effect by the total variance. The total variance is the sum of the variance between practices and within practices. An ICC > 0.1 can be interpreted as an adequate measure to discriminate between practices.

The data were processed using Python (version 3.7.4), and linear mixed models were created with the python module Statsmodels (v0.13.2).

Individual PE-_Index and PE-_Chart

In the final step we calculated the corrected PE-_Index and its four components, namely, the delta NPRS, delta PSFS, number of treatments and episode duration, for all episodes using adjusted linear mixed models and aggregated them per practice. Next, these outcomes were converted to z scores and visualized, thus creating a radar plot with 5 factors, known as the PE-_Chart. This method enabled comparison of all corrected outcomes of an individual practice to the average of all practices or to a specific practice.

Evaluation of the PE-_Index and PE-_Chart with stakeholders

In phase three, we evaluated the PE-_Index and associated chart’s appropriateness for ‘internal’ quality improvement in SKF’s learning health system, which includes peer learning sessions among physical therapists, management reports for practice executives, and practice visits, and ‘external’ transparency, such as reimbursement choices and patient decision aids, and as a comparison tool among care disciplines. We gathered data through stakeholder surveys and advisory board meetings, with input from physical therapists, patients, insurance companies, and statisticians.

Survey

The appropriateness for internal quality improvement was evaluated for the PE-_Chart, and the appropriateness for external transparency was evaluated for both the PE-_Index and PE-_Chart using a survey specifically developed for this study. Participants were asked to rate the index and chart with the following question: “To what degree do you think the PE-_Index and PE-_Chart are appropriate for this specific goal?”, using a 9-point Likert scale, in which 1 was “Absolutely not appropriate” and 9 was “Absolutely appropriate”. In this study, we categorized the questions into three distinct groups. The first category addressed the utilization of the PE-_Index for internal quality purposes, specifically focusing on peer learning sessions without any form of judgment. The second category pertained to external transparency and consisted of two subcategories: one for the PE-_Index and the other for the PE-_Chart. Within these subcategories, we explored the suitability of the tools for reimbursement choices, their effectiveness as decision aids for patients, and their potential to facilitate comparisons between different health care professions (Table 1).

Insurance companies currently employ a “treatment index” as a guiding metric for their reimbursement policies. This index is based on the mean number of treatments stratified according to subgroups within the patient population [11]. In the latter category of our study, we inquired explicitly whether the respondents perceived an added value of the PE-_Index in comparison to the existing treatment index.

With a median score ≥ 7 on the Likert scale, the instrument was considered appropriate, with a median score ≤ 3 as not appropriate. We arbitrarily chose an interquartile range ≤ 3 to assess whether a consensus was reached between respondents. To correct for possible ceiling effects, we also calculated a disagreement index in accordance with the RAND/UCLA manual for consensus studies [12]. An index > 1 indicates a disagreement. The survey was conducted online via Google Forms. The outcomes of the individual participants were anonymized.

Physical therapists

We aimed to include 21 physical therapists who had experience treating patients with musculoskeletal disorders. Each physical therapist provided data to the LDK and received an aggregated feedback report, including the PE-_Index and results chart, prior to the survey. Fifteen physical therapists were randomly selected from the LDK database, and 6 were purposefully selected for their experience with routinely collected data for quality improvement initiatives. The physical therapists had no personal relationship with the researchers. The answers given by the physical therapists during the interviews and in the survey were processed anonymously.

Knowledge brokers

We invited 18 knowledge brokers who were independent employees with experience in quality evaluation in primary care physical therapy. These knowledge brokers typically visit practices for physical therapy every two years to provide guidance on quality regulations. After receiving an anonymized report with the PE-_Index and PE-_Chart based on real-world data, the knowledge brokers participated in the survey. As was the case with the physical therapists, no personal relationship was present with the researchers, and all answers were processed anonymously.

Vektis statisticians

Vektis is an organization that collects, analyses, and manages healthcare data on behalf of health insurance companies. Three statisticians of the organization were consulted once about the methodological aspects of the PE-_Index and PEC and were invited to share their experiences with the existing treatment index. Due to the low number of statisticians, they did not participate in the survey.

Advisory board

We created an advisory board with one representative of the Netherlands Patient Association (PN), two representatives of the Royal Dutch Society for Physical Therapy (KNGF), one representative of the Dutch Society for Exercise Therapy (VvOCM) and two representatives of Dutch Health Insurers (ZN). After receiving an anonymized report with the result chart and PE-_Index based on real-world data, the members of the advisory board participated in the survey and four board meetings. The board provided advice on the study design and practical aspects throughout the ongoing process. During the final meeting, the study’s results were presented and thoroughly examined.”