Development of models for predicting the objective response of immune checkpoint inhibitor therapy in patients with nonsmall-cell lung cancer

Qiao Yang; Lin Cui; Yu Gao; Xiu Yang; Jianguo Sun

doi:10.4103/digm.digm_20_22

ORIGINAL ARTICLE

Year : 2022 | Volume : 8 | Issue : 1 | Page : 27

Development of models for predicting the objective response of immune checkpoint inhibitor therapy in patients with nonsmall-cell lung cancer

Qiao Yang¹, Lin Cui², Yu Gao², Xiu Yang³, Jianguo Sun³
¹ Department of Ultrasound, The 941^st Hospital of the PLA Joint Logistic Support Force, Xining, Qinghai, China
² Department of Emergency, The 941^st Hospital of the PLA Joint Logistic Support Force, Xining, Qinghai, China
³ Cancer Institute, Xinqiao Hospital, Army Medical University, Chongqing, China

Date of Submission	13-Apr-2022
Date of Decision	29-Jun-2022
Date of Acceptance	04-Jul-2022
Date of Web Publication	09-Nov-2022

Correspondence Address:
Jianguo Sun
Department of Oncology, Xinqiao Hospital, Army Medical University, Chongqing 400037
China

Source of Support: None, Conflict of Interest: None

Check

DOI: 10.4103/digm.digm_20_22

Abstract

Background: Immune checkpoint inhibitors (ICIs) have significantly improved the therapeutic efficacy and extended the survival of patients with advanced nonsmall-cell lung cancer (NSCLC); however, the benefits were confined to certain populations. How to optimally select eligible patients for ICI therapy needs to be solved. Materials and Methods: A total of 100 NSCLC patients treated with ICIs from two independent studies were included and then were randomly assigned to the train dataset and the validation dataset. A logistic regression model and a decision tree model were developed to predict the objective response outcome. The area under the receiver operating characteristic curves (AUC) of the two models was evaluated. The confusion matrix for decision tree model was also calculated. Results: The comparison of baseline clinical features between the train and validation datasets showed no significant difference. The multivariable logistic regression identified three variables associated with objective response, i.e, age, tumor mutational burden (TMB), and programmed cell death ligand 1. Moreover, a logistic regression model was developed based on these variables. A decision tree, including age and TMB, was also developed. The AUCs of the two models were both 0.85 in the train dataset. And, that was better than each single variable. The AUC of the decision tree in the validation dataset was inferior to that of the logistic regression model. The performance showed that the decision tree could not effectively identify responders. Conclusion: The performance of the decision tree developed in this study was dismal. Its predictive ability was inferior to the logistic regression model.

Keywords: Decision tree, Immune checkpoint inhibitors, Logistic regression, Nonsmall cell lung cancer, Programmed cell death ligand 1

How to cite this article:
Yang Q, Cui L, Gao Y, Yang X, Sun J. Development of models for predicting the objective response of immune checkpoint inhibitor therapy in patients with nonsmall-cell lung cancer. Digit Med 2022;8:27

How to cite this URL:
Yang Q, Cui L, Gao Y, Yang X, Sun J. Development of models for predicting the objective response of immune checkpoint inhibitor therapy in patients with nonsmall-cell lung cancer. Digit Med [serial online] 2022 [cited 2023 Jun 9];8:27. Available from: http://www.digitmedicine.com/text.asp?2022/8/1/27/360638

Introduction

Immune checkpoint inhibitor (ICI) therapy has changed the treatment strategy of some malignant solid tumors, especially advanced nonsmall-cell lung cancer (NSCLC). At first, some ICIs, such as pembrolizumab, nivolumab, and atezolizumab, were approved by the U.S. Food and Drug Administration (FDA) for the second-line treatment of advanced NSCLC.^[1],[2] Recently, ICIs were also approved for first-line treatment in advanced NSCLC either in combination with chemotherapy^[3] or as monotherapy.^[4] Although advanced NSCLC patients treated with ICIs had improved efficacy and extended survival compared to those treated with chemotherapy alone, the benefits remain limited. Only approximately 20% of patients receiving second-line ICI therapy obtained objective response,^[5] and the 5-year overall survival (OS) was approximately 23% in first-line ICI therapy patients and approximately 15% in previously treated patients.^[6] Therefore, the identification of eligible populations that will respond to ICIs is an urgent issue that needs to be addressed.

According to the result of KEYNOTE 024,^[7] pembrolizumab was approved for the first-line treatment of metastatic NSCLC patients with strong programmed cell death ligand 1 (PD-L1) expression (PD-L1 ≥50%). The results of KEYNOTE 001^[8] showed that NSCLC patients with PD-L1 ≥50% obtained better clinical benefits than those with PD-L1 expression ranging from 1% to 49% or <1% when treated with pembrolizumab. However, the CheckMate 026 study^[9] showed no survival difference between the nivolumab group and chemotherapy group in NSCLC patients with PD-L1 ≥5%, and further analysis of the PD-L1 ≥50% population still showed no difference. In the CheckMate 017^[10] and OAK^[1] studies, all NSCLC patients treated with ICIs obtained clinical benefit compared to those treated with chemotherapy, irrespective of PD-L1 expression. These data demonstrated that PD-L1 was not the optimal biomarker for selecting eligible patients. In addition, microsatellite instability-high (MSI-H)/mismatch repair deficient (dMMR) has also been approved by FDA. The objective response rate (ORR) of patients with MSI-H/dMMR solid tumors could be as high as 39.6%.^[11] This biomarker is the first pan-cancer approved biomarker for ICI therapy. However, the incidence of MSI-H/dMMR was low in most advanced solid tumors, especially in advanced NSCLC. Recently, tumor mutational burden (TMB) has emerged as a potential biomarker for screening patients who could benefit from ICIs. In a pan-cancer analysis, patients with the top 20% TMB in each histology obtained better OS rates from ICI therapy.^[12] In advanced NSCLC patients treated with combined ICIs, those with TMB-high obtained better efficacy and OS rates than those with TMB-low. In certain situations, TMB may serve as a biomarker combination with PD-L1.^[13],[14]

The use of machine learning methods to predict outcome in the medicine has drawn much attention.^[15],[16] Machine learning model could help address complex problems.^[17] Machine learning had emerged as an alternative method for illness diagnosis and prognostic prediction, especially in big data processing, medical image recognition, treatment support, biomedical research, etc.^[18–21]

With these considerations, in this study, we aimed to develop models to predict response in NSCLC patients treated with ICIs, using logistic regression model and decision tree model, and compare the predictive power of the two models.

Materials and Methods

Data sources

All data were downloaded from cBioPortal (www.cbioportal.org). Cohort 1 included 75 advanced NSCLC patients treated with nivolumab plus ipilimumab.^[13] Cohort 2 included 34 advanced NSCLC patients treated with pembrolizumab.^[22] The inclusion criteria included the following: first, advanced NSCLC patients; second, treated with ICIs; third, with full clinical information and PD-L1, TMB, tumor neoantigen burden (TNB) expression. Hence, nine patients were excluded because of a lack of PD-L1 expression or clinical information. Finally, a total of 100 patients were included for further analysis. The TMB, TNB, PD-L1 and clinical data of all patients were collected.

Clinical outcome metrics and variables

Objective response (complete response or partial response) was the main outcome in this study. The variables used included age, sex, smoking status, histological type, PD-L1 expression, TMB, and TNB. PD-L1 expression was divided into three groups: strong (PD-L1 ≥50%), weak (PD-L1 1%–49%), and negative (PD-L1 <1%). Age, TMB, and TNB were continuous variables, while the others were binary variables.

Development and validation of predictive models

All patients were randomly assigned to the train dataset and the validation dataset, with a spilt ratio of 6:4. The train dataset was used to develop the predictive models, and the validation dataset was used to validate the predictive power. To construct the logistic regression model, first, multivariable logistic regression was performed to screen out significant variables related to objective response. Then, significant variables with P < 0.05 were used to construct the logistic regression model. The decision tree model was built by a two-stage process. First, to find significant variable that could best split the outcome into two groups. A certain but complex tree model was built in this step. Then, cross-validation with the 1-SE rule was used to trim back the full tree. Last, to prune the tree in order to obtain a simple and meaningful decision tree model. The performance of the two models was evaluated by the area under the receiver operating characteristic curve (AUC). For decision tree model, accuracy and a confusion matrix were also calculated to describe how many results were correctly and incorrectly classified. To validate the two models, all indexes mentioned above were also evaluated in the validation dataset.

Statistical analysis

Continuous variables were described as median and interquartile range (IQR) and compared by the Mann–Whitney U-test. Categorical variables were represented as frequencies and compared by Pearson's Chi-square test. All statistical analyses were conducted with R software (version 4.0.3, GNU General). A two-sided P < 0.05 was considered statistically significant.

Results

The baseline clinical features and efficacy outcomes of this study are summarized in [Table 1]. Among the one hundred patients, 47 were male, 81 were current or former smokers, 18 had squamous carcinoma, and 82 had nonsquamous carcinoma. The median age was 63 years (IQR: 57–71, range: 41–87 years). The median TMB was 184 (IQR: 78.8–309.8, range: 7–1358), and the median TNB was 227 (IQR: 99–511.8, range: 0–4350). Thirty-one patients had negative PD-L1 expression, 49 had weak expression, and 20 had strong expression. Of all patients, 33% of patients obtained objective response. The comparison of baseline clinical features between the train and validation datasets showed no significant difference in [Table 1].

Table 1: Baseline clinical features comparison between train dataset and validation dataset

Click here to view

We used the multivariable logistic regression to screen out variables associated with objective response outcome using the train dataset. The results turned out that age, TMB, and PD-L1 expression were associated with objective response in [Supplementary Table 1]. Then, a logistic regression model based on these three variables was constructed.

In addition, a decision tree model was also developed to differentiate responders from nonresponders. As shown in [Supplementary Figure 1], two biomarkers were included in the decision tree model, including age and TMB. The thresholds of the two biomarkers could help classify each patient into responder group or nonresponder group.

The AUCs of the logistic regression model and decision tree model were both 0.85 in [Figure 1]a and [Figure 1]b. The AUC of age, TMB, and PD-L1 alone was 0.64, 0.70, and 0.66, respectively in [Figure 1]c, [Figure 1]d, [Figure 1]e. The performance of the decision tree model is summarized in [Table 2]. The accuracy of this model was 0.80. For responder prediction, the precision, recall, and F1 score were 0.91, 0.67, and 0.77, respectively. For nonresponder prediction, the precision, recall, and F1 score were 0.74, 0.93, and 0.82, respectively.

Figure 1: Receiver operating characteristic curves for two models in the train dataset. (a) Logistic regression model; (b) decision tree mode; (c) age; (d) TMB; (e) PD-L1. AUC: Area under the receiver operating characteristic curve, TMB: Tumor mutational burden, PD-L1: Programmed cell death ligand 1.

Click here to view

Table 2: Performance of the decision tree model

Click here to view

Furthermore, we applied the two models to the validation dataset. The AUC of the logistic regression model and decision tree model was 0.89 and 0.83, respectively in [Figure 2]a and [Figure 2]b. The performance of the decision tree model in the test dataset is summarized in [Table 2]. The accuracy was 0.73. For responder prediction, the precision, recall, and F1 score were 0.92, 0.50, and 0.65, respectively. For nonresponder prediction, the precision, recall, and F1 score were 0.66, 0.95, and 0.78, respectively.

Figure 2: Receiver operating characteristic curves for two models in the validation dataset. (a) Logistic regression model; (b) decision tree mode. AUC: Area under the receiver operating characteristic curve.

Click here to view

Discussion

In this study, we constructed two models to predict objective response in NSCLC patients treated with ICIs. Although the AUCs of the two models were the same in the train dataset, the AUC of logistic regression model in the validation dataset was better than that of decision tree model. The performance of the decision tree model showed that this model could not effectively identify responders (the recall value was 0.67 and 0.50 for train dataset and test dataset, respectively).

Although many biomarkers have been found to be associated with survival or efficacy in malignant patients treated with ICIs, the predictive precision of a single biomarker is limited. Strategies of combined biomarkers might be the key to this problem. The CheckMate 026 study^[9] demonstrated that in NSCLC patients treated with nivolumab, those with both high PD-L1 expression (≥50%) and high TMB had a much higher ORR (75%) than those with only high TMB (32%) or only high PD-L1 expression (34%). In the cohort 1 study,^[12] patients with both TMB-high or -medium and positive PD-L1 expression (≥1%) had an ORR of 62.5%. However, the ORR of only TMB-high, only positive PD-L1 expression, or low for both factors was 33.3%, 14.3%, and 7.7%, respectively. In this study, we found that both the logistic regression model (combination of three biomarkers, i.e, age, TMB and PD-L1) and the decision tree model (combination of age and TMB) had a much higher AUC than each biomarker, which suggested that the two models had better values in predicting objective response.

In this study, the decision tree model identified a threshold of age 73 years to classify patients. In the decision tree model, 10 out of 13 (76.9%) patients with age ≥73 years obtained objective response. While for 47 patients with age <73 years, only 12 of them (25.5%) obtained objective response. The role of age in ICI therapy is complicated in literature. In the KEYNOTE 024 study,^[7] comparing the pembrolizumab group to the chemotherapy group, the hazard ratio (HR) for disease progression was 0.45 (95% confidence interval [CI]: 0.29–0.70) in NSCLC patients aged ≥65 years, which was lower than that in patients aged <65 years (HR: 0.61, 95% CI: 0.40–0.92). In the OAK study,^[1] a lower HR for death was also found in patients aged ≥65 years than in those aged <65 years (0.66 vs. 0.80). However, in the CheckMate 057 study,^[23] only patients with ages ranging from 65 to 75 years obtained a survival benefit from nivolumab compared with chemotherapy, and patients aged ≥75 or <65 years did not. Moreover, a multicenter retrospective study demonstrated that no immune toxicity or OS differences were found among different age groups (≤64 years, 65–74 years, and ≥75 years).^[24] A preclinical experiment^[25] found that young mice had more regulatory T cells (Tregs) and fewer cytotoxic T cells than aged mice, which might contribute to worse outcome in ICI treatment. Champiat et al. suggested that elderly patients (>65 years) were prone to hyperprogressive disease, which was associated with worse survival.^[26]

By now, logistic regression is the most common traditional statistical algorithm to identify risk factors, while it has limitation in handling nonlinear data.^[15],[27] In contrast to that, machine learning has advantages in handling nonlinear data and processing big data.^[28] The decision tree model also has advantages of visualization, clinical practicality, and easy to interpret. Hence, the machine learning model is expected to be more effective than the logistic regression model in predicting the prognosis.^[15] In addition, using kinds of machine learning methods, such as support vector machine, random forest, could help get more comprehensive analysis results. While in this study, the performance of decision tree model was not satisfied. The AUC was inferior for decision tree model in the validation dataset. The probable reasons could be the following. First, the sample size of this study was small, which could not exploit the advantages of machine learning to the full. However, to the best of our knowledge, no more public data includes advanced NSCLC patients treated with ICIs and related clinical data, such as PD-L1, TMB, and TNB expression. Because of the limited data resource, the analysis in this study is of great scientific significance and provides a novel predictive model biomarker for NSCLC and immunotherapy. Second, the variables used in this study are limited. Some significant clinical features related to the efficacy of ICI therapy in previous literature were not included in this study, such as Eastern Cooperative Oncology Group performance status (PS score) and body mass index.^[29],[30] All these limitations should be addressed in further study to obtain a better machine learning model to predict the efficacy outcome and guide the clinical practice.

Conclusion

This study developed two models for predicting objective response in NSCLC patients treated with ICIs. Although the machine learning model has advantages of visualization, clinical practicality, and easy to interpret, the performance of this model was dismal. Its predictive ability was inferior to the logistic regression model.

Financial support and sponsorship

This study was supported by the National Natural Science Foundation of China (81773245, 81972858, 82172670), the Technology Innovation and Application Development Project of Chongqing (cstccxljrc201910), and the Cultivation Program for Clinical Research Talents of Army Medical University (2018XLC1010).

Conflicts of interest

Jianguo Sun is an Associate Editor of the journal. The article was subject to the journal's standard procedures, with peer review handled independently of this editor and his research groups.

References

1.	Rittmeyer A, Barlesi F, Waterkamp D, Park K, Ciardiello F, von Pawel J, et al. Atezolizumab versus docetaxel in patients with previously treated non-small-cell lung cancer (OAK): A phase 3, open-label, multicentre randomised controlled trial. Lancet 2017;389:255-65.
2.	Morgensztern D, Herbst RS. Nivolumab and pembrolizumab for non-small cell lung cancer. Clin Cancer Res 2016;22:3713-7.
3.	West H, McCleod M, Hussein M, Morabito A, Rittmeyer A, Conter HJ, et al. Atezolizumab in combination with carboplatin plus nab-paclitaxel chemotherapy compared with chemotherapy alone as first-line treatment for metastatic non-squamous non-small-cell lung cancer (IMpower130): A multicentre, randomised, open-label, phase 3 trial. Lancet Oncol 2019;20:924-37.
4.	Mok TSK, Wu YL, Kudaba I, Kowalski DM, Cho BC, Turna HZ, et al. Pembrolizumab versus chemotherapy for previously untreated, PD-L1-expressing, locally advanced or metastatic non-small-cell lung cancer (KEYNOTE-042): A randomised, open-label, controlled, phase 3 trial. Lancet 2019;393:1819-30.
5.	Chen DS, Mellman I. Elements of cancer immunity and the cancer-immune set point. Nature 2017;541:321-30.
6.	Garon EB, Hellmann MD, Rizvi NA, Carcereny E, Leighl NB, Ahn MJ, et al. Five-year overall survival for patients with advanced non-small-cell lung cancer treated with pembrolizumab: Results from the phase I KEYNOTE-001 study. J Clin Oncol 2019;37:2518-27.
7.	Reck M, Rodríguez-Abreu D, Robinson AG, Hui R, Csőszi T, Fülöp A, et al. Pembrolizumab versus chemotherapy for PD-L1-positive non-small-cell lung cancer. N Engl J Med 2016;375:1823-33.
8.	Garon EB, Rizvi NA, Hui R, Leighl N, Balmanoukian AS, Eder JP, et al. Pembrolizumab for the treatment of non-small-cell lung cancer. N Engl J Med 2015;372:2018-28.
9.	Carbone DP, Reck M, Paz-Ares L, Creelan B, Horn L, Steins M, et al. First-line nivolumab in stage IV or recurrent non-small-cell lung cancer. N Engl J Med 2017;376:2415-26.
10.	Brahmer J, Reckamp KL, Baas P, Crinò L, Eberhardt WE, Poddubskaya E, et al. Nivolumab versus docetaxel in advanced squamous-cell non-small-cell lung cancer. N Engl J Med 2015;373:123-35.
11.	Marcus L, Lemery SJ, Keegan P, Pazdur R. FDA approval summary: Pembrolizumab for the treatment of microsatellite instability-high solid tumors. Clin Cancer Res 2019;25:3753-8.
12.	Samstein RM, Lee CH, Shoushtari AN, Hellmann MD, Shen R, Janjigian YY, et al. Tumor mutational load predicts survival after immunotherapy across multiple cancer types. Nat Genet 2019;51:202-6.
13.	Hellmann MD, Nathanson T, Rizvi H, Creelan BC, Sanchez-Vega F, Ahuja A, et al. Genomic features of response to combination immunotherapy in patients with advanced non-small-cell lung cancer. Cancer Cell 2018;33:843-52.e4.
14.	Hellmann MD, Paz-Ares L, Bernabe Caro R, Zurawski B, Kim SW, Carcereny Costa E, et al. Nivolumab plus ipilimumab in advanced non-small-cell lung cancer. N Engl J Med 2019;381:2020-31.
15.	Cheng X, Li J, Xu T, Li K, Li J. Predicting survival of patients with rectal neuroendocrine tumors using machine learning: A SEER-based population study. Front Surg 2021;8:745220.
16.	Arslan E, Schulz J, Rai K. Machine learning in epigenomics: Insights into cancer biology and medicine. Biochim Biophys Acta Rev Cancer 2021;1876:188588.
17.	Deo RC. Machine learning in medicine. Circulation 2015;132:1920-30.
18.	Wei L, Huang Y, Chen Z, Li J, Huang G, Qin X, et al. A novel machine learning algorithm combined with multivariate analysis for the prognosis of renal collecting duct carcinoma. Front Oncol 2021;11:777735.
19.	Komura D, Ishikawa S. Machine learning approaches for pathologic diagnosis. Virchows Arch 2019;475:131-8.
20.	Handelman GS, Kok HK, Chandra RV, Razavi AH, Lee MJ, Asadi H. eDoctor: Machine learning and the future of medicine. J Intern Med 2018;284:603-19.
21.	Hajiloo M, Rabiee HR, Anooshahpour M. Fuzzy support vector machine: An efficient rule-based classification technique for microarrays. BMC Bioinformatics 2013;14 Suppl 13:S4.
22.	Rizvi NA, Hellmann MD, Snyder A, Kvistborg P, Makarov V, Havel JJ, et al. Cancer immunology. Mutational landscape determines sensitivity to PD-1 blockade in non-small cell lung cancer. Science 2015;348:124-8.
23.	Borghaei H, Paz-Ares L, Horn L, Spigel DR, Steins M, Ready NE, et al. Nivolumab versus docetaxel in advanced nonsquamous non-small-cell lung cancer. N Engl J Med 2015;373:1627-39.
24.	Ksienski D, Wai ES, Croteau NS, Freeman AT, Chan A, Fiorino L, et al. Association of age with differences in immune related adverse events and survival of patients with advanced nonsmall cell lung cancer receiving pembrolizumab or nivolumab. J Geriatr Oncol 2020;11:807-13.
25.	Kugel CH 3^rd, Douglass SM, Webster MR, Kaur A, Liu Q, Yin X, et al. Age correlates with response to anti-PD1, reflecting age-related differences in intratumoral effector and regulatory T-cell populations. Clin Cancer Res 2018;24:5347-56.
26.	Champiat S, Dercle L, Ammari S, Massard C, Hollebecque A, Postel-Vinay S, et al. Hyperprogressive disease is a new pattern of progression in cancer patients treated by anti-PD-1/PD-L1. Clin Cancer Res 2017;23:1920-8.
27.	Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD statement. Ann Intern Med 2015;162:55-63.
28.	Hamet P, Tremblay J. Artificial intelligence in medicine. Metabolism 2017;69S: S36-40.
29.	Yoneda T, Sone T, Koba H, Shibata K, Suzuki J, Tani M, et al. Long-term survival of patients with non-small cell lung cancer treated with immune checkpoint inhibitor monotherapy in real-world settings. Clin Lung Cancer 2022;23:467-76.
30.	Ichihara E, Harada D, Inoue K, Sato K, Hosokawa S, Kishino D, et al. The impact of body mass index on the efficacy of anti-PD-1/PD-L1 antibodies in patients with non-small cell lung cancer. Lung Cancer 2020;139:140-5.