Title: Predicting Cholangiocarcinoma in Primary Sclerosing Cholangitis: Using Artificial
Intelligence, Clinical and Laboratory Data
Version: 2 Date: 15 Dec 2022
Reviewer's report:
General comments:
1 This is a large PSC cohort from a single medical center system with 1,459 PSC patients, in
which 125 (8.6%) patients developed cholangiocarcinoma (CCA) and 1,035 (70.9%) had been
diagnosed with IBD.
2 The study adopted three prediction models: (1) statistics-based model (Cox proportional
hazards model, CoxPH) and (2) two machine learning models (decision trees methods: random
survival forest, RSF, and gradient boosting survival analysis, GBSA) to predict PSC patients
who would develop cholangiocarcinoma in shorter times (ie, shorter CCA-free survival among
PSC patients).
3 The performance of these three prediction models were evaluated by concordance index (C-
index), using 20-fold Monte Carlo cross-validation with 80%-20% train-test split.
4 The main finding in Table 4 showed that (1) the CoxPH, RSF, GBSA prediction model
performed better than current state-of-the-art prediction models: Mayo PSC risk score, MELD
score, and PREsTo score by C-index.
5 These CoxPH, RSF, GBSA models also showed good prediction C-index at 2yrs post diagnosis
of PSC and 5 yrs post diagnosis of PSC, which was not addressed by other state-of-the-art
prediction models.
6 The authors’ attempt to include bile acid profiles into prediction models demonstrated the lack
of improved prediction performance.
7 The authors are to be applauded by the adoption of modern technologies of machine learning
methods to provide better prediction models.
8 However, there are some major issues regarding the decision trees models:
8.1 Overfitting: overfitting by decision trees methods cannot be teased out in the performance
of RSF and GBSA. External validation of the RSF and GBSA by external dataset is strongly
warranted (as the 20-fold cross-validation inherently reduced the numbers of patients in each
fold).
8.2 Are there inherent difference or uniqueness between Cox proportional hazards model, and
RSF, GBSA decision trees models?
8.3 Clinical applicability: The authors are advised to add descriptions on how gastroenterologists
and patients can benefit from the newly proposed prediction models in this study, because
most practitioners are probably practicing in hospitals where PSC cohorts have less frequent
CCA. Will the prediction performance still be the same as compared to PSC cohort with higher
CCA percentage? Furthermore, in addition to more accurate prediction of CCA risk, will these
earlier detection trigger more frequent / invasive diagnostic efforts to find earlier CCA, thereby
reducing the overall mortality rate of life-long PSC disease course? These may deserve some
discussion.
8.4 Evaluation of performance by C-Index: Other than C-Index, what additional indicator can
be tested to demonstrate improved performance of these three prediction models than state-of-
the-art models in Table 4?
9 The manuscript is well-written and the study results well-explained and discussed.
Specific comments:
1. Table 1S was extremely lengthy (283 pages) and difficult to understand.
2. Supplementary tables 2, 3, 4 were not explained in the maintext. The Tables 2S, 3S, and 4S
were to predict the 5-yr risk of hepatic decompensation (while the main text was to predict the
development of CCA)?
Title: Predicting Cholangiocarcinoma in Primary Sclerosing Cholangitis: Using Artificial
Intelligence, Clinical and Laboratory Data
Version: 2 Date: 15 Dec 2022
Reviewer's report:
General comments:
1 This is a large PSC cohort from a single medical center system with 1,459 PSC patients, in
which 125 (8.6%) patients developed cholangiocarcinoma (CCA) and 1,035 (70.9%) had been
diagnosed with IBD.
2 The study adopted three prediction models: (1) statistics-based model (Cox proportional
hazards model, CoxPH) and (2) two machine learning models (decision trees methods: random
survival forest, RSF, and gradient boosting survival analysis, GBSA) to predict PSC patients
who would develop cholangiocarcinoma in shorter times (ie, shorter CCA-free survival among
PSC patients).
3 The performance of these three prediction models were evaluated by concordance index (C-
index), using 20-fold Monte Carlo cross-validation with 80%-20% train-test split.
4 The main finding in Table 4 showed that (1) the CoxPH, RSF, GBSA prediction model
performed better than current state-of-the-art prediction models: Mayo PSC risk score, MELD
score, and PREsTo score by C-index.
5 These CoxPH, RSF, GBSA models also showed good prediction C-index at 2yrs post diagnosis
of PSC and 5 yrs post diagnosis of PSC, which was not addressed by other state-of-the-art
prediction models.
6 The authors’ attempt to include bile acid profiles into prediction models demonstrated the lack
of improved prediction performance.
7 The authors are to be applauded by the adoption of modern technologies of machine learning
methods to provide better prediction models.
8 However, there are some major issues regarding the decision trees models:
8.1 Overfitting: overfitting by decision trees methods cannot be teased out in the performance
of RSF and GBSA. External validation of the RSF and GBSA by external dataset is strongly
warranted (as the 20-fold cross-validation inherently reduced the numbers of patients in each
fold).
8.2 Are there inherent difference or uniqueness between Cox proportional hazards model, and
RSF, GBSA decision trees models?
8.3 Clinical applicability: The authors are advised to add descriptions on how gastroenterologists
and patients can benefit from the newly proposed prediction models in this study, because
most practitioners are probably practicing in hospitals where PSC cohorts have less frequent
CCA. Will the prediction performance still be the same as compared to PSC cohort with higher
CCA percentage? Furthermore, in addition to more accurate prediction of CCA risk, will these
earlier detection trigger more frequent / invasive diagnostic efforts to find earlier CCA, thereby
reducing the overall mortality rate of life-long PSC disease course? These may deserve some
discussion.
8.4 Evaluation of performance by C-Index: Other than C-Index, what additional indicator can
be tested to demonstrate improved performance of these three prediction models than state-of-
the-art models in Table 4?
9 The manuscript is well-written and the study results well-explained and discussed.
Specific comments:
1. Table 1S was extremely lengthy (283 pages) and difficult to understand.
2. Supplementary tables 2, 3, 4 were not explained in the maintext. The Tables 2S, 3S, and 4S
were to predict the 5-yr risk of hepatic decompensation (while the main text was to predict the
development of CCA)?
Please review this paper.
user
Title: Predicting Cholangiocarcinoma in Primary Sclerosing Cholangitis: Using Artificial
Intelligence, Clinical and Laboratory Data
Version: 2 Date: 15 Dec 2022
Reviewer's report:
General comments:
1 This is a large PSC cohort from a single medical center system with 1,459 PSC patients, in
which 125 (8.6%) patients developed cholangiocarcinoma (CCA) and 1,035 (70.9%) had been
diagnosed with IBD.
2 The study adopted three prediction models: (1) statistics-based model (Cox proportional
hazards model, CoxPH) and (2) two machine learning models (decision trees methods: random
survival forest, RSF, and gradient boosting survival analysis, GBSA) to predict PSC patients
who would develop cholangiocarcinoma in shorter times (ie, shorter CCA-free survival among
PSC patients).
3 The performance of these three prediction models were evaluated by concordance index (C-
index), using 20-fold Monte Carlo cross-validation with 80%-20% train-test split.
4 The main finding in Table 4 showed that (1) the CoxPH, RSF, GBSA prediction model
performed better than current state-of-the-art prediction models: Mayo PSC risk score, MELD
score, and PREsTo score by C-index.
5 These CoxPH, RSF, GBSA models also showed good prediction C-index at 2yrs post diagnosis
of PSC and 5 yrs post diagnosis of PSC, which was not addressed by other state-of-the-art
prediction models.
6 The authors’ attempt to include bile acid profiles into prediction models demonstrated the lack
of improved prediction performance.
7 The authors are to be applauded by the adoption of modern technologies of machine learning
methods to provide better prediction models.
8 However, there are some major issues regarding the decision trees models:
8.1 Overfitting: overfitting by decision trees methods cannot be teased out in the performance
of RSF and GBSA. External validation of the RSF and GBSA by external dataset is strongly
warranted (as the 20-fold cross-validation inherently reduced the numbers of patients in each
fold).
8.2 Are there inherent difference or uniqueness between Cox proportional hazards model, and
RSF, GBSA decision trees models?
8.3 Clinical applicability: The authors are advised to add descriptions on how gastroenterologists
and patients can benefit from the newly proposed prediction models in this study, because
most practitioners are probably practicing in hospitals where PSC cohorts have less frequent
CCA. Will the prediction performance still be the same as compared to PSC cohort with higher
CCA percentage? Furthermore, in addition to more accurate prediction of CCA risk, will these
earlier detection trigger more frequent / invasive diagnostic efforts to find earlier CCA, thereby
reducing the overall mortality rate of life-long PSC disease course? These may deserve some
discussion.
8.4 Evaluation of performance by C-Index: Other than C-Index, what additional indicator can
be tested to demonstrate improved performance of these three prediction models than state-of-
the-art models in Table 4?
9 The manuscript is well-written and the study results well-explained and discussed.
Specific comments:
1. Table 1S was extremely lengthy (283 pages) and difficult to understand.
2. Supplementary tables 2, 3, 4 were not explained in the maintext. The Tables 2S, 3S, and 4S
were to predict the 5-yr risk of hepatic decompensation (while the main text was to predict the
development of CCA)?
assistant