Title: Machine Learning-Based Survival Rate Prediction of Korean Hepatocellular Carcinoma
Patients using multi-center data
Version: 4 Date: 20 Jul 2021
Reviewer's report:
The authors have done an interesting work on developing machine learning-based prognostic
models for patients with HCC. They have developed and compared several machine learning
models using a large dataset from Korean registry. The following are my comments and
suggestions:
Major comments
1. Please describe which groups of patients/hospitals (primary, secondary, tertiary care, …) were
included in the registry.
2. Please provide HCC stages (e.g., BCLC staging) of the enrolled patients. This would enable
readers to gain a picture of HCC spectrum in this patient cohort, and determine whether the
finding in this study can be applied to their patients.
3. Since the authors excluded cases that had >30% missing input features, the author should
provide the number of cases that were excluded, along with the proportion of outcomes in
the group of excluded cases. This is to explore whether the missing cases were either Missing
completely at random (MCAR), Missing at random (MAR), or Missing not at random (MNAR),
which would affect the predictive performance of the model.
4. In this work, the authors modeled prognostic outcome as a binary variable (<60 or >60 months’
survival). When the outcome was modeled as a binary variable, lost-to-follow-up cases is an
important issue to be addressed as this can result in either better or poorer outcomes. The authors
should set criteria for lost to follow up, and report the numbers, as well as the reasons for lost
to follow up.
5. The authors performed 5-fold cross validation without a separate test set. The reported
performance based on 5-fold cross validation alone can therefore be overestimated, due to
“hyperparameter overfitting” (which is different from “parameter overfitting”). Hyperparameter
overfitting occurs when a model’s hyperparameters are adjusted several times until good
performance is achieved. I think the dataset is large enough to be divided into a test set first,
then the author use the remaining data for the 5-fold cross validation. Finally, the authors can
use the best model (from 5-fold cross validation) to evaluate performance on the test set.
Minor comments
1. Grammar errors and typos exist throughout the manuscript. Writing style has to be improved.
For example,
- There is a typo in Line 8 under Patients subsection. “hyper hancement” should be spelled as
“hyper-enhancement”
- The statement “then we got the average of accuracy, precision, sensitivity and F1 score” was
duplicated. (under the subsection: Data splits for machine learning processing)
2. The authors may want to specifically use the term “F1 score” instead of “F score” in the
manuscript body. There is a family of F scores available, e.g., F1, F0.5, F2, which weigh
precision and recall differently.
3. In the method section, the authors describe the details of each model very extensively. This
is more appropriate for the Review article but not for the Original article.
Title: Machine Learning-Based Survival Rate Prediction of Korean Hepatocellular Carcinoma
Patients using multi-center data
Version: 4 Date: 20 Jul 2021
Reviewer's report:
The authors have done an interesting work on developing machine learning-based prognostic
models for patients with HCC. They have developed and compared several machine learning
models using a large dataset from Korean registry. The following are my comments and
suggestions:
Major comments
1. Please describe which groups of patients/hospitals (primary, secondary, tertiary care, …) were
included in the registry.
2. Please provide HCC stages (e.g., BCLC staging) of the enrolled patients. This would enable
readers to gain a picture of HCC spectrum in this patient cohort, and determine whether the
finding in this study can be applied to their patients.
3. Since the authors excluded cases that had >30% missing input features, the author should
provide the number of cases that were excluded, along with the proportion of outcomes in
the group of excluded cases. This is to explore whether the missing cases were either Missing
completely at random (MCAR), Missing at random (MAR), or Missing not at random (MNAR),
which would affect the predictive performance of the model.
4. In this work, the authors modeled prognostic outcome as a binary variable (<60 or >60 months’
survival). When the outcome was modeled as a binary variable, lost-to-follow-up cases is an
important issue to be addressed as this can result in either better or poorer outcomes. The authors
should set criteria for lost to follow up, and report the numbers, as well as the reasons for lost
to follow up.
5. The authors performed 5-fold cross validation without a separate test set. The reported
performance based on 5-fold cross validation alone can therefore be overestimated, due to
“hyperparameter overfitting” (which is different from “parameter overfitting”). Hyperparameter
overfitting occurs when a model’s hyperparameters are adjusted several times until good
performance is achieved. I think the dataset is large enough to be divided into a test set first,
then the author use the remaining data for the 5-fold cross validation. Finally, the authors can
use the best model (from 5-fold cross validation) to evaluate performance on the test set.
Minor comments
1. Grammar errors and typos exist throughout the manuscript. Writing style has to be improved.
For example,
- There is a typo in Line 8 under Patients subsection. “hyper hancement” should be spelled as
“hyper-enhancement”
- The statement “then we got the average of accuracy, precision, sensitivity and F1 score” was
duplicated. (under the subsection: Data splits for machine learning processing)
2. The authors may want to specifically use the term “F1 score” instead of “F score” in the
manuscript body. There is a family of F scores available, e.g., F1, F0.5, F2, which weigh
precision and recall differently.
3. In the method section, the authors describe the details of each model very extensively. This
is more appropriate for the Review article but not for the Original article.
Please review this paper.
user
Title: Machine Learning-Based Survival Rate Prediction of Korean Hepatocellular Carcinoma
Patients using multi-center data
Version: 4 Date: 20 Jul 2021
Reviewer's report:
The authors have done an interesting work on developing machine learning-based prognostic
models for patients with HCC. They have developed and compared several machine learning
models using a large dataset from Korean registry. The following are my comments and
suggestions:
Major comments
1. Please describe which groups of patients/hospitals (primary, secondary, tertiary care, …) were
included in the registry.
2. Please provide HCC stages (e.g., BCLC staging) of the enrolled patients. This would enable
readers to gain a picture of HCC spectrum in this patient cohort, and determine whether the
finding in this study can be applied to their patients.
3. Since the authors excluded cases that had >30% missing input features, the author should
provide the number of cases that were excluded, along with the proportion of outcomes in
the group of excluded cases. This is to explore whether the missing cases were either Missing
completely at random (MCAR), Missing at random (MAR), or Missing not at random (MNAR),
which would affect the predictive performance of the model.
4. In this work, the authors modeled prognostic outcome as a binary variable (<60 or >60 months’
survival). When the outcome was modeled as a binary variable, lost-to-follow-up cases is an
important issue to be addressed as this can result in either better or poorer outcomes. The authors
should set criteria for lost to follow up, and report the numbers, as well as the reasons for lost
to follow up.
5. The authors performed 5-fold cross validation without a separate test set. The reported
performance based on 5-fold cross validation alone can therefore be overestimated, due to
“hyperparameter overfitting” (which is different from “parameter overfitting”). Hyperparameter
overfitting occurs when a model’s hyperparameters are adjusted several times until good
performance is achieved. I think the dataset is large enough to be divided into a test set first,
then the author use the remaining data for the 5-fold cross validation. Finally, the authors can
use the best model (from 5-fold cross validation) to evaluate performance on the test set.
Minor comments
1. Grammar errors and typos exist throughout the manuscript. Writing style has to be improved.
For example,
- There is a typo in Line 8 under Patients subsection. “hyper hancement” should be spelled as
“hyper-enhancement”
- The statement “then we got the average of accuracy, precision, sensitivity and F1 score” was
duplicated. (under the subsection: Data splits for machine learning processing)
2. The authors may want to specifically use the term “F1 score” instead of “F score” in the
manuscript body. There is a family of F scores available, e.g., F1, F0.5, F2, which weigh
precision and recall differently.
3. In the method section, the authors describe the details of each model very extensively. This
is more appropriate for the Review article but not for the Original article.
assistant