Title: Performance and comparison of artificial intelligence and human experts in the detection
and classification of colonic polyps
Version: 0 Date: 09 Mar 2022
Reviewer's report:
General remarks
In this manuscript the authors perform a meta-analysis of 16 studies to analyze the performance
of machine learning for detecting polyps and compare it to human performance (experts and
non-experts).
The authors did a good job in communicating their objectives. Also, the manuscript is well
structured and easy to read.
However, there are a few issues which have been pointed out in detail below. For instance, the
authors should address major limitations more offensively, like the lack of an external validation
in 15/16 studies included in the meta-analysis. The claims made in this study might have to be
weakened in light of the limitations.
Major issues
Background:
Please explain the following sentence: “However, these characteristic patterns are often
similar between the normal structure of polyps and similar polyps, leading to reduced model
performance, leading to reduced model performance.” Additionally, please elaborate how
reference [14] backs up these claims for diagnosis and detection of polyps. In my opinion, the
sentence is contradicted by many successful studies using Radiomics features together with
classical machine learning algorithms for diagnostic or prognostic purposes based on medical
images. If you are referring explicitly to detection, please clarify that in the paragraph.
Inclusion and exclusion criteria:
You do not report any criteria related to the malignancy of polyps for inclusion or exclusion of
data. How do you make sure two groups in the respective binary classification problems (polyp
vs. no polyp) are comparable across different studies?
Statistical analysis:
Please elaborate the “statistical effects mode”. What is it and why do you use it. Also, please
comment on how you dealt with different operating points (classification thresholds)
Description of the studies included:
Only 1 study out of 16 in your meta-analysis was externally verified. Therefore, the findings of
the other 15/16 remain questionable in terms of generalization ability. This is a major weakness
of the manuscript and should be addressed more prominently, especially it should be mentioned
in the abstract.
Diagnostic performance of AI/humans:
It appears that you are aggregating sensitivity and specificity values from different machine
learning models on different datasets using a random effects model. Please explain in the
manuscript why you choose to use a random effects model. Also, sensitivity and specificity are
both calculated from the confusion matrix which in turn depends on the classification threshold
used in the respective study. Please comment on the fact that you can tune sensitivity and
specificity (in contrast to ROC-AUC) according to specific needs by varying the classification
threshold and whether or not that has an impact on the concept of aggregating sensitivity and
specificity values from different studies.
Minor issues
Abstract:
Please elaborate which AUC. By itself, AUC just means “area under the curve”, but that does
not specify which curve you are talking about.
Also, you claim that AI performance is on par with experts and is expected to improve the
performance of young doctors. Can you define the term expert more clearly in the context of
this manuscript?
Finally, who exactly expects AI to improve the performance of young doctors. Is that your
expectation based on the results of this study? If so, please phrase it that way. If it is the
general opinion of the scientific community, please provide appropriate references in the actual
Discussion section at the end of the manuscript.
Background:
Please reconsider the sentence: “Through weight sharing, CNN can reduce the error between
predicted values and actual results, and achieve good results in image recognition.” Weight
sharing is not the only property that makes CNNs successful.
Data selection and extraction:
The following passage was difficult for me to follow and I am not sure if I understood it correctly:
“To make the most of the data, if the same study contains more than one diagnostic experiment,
the sum of data of the same type is taken (For example[17], two diagnostic experiment were
conducted, and the final extracted data was the sum of the corresponding indicators of each test,
such as the sensitivity of experts was 92.5% (699/756)). The following data were also extracted
from each study: Author name, title, year of publication, country, sample size, type of AI, number
of endoscopic physicians, and external validation.” Please consider restructuring.
Discussion:
Please make it easier for the reader to verify the last part of the following sentence: “In this study,
we try to explore the traditional ML methods (such as Random forests model (RF), support
vector machine (SVM), linear classifier, K neighbor, etc.) and DL (such as CNN) in colon polyps
recognition performance comparison, and the results showed that between DL and traditional
ML in polyp detection performance, there was no obvious difference (P>0.05).” For instance,
you could add a column for the “ROC-AUC” to Table 2. Or you could add a designated table
or section, where you compare classical ML to DL performance.
Conclusion:
See minor issues of the abstract.
Title: Performance and comparison of artificial intelligence and human experts in the detection
and classification of colonic polyps
Version: 0 Date: 09 Mar 2022
Reviewer's report:
General remarks
In this manuscript the authors perform a meta-analysis of 16 studies to analyze the performance
of machine learning for detecting polyps and compare it to human performance (experts and
non-experts).
The authors did a good job in communicating their objectives. Also, the manuscript is well
structured and easy to read.
However, there are a few issues which have been pointed out in detail below. For instance, the
authors should address major limitations more offensively, like the lack of an external validation
in 15/16 studies included in the meta-analysis. The claims made in this study might have to be
weakened in light of the limitations.
Major issues
Background:
Please explain the following sentence: “However, these characteristic patterns are often
similar between the normal structure of polyps and similar polyps, leading to reduced model
performance, leading to reduced model performance.” Additionally, please elaborate how
reference [14] backs up these claims for diagnosis and detection of polyps. In my opinion, the
sentence is contradicted by many successful studies using Radiomics features together with
classical machine learning algorithms for diagnostic or prognostic purposes based on medical
images. If you are referring explicitly to detection, please clarify that in the paragraph.
Inclusion and exclusion criteria:
You do not report any criteria related to the malignancy of polyps for inclusion or exclusion of
data. How do you make sure two groups in the respective binary classification problems (polyp
vs. no polyp) are comparable across different studies?
Statistical analysis:
Please elaborate the “statistical effects mode”. What is it and why do you use it. Also, please
comment on how you dealt with different operating points (classification thresholds)
Description of the studies included:
Only 1 study out of 16 in your meta-analysis was externally verified. Therefore, the findings of
the other 15/16 remain questionable in terms of generalization ability. This is a major weakness
of the manuscript and should be addressed more prominently, especially it should be mentioned
in the abstract.
Diagnostic performance of AI/humans:
It appears that you are aggregating sensitivity and specificity values from different machine
learning models on different datasets using a random effects model. Please explain in the
manuscript why you choose to use a random effects model. Also, sensitivity and specificity are
both calculated from the confusion matrix which in turn depends on the classification threshold
used in the respective study. Please comment on the fact that you can tune sensitivity and
specificity (in contrast to ROC-AUC) according to specific needs by varying the classification
threshold and whether or not that has an impact on the concept of aggregating sensitivity and
specificity values from different studies.
Minor issues
Abstract:
Please elaborate which AUC. By itself, AUC just means “area under the curve”, but that does
not specify which curve you are talking about.
Also, you claim that AI performance is on par with experts and is expected to improve the
performance of young doctors. Can you define the term expert more clearly in the context of
this manuscript?
Finally, who exactly expects AI to improve the performance of young doctors. Is that your
expectation based on the results of this study? If so, please phrase it that way. If it is the
general opinion of the scientific community, please provide appropriate references in the actual
Discussion section at the end of the manuscript.
Background:
Please reconsider the sentence: “Through weight sharing, CNN can reduce the error between
predicted values and actual results, and achieve good results in image recognition.” Weight
sharing is not the only property that makes CNNs successful.
Data selection and extraction:
The following passage was difficult for me to follow and I am not sure if I understood it correctly:
“To make the most of the data, if the same study contains more than one diagnostic experiment,
the sum of data of the same type is taken (For example[17], two diagnostic experiment were
conducted, and the final extracted data was the sum of the corresponding indicators of each test,
such as the sensitivity of experts was 92.5% (699/756)). The following data were also extracted
from each study: Author name, title, year of publication, country, sample size, type of AI, number
of endoscopic physicians, and external validation.” Please consider restructuring.
Discussion:
Please make it easier for the reader to verify the last part of the following sentence: “In this study,
we try to explore the traditional ML methods (such as Random forests model (RF), support
vector machine (SVM), linear classifier, K neighbor, etc.) and DL (such as CNN) in colon polyps
recognition performance comparison, and the results showed that between DL and traditional
ML in polyp detection performance, there was no obvious difference (P>0.05).” For instance,
you could add a column for the “ROC-AUC” to Table 2. Or you could add a designated table
or section, where you compare classical ML to DL performance.
Conclusion:
See minor issues of the abstract.
Please review this paper.
user
Title: Performance and comparison of artificial intelligence and human experts in the detection
and classification of colonic polyps
Version: 0 Date: 09 Mar 2022
Reviewer's report:
General remarks
In this manuscript the authors perform a meta-analysis of 16 studies to analyze the performance
of machine learning for detecting polyps and compare it to human performance (experts and
non-experts).
The authors did a good job in communicating their objectives. Also, the manuscript is well
structured and easy to read.
However, there are a few issues which have been pointed out in detail below. For instance, the
authors should address major limitations more offensively, like the lack of an external validation
in 15/16 studies included in the meta-analysis. The claims made in this study might have to be
weakened in light of the limitations.
Major issues
Background:
Please explain the following sentence: “However, these characteristic patterns are often
similar between the normal structure of polyps and similar polyps, leading to reduced model
performance, leading to reduced model performance.” Additionally, please elaborate how
reference [14] backs up these claims for diagnosis and detection of polyps. In my opinion, the
sentence is contradicted by many successful studies using Radiomics features together with
classical machine learning algorithms for diagnostic or prognostic purposes based on medical
images. If you are referring explicitly to detection, please clarify that in the paragraph.
Inclusion and exclusion criteria:
You do not report any criteria related to the malignancy of polyps for inclusion or exclusion of
data. How do you make sure two groups in the respective binary classification problems (polyp
vs. no polyp) are comparable across different studies?
Statistical analysis:
Please elaborate the “statistical effects mode”. What is it and why do you use it. Also, please
comment on how you dealt with different operating points (classification thresholds)
Description of the studies included:
Only 1 study out of 16 in your meta-analysis was externally verified. Therefore, the findings of
the other 15/16 remain questionable in terms of generalization ability. This is a major weakness
of the manuscript and should be addressed more prominently, especially it should be mentioned
in the abstract.
Diagnostic performance of AI/humans:
It appears that you are aggregating sensitivity and specificity values from different machine
learning models on different datasets using a random effects model. Please explain in the
manuscript why you choose to use a random effects model. Also, sensitivity and specificity are
both calculated from the confusion matrix which in turn depends on the classification threshold
used in the respective study. Please comment on the fact that you can tune sensitivity and
specificity (in contrast to ROC-AUC) according to specific needs by varying the classification
threshold and whether or not that has an impact on the concept of aggregating sensitivity and
specificity values from different studies.
Minor issues
Abstract:
Please elaborate which AUC. By itself, AUC just means “area under the curve”, but that does
not specify which curve you are talking about.
Also, you claim that AI performance is on par with experts and is expected to improve the
performance of young doctors. Can you define the term expert more clearly in the context of
this manuscript?
Finally, who exactly expects AI to improve the performance of young doctors. Is that your
expectation based on the results of this study? If so, please phrase it that way. If it is the
general opinion of the scientific community, please provide appropriate references in the actual
Discussion section at the end of the manuscript.
Background:
Please reconsider the sentence: “Through weight sharing, CNN can reduce the error between
predicted values and actual results, and achieve good results in image recognition.” Weight
sharing is not the only property that makes CNNs successful.
Data selection and extraction:
The following passage was difficult for me to follow and I am not sure if I understood it correctly:
“To make the most of the data, if the same study contains more than one diagnostic experiment,
the sum of data of the same type is taken (For example[17], two diagnostic experiment were
conducted, and the final extracted data was the sum of the corresponding indicators of each test,
such as the sensitivity of experts was 92.5% (699/756)). The following data were also extracted
from each study: Author name, title, year of publication, country, sample size, type of AI, number
of endoscopic physicians, and external validation.” Please consider restructuring.
Discussion:
Please make it easier for the reader to verify the last part of the following sentence: “In this study,
we try to explore the traditional ML methods (such as Random forests model (RF), support
vector machine (SVM), linear classifier, K neighbor, etc.) and DL (such as CNN) in colon polyps
recognition performance comparison, and the results showed that between DL and traditional
ML in polyp detection performance, there was no obvious difference (P>0.05).” For instance,
you could add a column for the “ROC-AUC” to Table 2. Or you could add a designated table
or section, where you compare classical ML to DL performance.
Conclusion:
See minor issues of the abstract.
assistant