Title: Performance and comparison of artificial intelligence and human experts in the detection
and classification of colonic polyps
Version: 1 Date: 26 May 2022
Reviewer's report:
Even though the authors gave feedback to my comments, several parts are unclear. Thus, the
real outcome of this work is still unclear.
In the reply, the authors wrote “Thus, our paper mainly compares the detection and classification
performance of polyps. In order to clarify the purpose, we have revised the title and related
expression.”
However, in the last paragraph of the Introduction, the authors wrote “The main purpose of this
study is to analyze the performance of different AI models in endoscopic colonic polyp detection
and to compare them with doctors with different experience.” In given figures, the authors
compared performances of detection and pathological classification of different modalities. The
logical flow of the descriptions still sounds weird.
The given explanations for interpretation and meaning of Fig 2 and are unsatisfactory. These
machine learning models and results came from different modalities, purposes, and datasets.
The purpose and real outcome should be Cleary described in the manuscript. In Figs. 2 and 3, the
authors picked up the models given by Refs. [24,38,29,7,17,40,23,41,42,21,43,11,44,35,45,14].
Since these models are developed for different purposed (detection and classification) with
different modality (normal, magnified, ultramagnified colonoscopic images under white light
and narrow band imaging, CT colonography data, and …), these differences should be
summarized for explaining the models. Furthermore, the construction and evaluation of machine
learning models highly depend on the scale and property of dataset (the number of images,
patients, how to split dataset for training, validation and testing, and so on). After re-summarising
these differences as a figure or table, the authors can describe more detail information for
discussion. From the current version, the real outcome of this work is still unclear. Please add
information, discussion and re-summarise the real outcome of this work.
The authors described ‘Threshold effect is due to studies published in different date and using
different thresholds to define positive or negative, which results in the difference in SEN, SPE
or likelihood ratio between the studies.’ I agree with this point. However, the description in
the Conclusion, ‘On the other hand, we also find that AI has a high sensitivity and moderate
specificity for polyp detection and classification, which is similar to that of human experts’,
needs additional explanation and discussion. Since the final output of machine learning model is
controlled by a thresholding criteria as a hyperparaemter, the characteristics of high Sensitivity
and moderate Specificity are designed by researchers.  By changing the a hyperparameter setting
for an AI’s decision, characteristics about sensitivity and specificity can change. What does
derive the characteristics of high sensitivity and moderate specificity? These points should be
clearly described in the Discussion.
The feedback of the authors to my third comments is off the points. As I commented, the quality
grading based QUADAS-2 is done by two reviewers. For example, the risk of bias was classified
as ‘low’, ‘high’, or ‘unclear’. In my opinion, their decisions are subjective. As fair grading,
how did the authors deal with the subject bias in this grading? Is the number of the reviewers
is appropriate?
In Figs, 3G-I, what each line and circle means is unclear. Please add appropriate labels as
legends.
Typo in the Conclusion (page 12, line 15): I is necessary to develop a new
Title: Performance and comparison of artificial intelligence and human experts in the detection
and classification of colonic polyps
Version: 1 Date: 26 May 2022
Reviewer's report:
Even though the authors gave feedback to my comments, several parts are unclear. Thus, the
real outcome of this work is still unclear.
In the reply, the authors wrote “Thus, our paper mainly compares the detection and classification
performance of polyps. In order to clarify the purpose, we have revised the title and related
expression.”
However, in the last paragraph of the Introduction, the authors wrote “The main purpose of this
study is to analyze the performance of different AI models in endoscopic colonic polyp detection
and to compare them with doctors with different experience.” In given figures, the authors
compared performances of detection and pathological classification of different modalities. The
logical flow of the descriptions still sounds weird.
The given explanations for interpretation and meaning of Fig 2 and are unsatisfactory. These
machine learning models and results came from different modalities, purposes, and datasets.
The purpose and real outcome should be Cleary described in the manuscript. In Figs. 2 and 3, the
authors picked up the models given by Refs. [24,38,29,7,17,40,23,41,42,21,43,11,44,35,45,14].
Since these models are developed for different purposed (detection and classification) with
different modality (normal, magnified, ultramagnified colonoscopic images under white light
and narrow band imaging, CT colonography data, and …), these differences should be
summarized for explaining the models. Furthermore, the construction and evaluation of machine
learning models highly depend on the scale and property of dataset (the number of images,
patients, how to split dataset for training, validation and testing, and so on). After re-summarising
these differences as a figure or table, the authors can describe more detail information for
discussion. From the current version, the real outcome of this work is still unclear. Please add
information, discussion and re-summarise the real outcome of this work.
The authors described ‘Threshold effect is due to studies published in different date and using
different thresholds to define positive or negative, which results in the difference in SEN, SPE
or likelihood ratio between the studies.’ I agree with this point. However, the description in
the Conclusion, ‘On the other hand, we also find that AI has a high sensitivity and moderate
specificity for polyp detection and classification, which is similar to that of human experts’,
needs additional explanation and discussion. Since the final output of machine learning model is
controlled by a thresholding criteria as a hyperparaemter, the characteristics of high Sensitivity
and moderate Specificity are designed by researchers.  By changing the a hyperparameter setting
for an AI’s decision, characteristics about sensitivity and specificity can change. What does
derive the characteristics of high sensitivity and moderate specificity? These points should be
clearly described in the Discussion.
The feedback of the authors to my third comments is off the points. As I commented, the quality
grading based QUADAS-2 is done by two reviewers. For example, the risk of bias was classified
as ‘low’, ‘high’, or ‘unclear’. In my opinion, their decisions are subjective. As fair grading,
how did the authors deal with the subject bias in this grading? Is the number of the reviewers
is appropriate?
In Figs, 3G-I, what each line and circle means is unclear. Please add appropriate labels as
legends.
Typo in the Conclusion (page 12, line 15): I is necessary to develop a new
Please review this paper.
user
Title: Performance and comparison of artificial intelligence and human experts in the detection
and classification of colonic polyps
Version: 1 Date: 26 May 2022
Reviewer's report:
Even though the authors gave feedback to my comments, several parts are unclear. Thus, the
real outcome of this work is still unclear.
In the reply, the authors wrote “Thus, our paper mainly compares the detection and classification
performance of polyps. In order to clarify the purpose, we have revised the title and related
expression.”
However, in the last paragraph of the Introduction, the authors wrote “The main purpose of this
study is to analyze the performance of different AI models in endoscopic colonic polyp detection
and to compare them with doctors with different experience.” In given figures, the authors
compared performances of detection and pathological classification of different modalities. The
logical flow of the descriptions still sounds weird.
The given explanations for interpretation and meaning of Fig 2 and are unsatisfactory. These
machine learning models and results came from different modalities, purposes, and datasets.
The purpose and real outcome should be Cleary described in the manuscript. In Figs. 2 and 3, the
authors picked up the models given by Refs. [24,38,29,7,17,40,23,41,42,21,43,11,44,35,45,14].
Since these models are developed for different purposed (detection and classification) with
different modality (normal, magnified, ultramagnified colonoscopic images under white light
and narrow band imaging, CT colonography data, and …), these differences should be
summarized for explaining the models. Furthermore, the construction and evaluation of machine
learning models highly depend on the scale and property of dataset (the number of images,
patients, how to split dataset for training, validation and testing, and so on). After re-summarising
these differences as a figure or table, the authors can describe more detail information for
discussion. From the current version, the real outcome of this work is still unclear. Please add
information, discussion and re-summarise the real outcome of this work.
The authors described ‘Threshold effect is due to studies published in different date and using
different thresholds to define positive or negative, which results in the difference in SEN, SPE
or likelihood ratio between the studies.’ I agree with this point. However, the description in
the Conclusion, ‘On the other hand, we also find that AI has a high sensitivity and moderate
specificity for polyp detection and classification, which is similar to that of human experts’,
needs additional explanation and discussion. Since the final output of machine learning model is
controlled by a thresholding criteria as a hyperparaemter, the characteristics of high Sensitivity
and moderate Specificity are designed by researchers.  By changing the a hyperparameter setting
for an AI’s decision, characteristics about sensitivity and specificity can change. What does
derive the characteristics of high sensitivity and moderate specificity? These points should be
clearly described in the Discussion.
The feedback of the authors to my third comments is off the points. As I commented, the quality
grading based QUADAS-2 is done by two reviewers. For example, the risk of bias was classified
as ‘low’, ‘high’, or ‘unclear’. In my opinion, their decisions are subjective. As fair grading,
how did the authors deal with the subject bias in this grading? Is the number of the reviewers
is appropriate?
In Figs, 3G-I, what each line and circle means is unclear. Please add appropriate labels as
legends.
Typo in the Conclusion (page 12, line 15): I is necessary to develop a new
assistant