Title: A single-center prospective study evaluating the usefulness of artificial intelligence for
the diagnosis of esophageal squamous cell carcinoma in a real-time setting
Version: 1 Date: 19 Mar 2023
Reviewer:  Ippei Tanaka
Reviewer's report:
In the present study, authors conducted a prospective study to investigate the real-time diagnostic
accuracy of AI system for ESCC in clinical practice. It is clinically important to create a useful AI
system which shows its diagnosis in real time. Although this study did not reveal the usefulness
of AI system in a clinical setting, it is a necessary step to proceed to the further improvement.
Thus, in my opinion, this study is valuable and meaningful. However, there are some concerns
that I would like you to address.
Major comments
Method
1. The experience of endoscopists who diagnosed the endoscopic images is quite different,
such as from 6 years to 26 years. It seems like the diagnostic accuracy of the endoscopist with
the experience of 6 years is inferior to that with 26 years. How about the agreement rate of
endoscopic diagnosis among 5 endoscopists? If possible, please calculate the kappa value among
5 endoscopists.
Result
2. Please discuss the reason why the number of participants in this study did not reach to the
required sample size? Additionally, if it could gather the required number of participants, are
there any possibility that the result of this study would be different?
3. The diagnostic sensitivity of endoscopists for ESCC in the current study is considerably low
compared to some previous studies, which is the most concerning point to me. For example,
according to the study by Dr. Ishihara (J Gastroenterol Hepatol, 2013), sensitivity and specificity
of NBI for differentiating mucosal high-grade from low-grade neoplasia were 85% and 79%,
respectively. Also, according to Dr. Minami (Gastroenterol Res Pract, 2012), those were 91.9%
and 76.7%, respectively. Please clarify the reason for this discrepancy.
4. Although this concern is almost similar to No.3, the sensitivity of endoscopists for the lesion
less than 10mm was only 46.2%. This result may not be easily acceptable for most endoscopists.
Could you please explain the reason for this significant low performance of endoscopists for
such lesions?
5. This study used various scopes such as GIF-H290Z, GIF-H260Z, GIF-Q240Z, GIF-XZ1200,
or GIF-EZ1500. If the quality of the images is related to the performance of AI system as authors
mentioned, did you consider using the same endoscope in this study? I mean why did you decide
to use some relatively old endoscope, such as GIF-H260Z and GIF-Q240Z?
Discussion
6. In page 13, authors mentioned that the failure to prove the non-inferiority of the AI system
was caused by the difference of the images used for validation and in the current clinical study. If
possible, I would like you to show some evidence that the performance of AI system is affected
by the quality of the images. Also, can you describe pixel count of compressed and uncompressed
images?
Minor comments
7. In page 3 line 2, the authors cited only 3, 4, 5. However, there are some reports showing the
effectiveness of definitive CRT treatment for stage # ESCC. Please consider to cite one of them.
How about the following report, Kato K, et al, Gastroenterology 2021?
8. Authors described “Furthermore, endoscopic diagnosis of ESCC is prone to interobserver
differences.” in page10 line7. Has this statement already been revealed by previous studies? If
so, please add such report to the reference.
9. Authors mentioned that “patients continuing antithrombotic drugs, for whom biopsy is
contraindicated according to the Japanese guidelines” in page4 line5. However, the Japanese
guidelines described that “Biopsy with antithrombotics should only be undertaken when
absolutely necessary, and caution should be exercised.” Please describe this point more precisely
based on the comment of the Japanese guideline.
10. In the preparation of the training dataset (page4), are the images used in this stage WLI or
NBI?
11. Although authors mentioned that “Our previous study showed an accuracy of 83% for the AI
system when the accuracy of endoscopic diagnosis was 78% 12.” in page7 line6, the meaning
of this sentence may be a bit unclear. Does the accuracy of endoscopic diagnosis mean the
diagnostic accuracy of endoscopists?
12. Please add the explanation for the abbreviation of Ce, Ut, Mt and Lt in the Table1 and Table3.
Moreover, in the latest Japanese guidelines, the description of “Ae” is not used and replaced
with “Jz”. Please consider changing it.
13. In the table1, noncancerous contains SIN, atypical epithelium, esophagitis and “others”
which shows the largest number (86). However, I am wondering what “others” contain. Could
you cite one or two examples of “others”?
14.Could you please add the pathological image of this lesion as Figure 3d.
15.  Authors mentioned “Second, the proportion of ESCC in the target lesions differs from that
in previous studies.” in page 14 line 9. Please consider adding a reference.
Title: A single-center prospective study evaluating the usefulness of artificial intelligence for
the diagnosis of esophageal squamous cell carcinoma in a real-time setting
Version: 1 Date: 19 Mar 2023
Reviewer:  Ippei Tanaka
Reviewer's report:
In the present study, authors conducted a prospective study to investigate the real-time diagnostic
accuracy of AI system for ESCC in clinical practice. It is clinically important to create a useful AI
system which shows its diagnosis in real time. Although this study did not reveal the usefulness
of AI system in a clinical setting, it is a necessary step to proceed to the further improvement.
Thus, in my opinion, this study is valuable and meaningful. However, there are some concerns
that I would like you to address.
Major comments
Method
1. The experience of endoscopists who diagnosed the endoscopic images is quite different,
such as from 6 years to 26 years. It seems like the diagnostic accuracy of the endoscopist with
the experience of 6 years is inferior to that with 26 years. How about the agreement rate of
endoscopic diagnosis among 5 endoscopists? If possible, please calculate the kappa value among
5 endoscopists.
Result
2. Please discuss the reason why the number of participants in this study did not reach to the
required sample size? Additionally, if it could gather the required number of participants, are
there any possibility that the result of this study would be different?
3. The diagnostic sensitivity of endoscopists for ESCC in the current study is considerably low
compared to some previous studies, which is the most concerning point to me. For example,
according to the study by Dr. Ishihara (J Gastroenterol Hepatol, 2013), sensitivity and specificity
of NBI for differentiating mucosal high-grade from low-grade neoplasia were 85% and 79%,
respectively. Also, according to Dr. Minami (Gastroenterol Res Pract, 2012), those were 91.9%
and 76.7%, respectively. Please clarify the reason for this discrepancy.
4. Although this concern is almost similar to No.3, the sensitivity of endoscopists for the lesion
less than 10mm was only 46.2%. This result may not be easily acceptable for most endoscopists.
Could you please explain the reason for this significant low performance of endoscopists for
such lesions?
5. This study used various scopes such as GIF-H290Z, GIF-H260Z, GIF-Q240Z, GIF-XZ1200,
or GIF-EZ1500. If the quality of the images is related to the performance of AI system as authors
mentioned, did you consider using the same endoscope in this study? I mean why did you decide
to use some relatively old endoscope, such as GIF-H260Z and GIF-Q240Z?
Discussion
6. In page 13, authors mentioned that the failure to prove the non-inferiority of the AI system
was caused by the difference of the images used for validation and in the current clinical study. If
possible, I would like you to show some evidence that the performance of AI system is affected
by the quality of the images. Also, can you describe pixel count of compressed and uncompressed
images?
Minor comments
7. In page 3 line 2, the authors cited only 3, 4, 5. However, there are some reports showing the
effectiveness of definitive CRT treatment for stage # ESCC. Please consider to cite one of them.
How about the following report, Kato K, et al, Gastroenterology 2021?
8. Authors described “Furthermore, endoscopic diagnosis of ESCC is prone to interobserver
differences.” in page10 line7. Has this statement already been revealed by previous studies? If
so, please add such report to the reference.
9. Authors mentioned that “patients continuing antithrombotic drugs, for whom biopsy is
contraindicated according to the Japanese guidelines” in page4 line5. However, the Japanese
guidelines described that “Biopsy with antithrombotics should only be undertaken when
absolutely necessary, and caution should be exercised.” Please describe this point more precisely
based on the comment of the Japanese guideline.
10. In the preparation of the training dataset (page4), are the images used in this stage WLI or
NBI?
11. Although authors mentioned that “Our previous study showed an accuracy of 83% for the AI
system when the accuracy of endoscopic diagnosis was 78% 12.” in page7 line6, the meaning
of this sentence may be a bit unclear. Does the accuracy of endoscopic diagnosis mean the
diagnostic accuracy of endoscopists?
12. Please add the explanation for the abbreviation of Ce, Ut, Mt and Lt in the Table1 and Table3.
Moreover, in the latest Japanese guidelines, the description of “Ae” is not used and replaced
with “Jz”. Please consider changing it.
13. In the table1, noncancerous contains SIN, atypical epithelium, esophagitis and “others”
which shows the largest number (86). However, I am wondering what “others” contain. Could
you cite one or two examples of “others”?
14.Could you please add the pathological image of this lesion as Figure 3d.
15.  Authors mentioned “Second, the proportion of ESCC in the target lesions differs from that
in previous studies.” in page 14 line 9. Please consider adding a reference.
Please review this paper.
user
Title: A single-center prospective study evaluating the usefulness of artificial intelligence for
the diagnosis of esophageal squamous cell carcinoma in a real-time setting
Version: 1 Date: 19 Mar 2023
Reviewer:  Ippei Tanaka
Reviewer's report:
In the present study, authors conducted a prospective study to investigate the real-time diagnostic
accuracy of AI system for ESCC in clinical practice. It is clinically important to create a useful AI
system which shows its diagnosis in real time. Although this study did not reveal the usefulness
of AI system in a clinical setting, it is a necessary step to proceed to the further improvement.
Thus, in my opinion, this study is valuable and meaningful. However, there are some concerns
that I would like you to address.
Major comments
Method
1. The experience of endoscopists who diagnosed the endoscopic images is quite different,
such as from 6 years to 26 years. It seems like the diagnostic accuracy of the endoscopist with
the experience of 6 years is inferior to that with 26 years. How about the agreement rate of
endoscopic diagnosis among 5 endoscopists? If possible, please calculate the kappa value among
5 endoscopists.
Result
2. Please discuss the reason why the number of participants in this study did not reach to the
required sample size? Additionally, if it could gather the required number of participants, are
there any possibility that the result of this study would be different?
3. The diagnostic sensitivity of endoscopists for ESCC in the current study is considerably low
compared to some previous studies, which is the most concerning point to me. For example,
according to the study by Dr. Ishihara (J Gastroenterol Hepatol, 2013), sensitivity and specificity
of NBI for differentiating mucosal high-grade from low-grade neoplasia were 85% and 79%,
respectively. Also, according to Dr. Minami (Gastroenterol Res Pract, 2012), those were 91.9%
and 76.7%, respectively. Please clarify the reason for this discrepancy.
4. Although this concern is almost similar to No.3, the sensitivity of endoscopists for the lesion
less than 10mm was only 46.2%. This result may not be easily acceptable for most endoscopists.
Could you please explain the reason for this significant low performance of endoscopists for
such lesions?
5. This study used various scopes such as GIF-H290Z, GIF-H260Z, GIF-Q240Z, GIF-XZ1200,
or GIF-EZ1500. If the quality of the images is related to the performance of AI system as authors
mentioned, did you consider using the same endoscope in this study? I mean why did you decide
to use some relatively old endoscope, such as GIF-H260Z and GIF-Q240Z?
Discussion
6. In page 13, authors mentioned that the failure to prove the non-inferiority of the AI system
was caused by the difference of the images used for validation and in the current clinical study. If
possible, I would like you to show some evidence that the performance of AI system is affected
by the quality of the images. Also, can you describe pixel count of compressed and uncompressed
images?
Minor comments
7. In page 3 line 2, the authors cited only 3, 4, 5. However, there are some reports showing the
effectiveness of definitive CRT treatment for stage # ESCC. Please consider to cite one of them.
How about the following report, Kato K, et al, Gastroenterology 2021?
8. Authors described “Furthermore, endoscopic diagnosis of ESCC is prone to interobserver
differences.” in page10 line7. Has this statement already been revealed by previous studies? If
so, please add such report to the reference.
9. Authors mentioned that “patients continuing antithrombotic drugs, for whom biopsy is
contraindicated according to the Japanese guidelines” in page4 line5. However, the Japanese
guidelines described that “Biopsy with antithrombotics should only be undertaken when
absolutely necessary, and caution should be exercised.” Please describe this point more precisely
based on the comment of the Japanese guideline.
10. In the preparation of the training dataset (page4), are the images used in this stage WLI or
NBI?
11. Although authors mentioned that “Our previous study showed an accuracy of 83% for the AI
system when the accuracy of endoscopic diagnosis was 78% 12.” in page7 line6, the meaning
of this sentence may be a bit unclear. Does the accuracy of endoscopic diagnosis mean the
diagnostic accuracy of endoscopists?
12. Please add the explanation for the abbreviation of Ce, Ut, Mt and Lt in the Table1 and Table3.
Moreover, in the latest Japanese guidelines, the description of “Ae” is not used and replaced
with “Jz”. Please consider changing it.
13. In the table1, noncancerous contains SIN, atypical epithelium, esophagitis and “others”
which shows the largest number (86). However, I am wondering what “others” contain. Could
you cite one or two examples of “others”?
14.Could you please add the pathological image of this lesion as Figure 3d.
15.  Authors mentioned “Second, the proportion of ESCC in the target lesions differs from that
in previous studies.” in page 14 line 9. Please consider adding a reference.
assistant