Title: Convolutional neural network-based system for endocytoscopic diagnosis of early gastric
cancer
Version: 2 Date: 28 Nov 2021
Reviewer's report:
The authors established a CNN based system in evaluating early gastric cancer from
endocystoscopy. The CNN based system was further tested in an independent set of images
evaluted by both CNN algorithm and endoscopist and showed good accuracy. However, the
difference between endoscopist and CNN program was small and the accuracy between lesions
and pateints are different when raised some questions in clinical application of this CNN
algorithm. Furthermore, the conlusion wasn't supported by the reported data as endoscopist
performance was similiar with the AI system. The issues are as following:
1. What is the gold standard of the imaging per image in this study? As we know, some non-
cancerous part may exist in a cancerous lesion. When doing per-image evaluation, some images
taken from the cancerous lesion may not really reflect the charcteristics of cancer due to imaging
the non-cancerous part in fact. This may lead to the huge difference of per image and per lesion
evaluation in both endoscopist and CNN protocol.
2. Following the question above, if the images were confirmed by endoscopic in the training
set which means 100% accuracy, why the ability of endoscopist deteriorated in evaluating the
training set photos? Is there any photo selection protocol in establishing the training set? Was
the training set evaluated by another expert which proved 100% accurate? If not, the system
seems to be established on a 90% accuracy based on the standard taken, and may cause false
negative in clinical diagnosis, esp. for doctors who are not familiar with endocytoscopy in early
gastric cancer diagnosis.
3. What is the examination protocol in doing endoscopy? Are the two experienced endoscopists
who took the photos are those who did the evaluation in training set or validation set? Are
the photos of non-cancerous part also taken? Were the images in non-cancerous part of the
cancer patient also included in the non-cancerous data set. Many early gastric cancer patients
had also multiple parts of intestinal metaplasia which may also be misdiagnosed with cancer
during endoscopic evaluaiton. Can this system help to diagnose the cancerous part and non-
cancerous part in the same patient?
4. In the evaluation of validation set by endoscopists, did the authors calucalated the data of 2
endoscopists separately? If so , how to calculate the sensitivity, specificity, ppv, npv? Did the
authors did an average analysis only?
5. In the diagnositic performances of CNN and endosocpist in per-image anlaysis, the sensitivity
and negative predictive value were both higher in endoscopists compared to CNN. In per-lesion
analysis, the accuracy, sensitiovity, and NPV were also higher in endoscopists. This was unlikely
to lead the conclusion where stated that CNN outperforms endoscopists. Is there any statistical
analysis comparing the performance of CNN and endoscopists?
6. In discussion, the authors menioned about the staining quality. Maybe addition of the staining
quality evaluation in the authors AI algorithm can really improve the diagnostic performance
of their AI system.
Title: Convolutional neural network-based system for endocytoscopic diagnosis of early gastric
cancer
Version: 2 Date: 28 Nov 2021
Reviewer's report:
The authors established a CNN based system in evaluating early gastric cancer from
endocystoscopy. The CNN based system was further tested in an independent set of images
evaluted by both CNN algorithm and endoscopist and showed good accuracy. However, the
difference between endoscopist and CNN program was small and the accuracy between lesions
and pateints are different when raised some questions in clinical application of this CNN
algorithm. Furthermore, the conlusion wasn't supported by the reported data as endoscopist
performance was similiar with the AI system. The issues are as following:
1. What is the gold standard of the imaging per image in this study? As we know, some non-
cancerous part may exist in a cancerous lesion. When doing per-image evaluation, some images
taken from the cancerous lesion may not really reflect the charcteristics of cancer due to imaging
the non-cancerous part in fact. This may lead to the huge difference of per image and per lesion
evaluation in both endoscopist and CNN protocol.
2. Following the question above, if the images were confirmed by endoscopic in the training
set which means 100% accuracy, why the ability of endoscopist deteriorated in evaluating the
training set photos? Is there any photo selection protocol in establishing the training set? Was
the training set evaluated by another expert which proved 100% accurate? If not, the system
seems to be established on a 90% accuracy based on the standard taken, and may cause false
negative in clinical diagnosis, esp. for doctors who are not familiar with endocytoscopy in early
gastric cancer diagnosis.
3. What is the examination protocol in doing endoscopy? Are the two experienced endoscopists
who took the photos are those who did the evaluation in training set or validation set? Are
the photos of non-cancerous part also taken? Were the images in non-cancerous part of the
cancer patient also included in the non-cancerous data set. Many early gastric cancer patients
had also multiple parts of intestinal metaplasia which may also be misdiagnosed with cancer
during endoscopic evaluaiton. Can this system help to diagnose the cancerous part and non-
cancerous part in the same patient?
4. In the evaluation of validation set by endoscopists, did the authors calucalated the data of 2
endoscopists separately? If so , how to calculate the sensitivity, specificity, ppv, npv? Did the
authors did an average analysis only?
5. In the diagnositic performances of CNN and endosocpist in per-image anlaysis, the sensitivity
and negative predictive value were both higher in endoscopists compared to CNN. In per-lesion
analysis, the accuracy, sensitiovity, and NPV were also higher in endoscopists. This was unlikely
to lead the conclusion where stated that CNN outperforms endoscopists. Is there any statistical
analysis comparing the performance of CNN and endoscopists?
6. In discussion, the authors menioned about the staining quality. Maybe addition of the staining
quality evaluation in the authors AI algorithm can really improve the diagnostic performance
of their AI system.
Please review this paper.
user
Title: Convolutional neural network-based system for endocytoscopic diagnosis of early gastric
cancer
Version: 2 Date: 28 Nov 2021
Reviewer's report:
The authors established a CNN based system in evaluating early gastric cancer from
endocystoscopy. The CNN based system was further tested in an independent set of images
evaluted by both CNN algorithm and endoscopist and showed good accuracy. However, the
difference between endoscopist and CNN program was small and the accuracy between lesions
and pateints are different when raised some questions in clinical application of this CNN
algorithm. Furthermore, the conlusion wasn't supported by the reported data as endoscopist
performance was similiar with the AI system. The issues are as following:
1. What is the gold standard of the imaging per image in this study? As we know, some non-
cancerous part may exist in a cancerous lesion. When doing per-image evaluation, some images
taken from the cancerous lesion may not really reflect the charcteristics of cancer due to imaging
the non-cancerous part in fact. This may lead to the huge difference of per image and per lesion
evaluation in both endoscopist and CNN protocol.
2. Following the question above, if the images were confirmed by endoscopic in the training
set which means 100% accuracy, why the ability of endoscopist deteriorated in evaluating the
training set photos? Is there any photo selection protocol in establishing the training set? Was
the training set evaluated by another expert which proved 100% accurate? If not, the system
seems to be established on a 90% accuracy based on the standard taken, and may cause false
negative in clinical diagnosis, esp. for doctors who are not familiar with endocytoscopy in early
gastric cancer diagnosis.
3. What is the examination protocol in doing endoscopy? Are the two experienced endoscopists
who took the photos are those who did the evaluation in training set or validation set? Are
the photos of non-cancerous part also taken? Were the images in non-cancerous part of the
cancer patient also included in the non-cancerous data set. Many early gastric cancer patients
had also multiple parts of intestinal metaplasia which may also be misdiagnosed with cancer
during endoscopic evaluaiton. Can this system help to diagnose the cancerous part and non-
cancerous part in the same patient?
4. In the evaluation of validation set by endoscopists, did the authors calucalated the data of 2
endoscopists separately? If so , how to calculate the sensitivity, specificity, ppv, npv? Did the
authors did an average analysis only?
5. In the diagnositic performances of CNN and endosocpist in per-image anlaysis, the sensitivity
and negative predictive value were both higher in endoscopists compared to CNN. In per-lesion
analysis, the accuracy, sensitiovity, and NPV were also higher in endoscopists. This was unlikely
to lead the conclusion where stated that CNN outperforms endoscopists. Is there any statistical
analysis comparing the performance of CNN and endoscopists?
6. In discussion, the authors menioned about the staining quality. Maybe addition of the staining
quality evaluation in the authors AI algorithm can really improve the diagnostic performance
of their AI system.
assistant