Title: Accuracy of ChatGPT3.5 in Answering Clinical Questions on Guidelines for Severe
Acute Pancreatitis
Version: 2 Date: 22 Apr 2024
Reviewer's report:
Evaluation of the Accuracy of the ChatGPT3.5 in Answering Clinical Questions in Guidelines
for Severe Acute Pancreatitis
Consider Changing the title to
Accuracy of ChatGPT3.5 in Answering Clinical Questions on Guidelines for Severe Acute
Pancreatitis
This observational study compared the accuracy of chatGPT3.5 in English and Chinese in
answering clinical questions using the 2019 guidelines for severe acute pancreatitis. The authors
formulated 34 short answer questions and 15 true/false questions to test the accuracy of AI,
potentially aiding clinicians in making accurate decisions whilst treating a patient.
What was the rationale for choosing Severe Acute Pancreatitis?
To better assess chatGPT3.5's ability to clarify the guidelines, 15 true/false questions regarding
the recommendations were created.  ChatGPT-3.5 was asked each question in Chinese and
English.
It would be useful to have a table with all the questions as asked and the researchers' responses
to each question, along with their executive decision to categorize some answers as positive or
negative. This would help the reader understand what was deemed correct and which ones were
deemed incorrect, as well as how the determination was made.
Each reviewer independently evaluated reproducibility by comparing the two responses to each
question. In cases of disagreement between the two reviewers, a third senior expert from the
Department of Critical Care Medicine was consulted to resolve the issue. What percentage
required a third reviewer?
ChatGPT demonstrated a higher accuracy rate for short-answer questions (76%) in English
compared to right-or-wrong questions (60%, P value:0.405) (Figure B). In Chinese, no
significant difference in accuracy was found between the right-or-wrong questions (60%) and
the short-answer questions (59%, P value:0.938) (Figure C).  Can they offer an explanation as
to why there was no difference in Chinese?
Why did the authors analyze both in English and in Chinese?
When answering a true/false question, the response is considered correct only if it is an exact
match. If the answer is 'maybe' or 'can be considered', it is considered a 'yes' response. Responses
such as 'Unavailable', 'Difference of opinion', or 'I'm not a doctor' are considered incorrect.
Please clarify what the authors mean by a ‘yes’ response. I feel that to accurately understand, it
would be useful to have a third answer for true or false questions- such as ‘not known’ or ‘don’t
know’ responses than to club it with true or false based on human intelligence. This adds to the
complexity of determining the accuracy of generative AI in this context. The more accurate way
would be to have a third response as described and present the data to the reader.
Title: Accuracy of ChatGPT3.5 in Answering Clinical Questions on Guidelines for Severe
Acute Pancreatitis
Version: 2 Date: 22 Apr 2024
Reviewer's report:
Evaluation of the Accuracy of the ChatGPT3.5 in Answering Clinical Questions in Guidelines
for Severe Acute Pancreatitis
Consider Changing the title to
Accuracy of ChatGPT3.5 in Answering Clinical Questions on Guidelines for Severe Acute
Pancreatitis
This observational study compared the accuracy of chatGPT3.5 in English and Chinese in
answering clinical questions using the 2019 guidelines for severe acute pancreatitis. The authors
formulated 34 short answer questions and 15 true/false questions to test the accuracy of AI,
potentially aiding clinicians in making accurate decisions whilst treating a patient.
What was the rationale for choosing Severe Acute Pancreatitis?
To better assess chatGPT3.5's ability to clarify the guidelines, 15 true/false questions regarding
the recommendations were created.  ChatGPT-3.5 was asked each question in Chinese and
English.
It would be useful to have a table with all the questions as asked and the researchers' responses
to each question, along with their executive decision to categorize some answers as positive or
negative. This would help the reader understand what was deemed correct and which ones were
deemed incorrect, as well as how the determination was made.
Each reviewer independently evaluated reproducibility by comparing the two responses to each
question. In cases of disagreement between the two reviewers, a third senior expert from the
Department of Critical Care Medicine was consulted to resolve the issue. What percentage
required a third reviewer?
ChatGPT demonstrated a higher accuracy rate for short-answer questions (76%) in English
compared to right-or-wrong questions (60%, P value:0.405) (Figure B). In Chinese, no
significant difference in accuracy was found between the right-or-wrong questions (60%) and
the short-answer questions (59%, P value:0.938) (Figure C).  Can they offer an explanation as
to why there was no difference in Chinese?
Why did the authors analyze both in English and in Chinese?
When answering a true/false question, the response is considered correct only if it is an exact
match. If the answer is 'maybe' or 'can be considered', it is considered a 'yes' response. Responses
such as 'Unavailable', 'Difference of opinion', or 'I'm not a doctor' are considered incorrect.
Please clarify what the authors mean by a ‘yes’ response. I feel that to accurately understand, it
would be useful to have a third answer for true or false questions- such as ‘not known’ or ‘don’t
know’ responses than to club it with true or false based on human intelligence. This adds to the
complexity of determining the accuracy of generative AI in this context. The more accurate way
would be to have a third response as described and present the data to the reader.
Please review this paper.
user
Title: Accuracy of ChatGPT3.5 in Answering Clinical Questions on Guidelines for Severe
Acute Pancreatitis
Version: 2 Date: 22 Apr 2024
Reviewer's report:
Evaluation of the Accuracy of the ChatGPT3.5 in Answering Clinical Questions in Guidelines
for Severe Acute Pancreatitis
Consider Changing the title to
Accuracy of ChatGPT3.5 in Answering Clinical Questions on Guidelines for Severe Acute
Pancreatitis
This observational study compared the accuracy of chatGPT3.5 in English and Chinese in
answering clinical questions using the 2019 guidelines for severe acute pancreatitis. The authors
formulated 34 short answer questions and 15 true/false questions to test the accuracy of AI,
potentially aiding clinicians in making accurate decisions whilst treating a patient.
What was the rationale for choosing Severe Acute Pancreatitis?
To better assess chatGPT3.5's ability to clarify the guidelines, 15 true/false questions regarding
the recommendations were created.  ChatGPT-3.5 was asked each question in Chinese and
English.
It would be useful to have a table with all the questions as asked and the researchers' responses
to each question, along with their executive decision to categorize some answers as positive or
negative. This would help the reader understand what was deemed correct and which ones were
deemed incorrect, as well as how the determination was made.
Each reviewer independently evaluated reproducibility by comparing the two responses to each
question. In cases of disagreement between the two reviewers, a third senior expert from the
Department of Critical Care Medicine was consulted to resolve the issue. What percentage
required a third reviewer?
ChatGPT demonstrated a higher accuracy rate for short-answer questions (76%) in English
compared to right-or-wrong questions (60%, P value:0.405) (Figure B). In Chinese, no
significant difference in accuracy was found between the right-or-wrong questions (60%) and
the short-answer questions (59%, P value:0.938) (Figure C).  Can they offer an explanation as
to why there was no difference in Chinese?
Why did the authors analyze both in English and in Chinese?
When answering a true/false question, the response is considered correct only if it is an exact
match. If the answer is 'maybe' or 'can be considered', it is considered a 'yes' response. Responses
such as 'Unavailable', 'Difference of opinion', or 'I'm not a doctor' are considered incorrect.
Please clarify what the authors mean by a ‘yes’ response. I feel that to accurately understand, it
would be useful to have a third answer for true or false questions- such as ‘not known’ or ‘don’t
know’ responses than to club it with true or false based on human intelligence. This adds to the
complexity of determining the accuracy of generative AI in this context. The more accurate way
would be to have a third response as described and present the data to the reader.
assistant