Reviewer’s report  
Title: Evidence of surgical outcomes fluctuates over time: results from a cumulative meta -
analysis  of laparoscopic versus open appendec tomy for acute appendicitis  
 
Version: 1 Date: 27 Dec 2015  
 
Reviewer: Luit Penninga  
Reviewer’s report:  
The authors have re -submitted the manuscript. The manuscript has been improved, but to my 
opinion there  are still some issues that need to be addressed.  
 
1) the authors write in line 23 -24 p8: The effect size began to favour laparoscopic 
procedures after 2001, leading to an insignificant result with an overall cumulative odds 
ratio of 11.3226 (95% confiden ce interval: 00.8490 -2.101.76).  
 
I think this sentence might be confusing, and somehow suggests that laparoscopic 
appendectomy is better than open appendectomy, while there is no statistically significant 
difference. The authors might write: The effect size  in favour of open procedures began 
to disappear after 2001, leading to an insignificant result with an overall cumulative odds 
ratio of 11.3226 (95% confidence interval: 00.8490 -2.101.76) when laparoscopic 
appendectomy was compared with open appendectomy.  
 
2) The authors write in line 36 -39 p11 'This argument suggests that the efficacy of a 
pharmaceutical treatment will not change once it is statistically significant established, 
which seems reasonable.'  
 
This is a topic raised by reviewer #3, comment 4. Al though the authors call this 
reasonable, this is far too simplistic, and not true. There are numerous examples of 
pharmaceutical treatments which are found to have significant results which later 
disappear when well -powered well -designed trials with suffic ient numbers of outcome 
and patients appear.  
The authors may write that they look at surgical trials because they are subject to 
numerous factors, but they cannot write that the efficacy of a pharmaceutical treatment 
will not change once a significant res ult is found.  
 
3) Figure 2 and figure 5: In the forest plots for intra -abdominal abscesses figure 2 and 
wound infection figure 5 the studies are listed in alphabetical order according to author 
name, it would be much more informative when the studies were listed in chronological 
order (starting with oldest studies).  
 
4) The authors have not replied sufficiently to reviewer #3 comment 3, This article  Meta -
analysis of randomised trials of laparoscopic versus open surgery for acute appendicitis; 
has firm evidence been reached;  Penninga et al: J Gastrointestinal Surg 201 4) deals with 
exactly the same topic; when can we stop performing randomised trials on 
appendectomy? The authors should refer to it and comment it in their discussion.  
 
5) RevMan cannot handle trials with zero events in both arms. Furthermore, when no ev ent 
occurs in one arm of the trial RevMAn puts 0.5 in the zero -event arm. Intra -abdominal 
abscesses is a seldom outcome, hence quite some trials have zero -events in either 1 or 2 
arms of the trial. The authors write that they add 0.5 to correct for zero -event in a trial 
arm. This a good idea, but might still be problematic, fx if you add  0.5 event in 20 zero -
event trials, you have created 10 events of intra -abdominal abcess. Events which never 
occurred. These artificially created events might influence res ults. Hence, it has been 
suggested to use another correction factor fx 0.01 By using this you only have created 0.2 
event when you apply this correction factor to 20 zero -event trials which seems more 
reasonable (Robustness assessments are needed to reduce  bias in meta -analyses that 
include zero -event randomized trials. Keus et al, Am J Gastroenter 2009;104(3):546 -51. 
doi: 10.1038/ajg.2008.22.) The authors should perform a sensitivity analysis for intra -
abdominal abscesses using a correction factor of 0.01 to test for robustness of their 
findings.  
 
Are the methods appropriate and well described?  
If not, please specify what is required in your comments to the authors.  
 
Yes 
 
Does the work include the necessary controls?  
If not, please spe cify which controls are required in your comments to the authors.  
 
Yes 
 
Are the conclusions drawn adequately supported by the data shown?  
If not, please explain in your comments to the authors.  
 
Yes 
 
Are you able to assess any statistics in the manuscript o r would you recommend an 
additional statistical review?  
If an additional statistical review is recommended, please specify what aspects require further 
assessment in your comments to the editors.  
 
I am able to assess the statistics  
 
Quality of written Engl ish 
Please indicate the quality of language in the manuscript:  
 
Needs some language corrections before being published  
 
Declaration of competing interests  
Please complete a declaration of competing interests, consideri ng the following questions:  
 
1. Have you in the past five years received reimbursements, fees, funding, or salary from an 
organisation that may in any way gain or lose financially from the publication of this 
manuscript, e ither now or in the future?  
 
2. Do you hold any stocks or shares in an organis ation that may in any way gain or lose 
financially from the publication of this manuscript, e ither now or in the future?  
 
3. Do you hold or are you currently applying for any patents relating to the  content of the 
manuscript?  
 
4. Have you received reimbursements , fees, funding, or salary from an organization that 
holds or has applied for patents relating to the  content of the manuscript?  
 
5. Do you have any other fin ancial competing interests?  
 
6. Do you have any non -financial competing interests  in relation to this pa per? 
 
If you can answer no to all of the above, write 'I declare that I have no competing interests' 
below. If your reply is yes to any, please give details below.  
 
I declare that I have no competing interests  
 
I agree to the open peer review policy of the  journal. I understand that my name will be included 
on my report to the authors and, if the manuscript is accepted for publication, my named report 
including any attachments I upload will be posted on the website along with the authors' 
responses. I agree  for my report to be made available under an Open Access Creative Commons 
CC-BY license ( http://creativecommons.org/licenses/by/4.0/ ). I understand that any comments 
which I do not wish to be inc luded in my named report can be included as confidential comments 
to the editors, which will not be published.  
 
I agree to the open peer review policy of the journal  
Reviewer’s report  
Title: Evidence of surgical outcomes fluctuates over time: results from a cumulative meta -
analysis  of laparoscopic versus open appendec tomy for acute appendicitis  
 
Version: 1 Date: 27 Dec 2015  
 
Reviewer: Luit Penninga  
Reviewer’s report:  
The authors have re -submitted the manuscript. The manuscript has been improved, but to my 
opinion there  are still some issues that need to be addressed.  
 
1) the authors write in line 23 -24 p8: The effect size began to favour laparoscopic 
procedures after 2001, leading to an insignificant result with an overall cumulative odds 
ratio of 11.3226 (95% confiden ce interval: 00.8490 -2.101.76).  
 
I think this sentence might be confusing, and somehow suggests that laparoscopic 
appendectomy is better than open appendectomy, while there is no statistically significant 
difference. The authors might write: The effect size  in favour of open procedures began 
to disappear after 2001, leading to an insignificant result with an overall cumulative odds 
ratio of 11.3226 (95% confidence interval: 00.8490 -2.101.76) when laparoscopic 
appendectomy was compared with open appendectomy.  
 
2) The authors write in line 36 -39 p11 'This argument suggests that the efficacy of a 
pharmaceutical treatment will not change once it is statistically significant established, 
which seems reasonable.'  
 
This is a topic raised by reviewer #3, comment 4. Al though the authors call this 
reasonable, this is far too simplistic, and not true. There are numerous examples of 
pharmaceutical treatments which are found to have significant results which later 
disappear when well -powered well -designed trials with suffic ient numbers of outcome 
and patients appear.  
The authors may write that they look at surgical trials because they are subject to 
numerous factors, but they cannot write that the efficacy of a pharmaceutical treatment 
will not change once a significant res ult is found.  
 
3) Figure 2 and figure 5: In the forest plots for intra -abdominal abscesses figure 2 and 
wound infection figure 5 the studies are listed in alphabetical order according to author 
name, it would be much more informative when the studies were listed in chronological 
order (starting with oldest studies).  
 
4) The authors have not replied sufficiently to reviewer #3 comment 3, This article  Meta -
analysis of randomised trials of laparoscopic versus open surgery for acute appendicitis; 
has firm evidence been reached;  Penninga et al: J Gastrointestinal Surg 201 4) deals with 
exactly the same topic; when can we stop performing randomised trials on 
appendectomy? The authors should refer to it and comment it in their discussion.  
 
5) RevMan cannot handle trials with zero events in both arms. Furthermore, when no ev ent 
occurs in one arm of the trial RevMAn puts 0.5 in the zero -event arm. Intra -abdominal 
abscesses is a seldom outcome, hence quite some trials have zero -events in either 1 or 2 
arms of the trial. The authors write that they add 0.5 to correct for zero -event in a trial 
arm. This a good idea, but might still be problematic, fx if you add  0.5 event in 20 zero -
event trials, you have created 10 events of intra -abdominal abcess. Events which never 
occurred. These artificially created events might influence res ults. Hence, it has been 
suggested to use another correction factor fx 0.01 By using this you only have created 0.2 
event when you apply this correction factor to 20 zero -event trials which seems more 
reasonable (Robustness assessments are needed to reduce  bias in meta -analyses that 
include zero -event randomized trials. Keus et al, Am J Gastroenter 2009;104(3):546 -51. 
doi: 10.1038/ajg.2008.22.) The authors should perform a sensitivity analysis for intra -
abdominal abscesses using a correction factor of 0.01 to test for robustness of their 
findings.  
 
Are the methods appropriate and well described?  
If not, please specify what is required in your comments to the authors.  
 
Yes 
 
Does the work include the necessary controls?  
If not, please spe cify which controls are required in your comments to the authors.  
 
Yes 
 
Are the conclusions drawn adequately supported by the data shown?  
If not, please explain in your comments to the authors.  
 
Yes 
 
Are you able to assess any statistics in the manuscript o r would you recommend an 
additional statistical review?  
If an additional statistical review is recommended, please specify what aspects require further 
assessment in your comments to the editors.  
 
I am able to assess the statistics  
 
Quality of written Engl ish 
Please indicate the quality of language in the manuscript:  
 
Needs some language corrections before being published  
 
Declaration of competing interests  
Please complete a declaration of competing interests, consideri ng the following questions:  
 
1. Have you in the past five years received reimbursements, fees, funding, or salary from an 
organisation that may in any way gain or lose financially from the publication of this 
manuscript, e ither now or in the future?  
 
2. Do you hold any stocks or shares in an organis ation that may in any way gain or lose 
financially from the publication of this manuscript, e ither now or in the future?  
 
3. Do you hold or are you currently applying for any patents relating to the  content of the 
manuscript?  
 
4. Have you received reimbursements , fees, funding, or salary from an organization that 
holds or has applied for patents relating to the  content of the manuscript?  
 
5. Do you have any other fin ancial competing interests?  
 
6. Do you have any non -financial competing interests  in relation to this pa per? 
 
If you can answer no to all of the above, write 'I declare that I have no competing interests' 
below. If your reply is yes to any, please give details below.  
 
I declare that I have no competing interests  
 
I agree to the open peer review policy of the  journal. I understand that my name will be included 
on my report to the authors and, if the manuscript is accepted for publication, my named report 
including any attachments I upload will be posted on the website along with the authors' 
responses. I agree  for my report to be made available under an Open Access Creative Commons 
CC-BY license ( http://creativecommons.org/licenses/by/4.0/ ). I understand that any comments 
which I do not wish to be inc luded in my named report can be included as confidential comments 
to the editors, which will not be published.  
 
I agree to the open peer review policy of the journal  
Please review this paper.
user
Reviewer’s report  
Title: Evidence of surgical outcomes fluctuates over time: results from a cumulative meta -
analysis  of laparoscopic versus open appendec tomy for acute appendicitis  
 
Version: 1 Date: 27 Dec 2015  
 
Reviewer: Luit Penninga  
Reviewer’s report:  
The authors have re -submitted the manuscript. The manuscript has been improved, but to my 
opinion there  are still some issues that need to be addressed.  
 
1) the authors write in line 23 -24 p8: The effect size began to favour laparoscopic 
procedures after 2001, leading to an insignificant result with an overall cumulative odds 
ratio of 11.3226 (95% confiden ce interval: 00.8490 -2.101.76).  
 
I think this sentence might be confusing, and somehow suggests that laparoscopic 
appendectomy is better than open appendectomy, while there is no statistically significant 
difference. The authors might write: The effect size  in favour of open procedures began 
to disappear after 2001, leading to an insignificant result with an overall cumulative odds 
ratio of 11.3226 (95% confidence interval: 00.8490 -2.101.76) when laparoscopic 
appendectomy was compared with open appendectomy.  
 
2) The authors write in line 36 -39 p11 'This argument suggests that the efficacy of a 
pharmaceutical treatment will not change once it is statistically significant established, 
which seems reasonable.'  
 
This is a topic raised by reviewer #3, comment 4. Al though the authors call this 
reasonable, this is far too simplistic, and not true. There are numerous examples of 
pharmaceutical treatments which are found to have significant results which later 
disappear when well -powered well -designed trials with suffic ient numbers of outcome 
and patients appear.  
The authors may write that they look at surgical trials because they are subject to 
numerous factors, but they cannot write that the efficacy of a pharmaceutical treatment 
will not change once a significant res ult is found.  
 
3) Figure 2 and figure 5: In the forest plots for intra -abdominal abscesses figure 2 and 
wound infection figure 5 the studies are listed in alphabetical order according to author 
name, it would be much more informative when the studies were listed in chronological 
order (starting with oldest studies).  
 
4) The authors have not replied sufficiently to reviewer #3 comment 3, This article  Meta -
analysis of randomised trials of laparoscopic versus open surgery for acute appendicitis; 
has firm evidence been reached;  Penninga et al: J Gastrointestinal Surg 201 4) deals with 
exactly the same topic; when can we stop performing randomised trials on 
appendectomy? The authors should refer to it and comment it in their discussion.  
 
5) RevMan cannot handle trials with zero events in both arms. Furthermore, when no ev ent 
occurs in one arm of the trial RevMAn puts 0.5 in the zero -event arm. Intra -abdominal 
abscesses is a seldom outcome, hence quite some trials have zero -events in either 1 or 2 
arms of the trial. The authors write that they add 0.5 to correct for zero -event in a trial 
arm. This a good idea, but might still be problematic, fx if you add  0.5 event in 20 zero -
event trials, you have created 10 events of intra -abdominal abcess. Events which never 
occurred. These artificially created events might influence res ults. Hence, it has been 
suggested to use another correction factor fx 0.01 By using this you only have created 0.2 
event when you apply this correction factor to 20 zero -event trials which seems more 
reasonable (Robustness assessments are needed to reduce  bias in meta -analyses that 
include zero -event randomized trials. Keus et al, Am J Gastroenter 2009;104(3):546 -51. 
doi: 10.1038/ajg.2008.22.) The authors should perform a sensitivity analysis for intra -
abdominal abscesses using a correction factor of 0.01 to test for robustness of their 
findings.  
 
Are the methods appropriate and well described?  
If not, please specify what is required in your comments to the authors.  
 
Yes 
 
Does the work include the necessary controls?  
If not, please spe cify which controls are required in your comments to the authors.  
 
Yes 
 
Are the conclusions drawn adequately supported by the data shown?  
If not, please explain in your comments to the authors.  
 
Yes 
 
Are you able to assess any statistics in the manuscript o r would you recommend an 
additional statistical review?  
If an additional statistical review is recommended, please specify what aspects require further 
assessment in your comments to the editors.  
 
I am able to assess the statistics  
 
Quality of written Engl ish 
Please indicate the quality of language in the manuscript:  
 
Needs some language corrections before being published  
 
Declaration of competing interests  
Please complete a declaration of competing interests, consideri ng the following questions:  
 
1. Have you in the past five years received reimbursements, fees, funding, or salary from an 
organisation that may in any way gain or lose financially from the publication of this 
manuscript, e ither now or in the future?  
 
2. Do you hold any stocks or shares in an organis ation that may in any way gain or lose 
financially from the publication of this manuscript, e ither now or in the future?  
 
3. Do you hold or are you currently applying for any patents relating to the  content of the 
manuscript?  
 
4. Have you received reimbursements , fees, funding, or salary from an organization that 
holds or has applied for patents relating to the  content of the manuscript?  
 
5. Do you have any other fin ancial competing interests?  
 
6. Do you have any non -financial competing interests  in relation to this pa per? 
 
If you can answer no to all of the above, write 'I declare that I have no competing interests' 
below. If your reply is yes to any, please give details below.  
 
I declare that I have no competing interests  
 
I agree to the open peer review policy of the  journal. I understand that my name will be included 
on my report to the authors and, if the manuscript is accepted for publication, my named report 
including any attachments I upload will be posted on the website along with the authors' 
responses. I agree  for my report to be made available under an Open Access Creative Commons 
CC-BY license ( http://creativecommons.org/licenses/by/4.0/ ). I understand that any comments 
which I do not wish to be inc luded in my named report can be included as confidential comments 
to the editors, which will not be published.  
 
I agree to the open peer review policy of the journal  
assistant