Reviewer’s report  
Title: Evidence of surgical outcomes fluctuates over time: results from a cumulative meta -
analysis  of laparoscopic versus open appendec tomy for acute appendicitis  
 
Version: 0 Date: 17 Nov 2015  
 
Reviewer: Dimitrios Mavridis  
Reviewer’s report:  
The authors employed a cumulative meta -analysis of laparoscopic versus open appendectomy for 
acute appendicitis. T hey claim that 'established' evidence may change with time and they show 
that in their cumulative meta -analysis. I am a big proponent of cumulative meta -analysis and I 
believe it is a useful tool. I have some statistical concerns that I explain in detail l ater on. My 
biggest concern however, is the use of the term 'established'. The authors do not provide any 
guidance on when evidence is considered established. A meta -analysis, just like any trial, would 
ideally stop either for efficacy or for futility but these are determined based on sequential 
analysis methods and power considerations. For instance, in the first outcome (abdominal 
abscess), although I have not employed any analysis, it is clear that uncertainty is too large for 
any power analysis to consi der that evidence is established and we should stop for efficacy. 
Hence, it is not surprising that evidence changes direction because it is not established. Hence, 
the authors overstate the change of direction in the cumulative meta -analysis. There is a lo t of 
work done about when evidence is considered established and the authors may consider apply 
such methods in their example (or in another)  
 
Roloff V, Higgins JPT, Sutton AJ: Planning future studies based on the conditional power of a 
meta -analysis. Stat  Med 2013, 32:11 -24. 
 
Wetterslev J, Thorlund K, Brok J, et al. Trial sequential analysis may establish when firm 
evidence is reached in cumulative meta -analysis Journal of clinical epidemiology 2008: 61; 64 -
75 
 
I believe results will not change when there is established (by a statistical model) evidence and 
there is the danger that the paper may have to change direction and suggest that we should not 
keep on conducting trials when there is established evidence. This has been documented in the 
literature but  probably not enough because the problem is not eliminated.  
 
I have other comments, some of which I consider important  
 
Title: "confidence in the best evidence wavers"  
 
I am not a native English speaker and I may be wrong but I cannot see what the author s mean or 
are trying to achieve with this sentence  
 
Abstract, background: 'expected to yield definite results'  
 
Meta analysis most likely is going to give more powerful results but there is not an a -priori 
guarantee that the results will be definite  
 
Abst ract, background: " However, even…has not been previously reported"  
 
This is a contradictory sentence, if the evidence is established it can not change. By reading the 
paper I understood what the authors mean. However, established evidence is proven by 
sequential analyses and power considerations that are not mentioned in the paper. Otherwise, it is 
just a trend that if it has not been established it is not unlikely to change. The authors are correct 
that at any point, future trials may differ considerably b y those published and the summary 
estimate may change direction but be careful that evidence is not established  
 
Abstract, methods "we included a broad…effect size''  
 
I do not understand what the authors mean by 'randomization was stated' since it is mentioned 
early that only RCTs are considered. I also do not understand how this is related to the aim of the 
study being to detect a trend and not the accurate effect size . 
 
Page 3, lines: 13 -17: poor explanation of cumulative meta -analysis  
 
I have several concerns regarding the statistical analysis  
 
* It is strongly recommended not to use the I^2 to decide between a fixed effect and a 
random effects model. Choice should b e driven by clinical considerations and not from a 
test or an index value. I would suggest the authors use both models and if results are 
similar random effects results should be preferred whereas if not we should consider the 
possibility of small study ef fects  
 
* I would suggest checking, at least visually, for small study effects (probably not for the 
first outcome due t rarity of events). Cumulative meta -analysis can be used for showing if 
small study effects operate on these studies  
 
* Peto odds ratio is rarely used. The default option for rare events is a Mantel -Haenszel 
model with a 0.5 correction for zero cells. Also the Peto odds ratio does not require a 
correction for zero cells as you did.  
 
* The authors use weighted mean difference, also known as mea n differences, but it seems 
that different scales are used and standardized mean differences should be used instead  
 
* More information about the aforementioned comments can be found in the ninth chapter 
of the Cochrane handbook (Deeks JJ, Higgins JPT, Altm an DG (editors). Chapter 9: 
Analysing data and undertaking meta -analyses. In: Higgins JPT, Green S (editors). 
Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated 
March 2011). The Cochrane Collaboration, 2011. Available from www .cochrane -
handbook.org.)  
 
* The heterogeneity in Figure 3 is enormous, if standardized mean differences do not 
reduce heterogeneity we should start thinking if it is meaningful to pool these studies.  
 
* Although there are a large number of trials, the rarit y of events for intra -abdominal 
abscess complicates the analysis. Another example may be more suitable  
 
* I believe that results for abdominal abscess are overstated. Because of the small number 
of events uncertainty is large.  The 37th trial (Pedersen stud y I assume) is large with a 
respectable weight in the meta -analysis that is why it affects heavily the results. The 
width of the summary estimate is large, hence there is no reason to stop trials because it is 
not unlikely for the direction of the effect t o change.  
 
* The authors state that relative effectiveness between a pair of treatments may change 
because technology is different e.t.c. This is an important point but probably a more 
narrow review is needed if there are important changes in technology e. g. exclude old 
trials with the old technology e.t.c.  
 
Are the methods appropriate and well described?  
If not, please specify what is required in your comments to the authors.  
 
No 
 
Does the work include the necessary controls?  
If not, please spe cify which controls are required in your comments to the authors.  
 
Unable to assess  
 
Are the conclusions drawn adequately supported by the data shown?  
If not, please explain in your comments to the authors.  
 
No 
 
Are you able to assess any statistics in the manuscript o r would you recommend an 
additional statistical review?  
If an additional statistical review is recommended, please specify what aspects require further 
assessment in your comments to the editors.  
 
I am able to assess the statistics  
 
Quality of written Engl ish 
Please indicate the quality of language in the manuscript:  
 
Acceptable  
 
Declaration of competing interests  
Please complete a declaration of competing interests, consideri ng the following questions:  
 
1. Have you in the past five years received reimbursements, fees, funding, or salary from an 
organisation that may in any way gain or lose financially from the publication of this 
manuscript, e ither now or in the future?  
 
2. Do you hold any stocks or shares in an organis ation that may in any way gain or lose 
financially from the publication of this manuscript, e ither now or in the future?  
 
3. Do you hold or are you currently applying for any patents relating to the  content of the 
manuscript?  
 
4. Have you received reimbursements , fees, funding, or salary from an organization that 
holds or has applied for patents relating to the  content of the manuscript?  
 
5. Do you have any other fin ancial competing interests?  
 
6. Do you have any non -financial competing interests  in relation to this pa per? 
 
If you can answer no to all of the above, write 'I declare that I have no competing interests' 
below. If your reply is yes to any, please give details below.  
 
I declare that I have no competing interests  
 
I agree to the open peer review policy of the  journal. I understand that my name will be included 
on my report to the authors and, if the manuscript is accepted for publication, my named report 
including any attachments I upload will be posted on the website along with the authors' 
responses. I agree  for my report to be made available under an Open Access Creative Commons 
CC-BY license ( http://creativecommons.org/licenses/by/4.0/ ). I understand that any comments 
which I do not wish to be inc luded in my named report can be included as confidential comments 
to the editors, which will not be published.  
 
I agree to the open peer review policy of the journal  
Reviewer’s report  
Title: Evidence of surgical outcomes fluctuates over time: results from a cumulative meta -
analysis  of laparoscopic versus open appendec tomy for acute appendicitis  
 
Version: 0 Date: 17 Nov 2015  
 
Reviewer: Dimitrios Mavridis  
Reviewer’s report:  
The authors employed a cumulative meta -analysis of laparoscopic versus open appendectomy for 
acute appendicitis. T hey claim that 'established' evidence may change with time and they show 
that in their cumulative meta -analysis. I am a big proponent of cumulative meta -analysis and I 
believe it is a useful tool. I have some statistical concerns that I explain in detail l ater on. My 
biggest concern however, is the use of the term 'established'. The authors do not provide any 
guidance on when evidence is considered established. A meta -analysis, just like any trial, would 
ideally stop either for efficacy or for futility but these are determined based on sequential 
analysis methods and power considerations. For instance, in the first outcome (abdominal 
abscess), although I have not employed any analysis, it is clear that uncertainty is too large for 
any power analysis to consi der that evidence is established and we should stop for efficacy. 
Hence, it is not surprising that evidence changes direction because it is not established. Hence, 
the authors overstate the change of direction in the cumulative meta -analysis. There is a lo t of 
work done about when evidence is considered established and the authors may consider apply 
such methods in their example (or in another)  
 
Roloff V, Higgins JPT, Sutton AJ: Planning future studies based on the conditional power of a 
meta -analysis. Stat  Med 2013, 32:11 -24. 
 
Wetterslev J, Thorlund K, Brok J, et al. Trial sequential analysis may establish when firm 
evidence is reached in cumulative meta -analysis Journal of clinical epidemiology 2008: 61; 64 -
75 
 
I believe results will not change when there is established (by a statistical model) evidence and 
there is the danger that the paper may have to change direction and suggest that we should not 
keep on conducting trials when there is established evidence. This has been documented in the 
literature but  probably not enough because the problem is not eliminated.  
 
I have other comments, some of which I consider important  
 
Title: "confidence in the best evidence wavers"  
 
I am not a native English speaker and I may be wrong but I cannot see what the author s mean or 
are trying to achieve with this sentence  
 
Abstract, background: 'expected to yield definite results'  
 
Meta analysis most likely is going to give more powerful results but there is not an a -priori 
guarantee that the results will be definite  
 
Abst ract, background: " However, even…has not been previously reported"  
 
This is a contradictory sentence, if the evidence is established it can not change. By reading the 
paper I understood what the authors mean. However, established evidence is proven by 
sequential analyses and power considerations that are not mentioned in the paper. Otherwise, it is 
just a trend that if it has not been established it is not unlikely to change. The authors are correct 
that at any point, future trials may differ considerably b y those published and the summary 
estimate may change direction but be careful that evidence is not established  
 
Abstract, methods "we included a broad…effect size''  
 
I do not understand what the authors mean by 'randomization was stated' since it is mentioned 
early that only RCTs are considered. I also do not understand how this is related to the aim of the 
study being to detect a trend and not the accurate effect size . 
 
Page 3, lines: 13 -17: poor explanation of cumulative meta -analysis  
 
I have several concerns regarding the statistical analysis  
 
* It is strongly recommended not to use the I^2 to decide between a fixed effect and a 
random effects model. Choice should b e driven by clinical considerations and not from a 
test or an index value. I would suggest the authors use both models and if results are 
similar random effects results should be preferred whereas if not we should consider the 
possibility of small study ef fects  
 
* I would suggest checking, at least visually, for small study effects (probably not for the 
first outcome due t rarity of events). Cumulative meta -analysis can be used for showing if 
small study effects operate on these studies  
 
* Peto odds ratio is rarely used. The default option for rare events is a Mantel -Haenszel 
model with a 0.5 correction for zero cells. Also the Peto odds ratio does not require a 
correction for zero cells as you did.  
 
* The authors use weighted mean difference, also known as mea n differences, but it seems 
that different scales are used and standardized mean differences should be used instead  
 
* More information about the aforementioned comments can be found in the ninth chapter 
of the Cochrane handbook (Deeks JJ, Higgins JPT, Altm an DG (editors). Chapter 9: 
Analysing data and undertaking meta -analyses. In: Higgins JPT, Green S (editors). 
Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated 
March 2011). The Cochrane Collaboration, 2011. Available from www .cochrane -
handbook.org.)  
 
* The heterogeneity in Figure 3 is enormous, if standardized mean differences do not 
reduce heterogeneity we should start thinking if it is meaningful to pool these studies.  
 
* Although there are a large number of trials, the rarit y of events for intra -abdominal 
abscess complicates the analysis. Another example may be more suitable  
 
* I believe that results for abdominal abscess are overstated. Because of the small number 
of events uncertainty is large.  The 37th trial (Pedersen stud y I assume) is large with a 
respectable weight in the meta -analysis that is why it affects heavily the results. The 
width of the summary estimate is large, hence there is no reason to stop trials because it is 
not unlikely for the direction of the effect t o change.  
 
* The authors state that relative effectiveness between a pair of treatments may change 
because technology is different e.t.c. This is an important point but probably a more 
narrow review is needed if there are important changes in technology e. g. exclude old 
trials with the old technology e.t.c.  
 
Are the methods appropriate and well described?  
If not, please specify what is required in your comments to the authors.  
 
No 
 
Does the work include the necessary controls?  
If not, please spe cify which controls are required in your comments to the authors.  
 
Unable to assess  
 
Are the conclusions drawn adequately supported by the data shown?  
If not, please explain in your comments to the authors.  
 
No 
 
Are you able to assess any statistics in the manuscript o r would you recommend an 
additional statistical review?  
If an additional statistical review is recommended, please specify what aspects require further 
assessment in your comments to the editors.  
 
I am able to assess the statistics  
 
Quality of written Engl ish 
Please indicate the quality of language in the manuscript:  
 
Acceptable  
 
Declaration of competing interests  
Please complete a declaration of competing interests, consideri ng the following questions:  
 
1. Have you in the past five years received reimbursements, fees, funding, or salary from an 
organisation that may in any way gain or lose financially from the publication of this 
manuscript, e ither now or in the future?  
 
2. Do you hold any stocks or shares in an organis ation that may in any way gain or lose 
financially from the publication of this manuscript, e ither now or in the future?  
 
3. Do you hold or are you currently applying for any patents relating to the  content of the 
manuscript?  
 
4. Have you received reimbursements , fees, funding, or salary from an organization that 
holds or has applied for patents relating to the  content of the manuscript?  
 
5. Do you have any other fin ancial competing interests?  
 
6. Do you have any non -financial competing interests  in relation to this pa per? 
 
If you can answer no to all of the above, write 'I declare that I have no competing interests' 
below. If your reply is yes to any, please give details below.  
 
I declare that I have no competing interests  
 
I agree to the open peer review policy of the  journal. I understand that my name will be included 
on my report to the authors and, if the manuscript is accepted for publication, my named report 
including any attachments I upload will be posted on the website along with the authors' 
responses. I agree  for my report to be made available under an Open Access Creative Commons 
CC-BY license ( http://creativecommons.org/licenses/by/4.0/ ). I understand that any comments 
which I do not wish to be inc luded in my named report can be included as confidential comments 
to the editors, which will not be published.  
 
I agree to the open peer review policy of the journal  
Please review this paper.
user
Reviewer’s report  
Title: Evidence of surgical outcomes fluctuates over time: results from a cumulative meta -
analysis  of laparoscopic versus open appendec tomy for acute appendicitis  
 
Version: 0 Date: 17 Nov 2015  
 
Reviewer: Dimitrios Mavridis  
Reviewer’s report:  
The authors employed a cumulative meta -analysis of laparoscopic versus open appendectomy for 
acute appendicitis. T hey claim that 'established' evidence may change with time and they show 
that in their cumulative meta -analysis. I am a big proponent of cumulative meta -analysis and I 
believe it is a useful tool. I have some statistical concerns that I explain in detail l ater on. My 
biggest concern however, is the use of the term 'established'. The authors do not provide any 
guidance on when evidence is considered established. A meta -analysis, just like any trial, would 
ideally stop either for efficacy or for futility but these are determined based on sequential 
analysis methods and power considerations. For instance, in the first outcome (abdominal 
abscess), although I have not employed any analysis, it is clear that uncertainty is too large for 
any power analysis to consi der that evidence is established and we should stop for efficacy. 
Hence, it is not surprising that evidence changes direction because it is not established. Hence, 
the authors overstate the change of direction in the cumulative meta -analysis. There is a lo t of 
work done about when evidence is considered established and the authors may consider apply 
such methods in their example (or in another)  
 
Roloff V, Higgins JPT, Sutton AJ: Planning future studies based on the conditional power of a 
meta -analysis. Stat  Med 2013, 32:11 -24. 
 
Wetterslev J, Thorlund K, Brok J, et al. Trial sequential analysis may establish when firm 
evidence is reached in cumulative meta -analysis Journal of clinical epidemiology 2008: 61; 64 -
75 
 
I believe results will not change when there is established (by a statistical model) evidence and 
there is the danger that the paper may have to change direction and suggest that we should not 
keep on conducting trials when there is established evidence. This has been documented in the 
literature but  probably not enough because the problem is not eliminated.  
 
I have other comments, some of which I consider important  
 
Title: "confidence in the best evidence wavers"  
 
I am not a native English speaker and I may be wrong but I cannot see what the author s mean or 
are trying to achieve with this sentence  
 
Abstract, background: 'expected to yield definite results'  
 
Meta analysis most likely is going to give more powerful results but there is not an a -priori 
guarantee that the results will be definite  
 
Abst ract, background: " However, even…has not been previously reported"  
 
This is a contradictory sentence, if the evidence is established it can not change. By reading the 
paper I understood what the authors mean. However, established evidence is proven by 
sequential analyses and power considerations that are not mentioned in the paper. Otherwise, it is 
just a trend that if it has not been established it is not unlikely to change. The authors are correct 
that at any point, future trials may differ considerably b y those published and the summary 
estimate may change direction but be careful that evidence is not established  
 
Abstract, methods "we included a broad…effect size''  
 
I do not understand what the authors mean by 'randomization was stated' since it is mentioned 
early that only RCTs are considered. I also do not understand how this is related to the aim of the 
study being to detect a trend and not the accurate effect size . 
 
Page 3, lines: 13 -17: poor explanation of cumulative meta -analysis  
 
I have several concerns regarding the statistical analysis  
 
* It is strongly recommended not to use the I^2 to decide between a fixed effect and a 
random effects model. Choice should b e driven by clinical considerations and not from a 
test or an index value. I would suggest the authors use both models and if results are 
similar random effects results should be preferred whereas if not we should consider the 
possibility of small study ef fects  
 
* I would suggest checking, at least visually, for small study effects (probably not for the 
first outcome due t rarity of events). Cumulative meta -analysis can be used for showing if 
small study effects operate on these studies  
 
* Peto odds ratio is rarely used. The default option for rare events is a Mantel -Haenszel 
model with a 0.5 correction for zero cells. Also the Peto odds ratio does not require a 
correction for zero cells as you did.  
 
* The authors use weighted mean difference, also known as mea n differences, but it seems 
that different scales are used and standardized mean differences should be used instead  
 
* More information about the aforementioned comments can be found in the ninth chapter 
of the Cochrane handbook (Deeks JJ, Higgins JPT, Altm an DG (editors). Chapter 9: 
Analysing data and undertaking meta -analyses. In: Higgins JPT, Green S (editors). 
Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated 
March 2011). The Cochrane Collaboration, 2011. Available from www .cochrane -
handbook.org.)  
 
* The heterogeneity in Figure 3 is enormous, if standardized mean differences do not 
reduce heterogeneity we should start thinking if it is meaningful to pool these studies.  
 
* Although there are a large number of trials, the rarit y of events for intra -abdominal 
abscess complicates the analysis. Another example may be more suitable  
 
* I believe that results for abdominal abscess are overstated. Because of the small number 
of events uncertainty is large.  The 37th trial (Pedersen stud y I assume) is large with a 
respectable weight in the meta -analysis that is why it affects heavily the results. The 
width of the summary estimate is large, hence there is no reason to stop trials because it is 
not unlikely for the direction of the effect t o change.  
 
* The authors state that relative effectiveness between a pair of treatments may change 
because technology is different e.t.c. This is an important point but probably a more 
narrow review is needed if there are important changes in technology e. g. exclude old 
trials with the old technology e.t.c.  
 
Are the methods appropriate and well described?  
If not, please specify what is required in your comments to the authors.  
 
No 
 
Does the work include the necessary controls?  
If not, please spe cify which controls are required in your comments to the authors.  
 
Unable to assess  
 
Are the conclusions drawn adequately supported by the data shown?  
If not, please explain in your comments to the authors.  
 
No 
 
Are you able to assess any statistics in the manuscript o r would you recommend an 
additional statistical review?  
If an additional statistical review is recommended, please specify what aspects require further 
assessment in your comments to the editors.  
 
I am able to assess the statistics  
 
Quality of written Engl ish 
Please indicate the quality of language in the manuscript:  
 
Acceptable  
 
Declaration of competing interests  
Please complete a declaration of competing interests, consideri ng the following questions:  
 
1. Have you in the past five years received reimbursements, fees, funding, or salary from an 
organisation that may in any way gain or lose financially from the publication of this 
manuscript, e ither now or in the future?  
 
2. Do you hold any stocks or shares in an organis ation that may in any way gain or lose 
financially from the publication of this manuscript, e ither now or in the future?  
 
3. Do you hold or are you currently applying for any patents relating to the  content of the 
manuscript?  
 
4. Have you received reimbursements , fees, funding, or salary from an organization that 
holds or has applied for patents relating to the  content of the manuscript?  
 
5. Do you have any other fin ancial competing interests?  
 
6. Do you have any non -financial competing interests  in relation to this pa per? 
 
If you can answer no to all of the above, write 'I declare that I have no competing interests' 
below. If your reply is yes to any, please give details below.  
 
I declare that I have no competing interests  
 
I agree to the open peer review policy of the  journal. I understand that my name will be included 
on my report to the authors and, if the manuscript is accepted for publication, my named report 
including any attachments I upload will be posted on the website along with the authors' 
responses. I agree  for my report to be made available under an Open Access Creative Commons 
CC-BY license ( http://creativecommons.org/licenses/by/4.0/ ). I understand that any comments 
which I do not wish to be inc luded in my named report can be included as confidential comments 
to the editors, which will not be published.  
 
I agree to the open peer review policy of the journal  
assistant