Robot-assisted vs. video-assisted thoracoscopic lobectomy: a systematic review of cost effectiveness

Introduction

Robot-assisted surgery has been widely adopted in the United States since the da Vinci robotic platform was approved by the FDA in 2000. The ability to convert open operations to minimally invasive operations using the robotic platform has grown with the introduction of robotics. However, the cost effectiveness of robot-assisted surgery compared to laparoscopic and thoracoscopic operations remains controversial (1,2). A mixture of surgeon, patient and hospital administrator perceptions appear to be driving the adoption of robotic surgery (3,4). Given the increased utilization of robot-assisted surgeries, further understanding of the benefits vs. the costs of such procedures is necessary. When looking at cost effectiveness, current research has only shown cost advantages of robotic surgery in select instances (5). Nevertheless, the proliferation of robotic surgery has continued apace across specialties and procedures even as its cost effectiveness has remained controversial (6,7).

Existing studies on cost effectiveness following both robot-assisted and laparoscopic/thoracoscopic surgeries often do not contain a detailed breakdown of the costs necessary to determine whether gains in clinical outcomes are financially sensible. Additionally, cost effectiveness should not be assumed to be the primary driver behind adoption of a specific procedure type. Although the first video-assisted thoracoscopic surgery (VATS) lobectomy cases were published in 1993 and first robot-assisted thoracoscopic surgery (RATS) lobectomy cases in 2002 (8,9), which showed good evidence for superior outcomes and costs with VATS (10,11), open surgery (thoracotomy) was still the procedure of choice for most lobectomies through 2010. The superiority of RATS over VATS, in turn, is significantly less clear (12-14). Multiple studies and systematic reviews have compared the two procedures (15,16), but have generally focused more on clinical outcomes than cost comparisons (17-22). Additionally, if any cost-effectiveness data were incorporated, most studies have not incorporated equipment depreciation and a breakdown of the costs by category. Including such data would be important for departmental and hospital purchasing decisions, given the high up-front cost and the complexity of calculating total costs of robotic surgery suites. The aim of this systematic literature review was to conduct such an in-depth cost analysis and comparison for RATS vs. VATS lobectomy, in order to inform robotics planning and purchasing decisions for robotic surgery programs.

Methods

Electronic searches

We performed and report this review in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines (23).

We performed a systematic search of the Cochrane Library, EMBASE, Google Scholar, Ovid Medline, PubMed and Web of Science from 1 January 1995 to 1 September 2019. Key free-text search terms included “robot”, “robotic”, “robot-assisted”, “da Vinci” AND “lobectomy” or “pneumonectomy”. We did not restrict by language or study type. We also searched the clinicaltrials.gov website for any registered randomized controlled trials (RCTs) comparing RATS lobectomy to VATS lobectomy that were not yet published. References in relevant previous papers and trials were analyzed to identify further publications not captured by the initial search terms. This review occurred as part of a broader study approved by the University of Michigan Institutional Review Board (HUM00089766).

Selection of studies

Study eligibility was independently determined by two of the study authors (TP Keeney-Bonthrone and RM Reddy). Any RCTs, observational studies and case series examining lobectomy with VATS and RATS comparison groups were included. We excluded publications that did not include procedural cost data.

Data extraction and quality assessment

Outcome measures were extracted into a standardized extraction form (Microsoft Excel) by TP Keeney-Bonthrone and verified by AM Hawes. Any discrepancies were resolved via and consensus by TP Keeney-Bonthrone, LM Frydrych and RM Reddy. We analyzed the quality and generalizability of any RCTs that met inclusion criteria based on the appraisal methodology set forth in Schulz et al. (24). Retrospective reviews were evaluated with the Newcastle-Ottawa Quality Assessment Scale (25). Median costs were converted into mean costs wherever possible for easier cross-study comparison.

Outcome measures

The primary outcome measure was difference in total hospital stay cost, including procedure costs, between RATS and VATS. Secondary outcomes were difference in total cost (measured in percent, to account for significant institutional and regional cost differences), operating room (OR) time, hospital stay length, conversion rate, complication rate and mortality rate.

Results

Literature search

The database search identified 1,447 unique citations. An additional 13 studies were identified from citation review. After removal of duplicates, 1,413 abstracts were screened for eligibility and 1,358 were excluded as part of that process. The remaining 55 articles underwent full-text review (Figure 1).

Figure 1 PRISMA 2009 flow diagram.

Study characteristics

Seven articles met inclusion and exclusion criteria (Table 1). We found no eligible RCTs comparing VATS to RATS lobectomy. Study periods ranged between 2001 and 2016, and sample size ranged between 52 and 40,093. All seven included articles were retrospective cohort studies. Five analyzed data from single institutions; one used the Premier Hospital Database and one the United States Nationwide Inpatient Sample. Three investigations used data from single sites outside the United States (Austria, China and Italy). The remainder used data from the U.S. The included articles were evaluated on the Newcastle-Ottawa Quality Assessment Scale (Table 2).

Table 1 Baseline characteristics of included studies
Full table

Table 2 Newcastle-Ottawa Scale (NOS) score of included studies
Full table

Individual study cost outcomes

To begin to understanding cost differences, a single-surgeon study that eliminates inter-surgeon variability is helpful. This type of single-surgeon study was completed in Austria. A comparison of 26 consecutive RATS lobectomies conducted between 2001 and 2008 vs. 26 VATS lobectomies conducted in 2009 (26), showed that the median procedure cost of RATS was 44% (EUR 771) higher than VATS. RATS procedures used a 3-arm approach and 96% of operations were for lung cancer. The only demographic difference between groups was clinical stage > IB was 23.1% of RATS patients vs. 0% in VATS. There was no significant difference in postoperative complications or length of stay between groups. Median operating time was 32 minutes (17.4%) longer in the RATS group. Rate of conversion to open was not statistically significant (P=0.42). Unfortunately, total cost was not broken down into individual components and the study did not measure non-operative costs, so further costs conclusions could not be drawn.

A single center study performed in a U.S. hospital between 2008 and 2012 analyzed open (n=69), VATS (n=58) and RATS (n=57) lobectomies and segmentectomies (27). The authors did not separate lobectomy and segmentectomy procedure costs. Twelve-point-three percent of RATS and 22.4% of VATS procedures were segmentectomies, which could skew cost in favor of VATS. The mean overall cost including depreciation for RATS was $3,182 (23.0%) higher than VATS. RATS was also $1,975 (13.1%) more expensive than open procedures, but only the RATS vs. VATS cost difference was statistically significant. This was the only study reviewed in which total costs included depreciation and in which costs were also broken by cost category, but with segmentectomies serving as a cost confounder for lobectomies. Without depreciation, RATS was $2,148 (15.7%) more expensive than VATS and $775 (5.2%) more expensive than thoracotomy. Depreciation therefore accounted for about a third of the cost difference between RATS and VATS. In terms of individual cost categories, only OR cost differences for RATS vs. VATS were statistically significant, with mean RATS costs being $723 (16%) higher and OR times being 21 minutes (10.4%) longer. There were no significant differences in hospital stay length or complication rates.

When looking at a larger scale, the same findings of increased costs with RATS hold true. Swanson et al. used the U.S. Premier multi-site hospital database from 40 sites to analyze 15,502 thoracic procedures conducted between 2009 and 2011 (28). This included 335 RATS lobectomies and 3,818 VATS lobectomies, 295 which were propensity-matched. Among these, RATS lobectomies were on average 22.3% ($4,564) more expensive. The study also included median RATS lobectomy costs, which were 20.8% ($3,753) higher than VATS. However, depreciation was not included in calculations, and costs were not broken down into individual components. Outcomes did not differ statistically in terms of OR time or hospital length of stay. Only 30% of participating hospitals performed RATS, compared to 100% performing VATS lobectomies. Eighty-seven-point-two percent of RATS lobectomies and 44% of VATS lobectomies were performed at teaching institutions.

Paul et al. (29) investigated data from the U.S. Nationwide Inpatient Sample from 2008 to 2011 to compare 2,478 RATS and 37,595 VATS lobectomy costs and outcomes. Given the large sample size of this study, we treated median costs as mean for comparison purposes. Median RATS cost was $4,708 (26.3%) higher compared to VATS. Once again, these figures did not include depreciation and there was no cost breakdown. The RATS complication rate was 4.9% higher (50.1% vs. 45.2%), with a noteworthy increase in iatrogenic bleeding (5.0% vs. 2.0%). Differences in length of stay were statistically insignificant. OR time was not measured.

The most recent U.S.-based study involved single-center data gathered between 2010 and 2012, with a sample size of 73 VATS and 25 RATS cases (18). The difference in mean total cost was $2,042 (18.4%), and the study also broke those figures by fixed, variable, OR and supply costs. The only significant outcome difference was in operative time, with RATS cases taking an average of 48 minutes (26.2%) longer.

The cost findings identified in the U.S. studies also hold true in international studies. A single-center study in China examined lobectomies (n=145) and segmentectomies (n=39) performed in 2014 and 2015 (30). There were 71 RATS and 113 VATS procedures, which were then matched into 69 propensity-matched pairs. Seven each (10.1%) of the RATS and VATS matched procedures were segmentectomies. The authors did not separate lobectomy and segmentectomy procedure costs, which again serve as a cost confounder for lobectomies. Mean combined RATS cost for the matched pairs was 44.9% ($3,739) higher and mean OR time was 25 minutes (22.7%) longer. Depreciation was not included in calculations. Differences in complication rates and stay length were not statistically significant.

In Italy, a single-center investigation compared patients undergoing open (n=38), VATS (n=42) and RATS (n=23) procedures (31). Two RATS and one open segmentectomy were included in the cohort. Mean total cost for RATS was $10,045. This was 21.4% higher than VATS ($8,271) and 19.7% higher than open lobectomy ($8,393). These figures did include depreciation, which accounted for 4.6% of the cost of robotic procedures. The authors’ detailed breakdown also revealed that cost of materials was almost four times higher for RATS than VATS and open procedures. The authors further found that RATS had the shortest length of stay and largest number of lymph nodes resected, while OR time was longest.

Cost comparison across studies

We compiled cost data from the seven included studies, including total cost and individual charges where available (Table 3). Differing methodologies among the seven included articles precluded several direct comparisons. Only two studies [Deen et al. (27) and Novellis et al. (31)], for instance, included depreciation costs. Augustin and colleagues (26) did not measure non-operative costs such as inpatient stay costs that were included elsewhere, which also precluded comparison. Neither Deen et al. nor Bao et al. (30) separated lobectomy and segmentectomy costs. For Deen et al., 12.3% of RATS and 22.4% of VATS procedures were segmentectomies. Due to the lower cost of segmentectomies, their inclusion confounded our lobectomy cost comparison by exacerbating the mean cost difference between VATS and RATS procedures. A much smaller number of segmentectomies were included in Novellis et al. (total n=3).

Table 3 Cost outcomes of included studies
Full table

For the six studies that provided total cost amounts, the total cost of RATS ultimately ranged from 18.4% to 44.9% greater than VATS (mean 25.7%), with a mean dollar cost difference of $3,335. Mean cost of VATS was $13,310, and mean cost of RATS was $16,645. Three studies provided non-operative costs, and four studies provided OR and supply costs. From these studies, we found that the non-operative costs of RATS ranged between 12.4% lower and 2.1% higher than VATS (mean −6.5%), while operative costs ranged between 8.6% and 112.0% higher (mean 54.4%). Average cost of supplies was 130.3% higher than VATS. As a percentage of total cost, VATS supplies constituted an average of 22.7%, whereas RATS supplies constituted an average of 33.2% of the total cost.

The mean cost difference among these studies is put into perspective when compared to the mean overall lobectomy cost in the U.S. A 2017 study conducted a cost analysis of 23,858 patients in the nationwide Premier Hospital Database who underwent a lobectomy between 2008 and 2014 (32). They found that the mean total cost for a lobectomy was $26,661. The authors noted that 59.4% of cases were open and 40.6% minimally invasive, but did not delineate cost based on procedure types. The mean total costs in our study of both VATS and RATS were substantially lower than the $26,661 amount. This difference raises the question of how representative our study data actually is, and how useful it can be for departmental decisions about robotics.

Outcome analysis

In addition to our primary outcomes of cost, we also compared OR time, length of hospital stay, conversion rate to open procedures, complication rate and mortality rate as secondary outcomes across all seven studies (Table 4). Novellis and colleagues found that RATS was associated with shorter OR times, whereas four studies found the opposite. Only Novellis et al. found a statistically significant difference in length of stay, again favoring RATS. There were no statistically significant differences in rates of mortality or conversion to open.

Table 4 Clinical outcomes of included studies
Full table

Next, we generated a pooled odds ratio of complication rates (minor and major combined) with VATS compared to RATS, using the DerSimonian and Laird random effects measure. Our pooled estimates suggest that VATS has a reduced risk of complications [odds ratio 0.83, 95% confidence interval (CI): 0.77–0.90, P<0.0001]. The chi-squared test for heterogeneity showed that the included studies were quite homogenous (I²=0%; P=0.537). Funnel plots examined to look for bias showed no publication bias due to symmetry. When Swanson et al. and Paul et al. were removed due to their large sample sizes, the remaining single-site studies showed no statistically significant differences in complication rates (odds ratio 0.90, 95% CI: 0.57–1.40, P=0.626). Overall, examination of clinical outcomes across our included studies indicates that RATS does not lead to improved outcomes compared to VATS.

Discussion

We completed the most in-depth systematic review of the costs associated with VATS and RATS lobectomies. The data suggest that RATS did not result in superior outcomes but was consistently more expensive than VATS.

While other systematic reviews have found that the RATS approach is non-inferior or slightly superior to VATS from a clinical perspective (16,17,20), the same cannot be said from a cost perspective. There is a lack of data to suggest any impending break-even point where the RATS approach might become cost effective. This means that the continued spread of robotic surgery essentially constitutes a faith-based rather than evidence-based effort, given that its adoption is predicated on assumptions rather than any kind of trend line indicating that robotics will justify its cost. Only a few robotic procedures have so far shown themselves to have superior cost effectiveness (5), and these are likely procedures where basic laparoscopy had lacked the fine control in tight spaces to be superior to open surgery (e.g., prostatectomy) (33).

A counterargument would be that the quality of current data is insufficient to pass judgment. Our original intent had been to conduct a statistically rigorous meta-analysis. However, differing methodologies, small sample sizes and limited number of eligible studies precluded us from conducting the kind of rigorous statistical analysis that would have given us more definite insight into the cost effectiveness of RATS lobectomy vs. VATS. We chose not to pool any of these studies for cost analysis due to lack of homogeneity. All included articles had NCO quality scores within appropriate limits; none were RCTs and all suffered from further limitations discussed above. No studies included long-term follow-up to adequately assess postoperative outcomes beyond 30 days. We hope that these limitations will be addressed to a certain extent in the coming years. We found four planned or active RCTs comparing VATS to RATS lobectomy registered at clinicaltrials.gov, based in Brazil, Canada, France and Italy (34-37). Only the Canadian trial description explicitly includes cost analysis. None of the four RCTs will have sites in the United States. Given that the U.S. has dramatic fluctuations in procedural costs based on location, it may be difficult to apply any cost-effectiveness findings of these RCTs to the U.S. On the other hand, the relative rarity of surgical RCTs has been discussed extensively (38-41), so we expect that most cost-effectiveness data will continue to come from retrospective studies.

Lastly, one might argue that longer time periods are still required to understand the whole cost picture. For perspective, it took laparoscopic surgery over a decade after its broad clinical introduction in the early 1990s to reliably show superior cost effectiveness to open surgery in studies such as the 2004 results of the Clinical Outcomes of Surgical Therapy (COST) Study Group published in the New England Journal of Medicine (42). However, the advance from open to laparoscopic surgery was arguably more dramatic than from laparoscopic to robotic, at least given current technology. As described by Lin, the advance to RATS is more evolutionary than revolutionary (43). Therefore, we expect that it would take substantially longer for robotic surgery to broadly establish itself as more cost effective—if ever—compared to how long it took laparoscopy.

Considering that the costs and learning curve involved are significant, robotics advocates and hospital administrators may ultimately find themselves at odds over further use of robotic systems unless costs go down. If robotics advocates can neither identify strong examples of cost effectiveness at present, nor provide data to predict cost effectiveness in the future, then the adoption of robotic surgery at the systems level ultimately becomes a fad, rather than practice of evidence-based medicine. In the worst-case outcome, continued use of the robotic approach despite lack of evidence for cost effectiveness may even lead to lead to medical reversal (44,45) if the robotic approach is ultimately proven inferior to laparoscopic and/or open approaches. Some have already begun to argue for safeguards against further adoption of the robotic approach without better evidence (7). The authors’ institution is currently undergoing a long-term effort to determine the cost effectiveness of robotic surgery across multiple departments and procedure types, in order to determine optimal allocation of robotic resources. Such optimal allocation cannot ultimately occur without a willingness to walk away from using robotics for certain procedures. Over time and as new robotic platforms are developed, the cost of robotics should decline, but it is unclear when that will occur.

Conclusions

This review presents a review of the current evidence for the cost effectiveness VATS vs. RATS lobectomy. Our findings show minimal differences in outcomes coupled with consistently higher cost of RATS procedures. Nevertheless, the quality and quantity of existing primary literature may still be insufficient to draw definitive conclusions, particularly due to a lack of RCTs comparing the approaches.

The key question that providers and administrators may have to ask themselves is when they are willing to walk away from robotics—for specific procedures in the case of surgeons or from robotics as a whole in the case of administrators. We expect such decisions to include career, skill, economic and political considerations for the surgeons and departments involved. Precisely because of the complexity of such decision making, we hope that future studies will expand upon the current, limited knowledge base with the routine inclusion of in-depth cost breakdowns. This would provide an appropriate evidence-based understanding of how hospitals can best allocate their surgical resources and the extent to which robotics should play a part.

Acknowledgments

Funding: TP Keeney-Bonthrone was supported by National Institutes of Health (NIH) training grants TL1TR000435 and TL1TR002242 awarded through the Michigan Institute for Clinical and Health Research (MICHR).

Footnote

Provenance and Peer Review: This article was commissioned by the Guest Editors (Natalie S. Lui and Sean C. Wightman) for the series “Robotic Surgery for Lung Cancer” published in Video-Assisted Thoracic Surgery. The article has undergone external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/vats.2019.12.07). The series “Robotic Surgery for Lung Cancer” was commissioned by the editorial office without any funding or sponsorship. RM Reddy has received consulting/speaking fees from Intuitive Surgical, Inc., consulting fees from Auris Surgical, Inc., and serves on an Advisory Board for Medtronic/Covidien. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. This review occurred as part of a broader study approved by the University of Michigan Institutional Review Board (HUM00089766).

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Wright JD. Robotic-Assisted Surgery: Balancing Evidence and Implementation. JAMA 2017;318:1545-7. [Crossref] [PubMed]
2. Barbash GI, Glied SA. New technology and health care costs--the case of robot-assisted surgery. N Engl J Med 2010;363:701-4. [Crossref] [PubMed]
3. Aggarwal A, Lewis D, Mason M, et al. Effect of patient choice and hospital competition on service configuration and technology adoption within cancer surgery: a national, population-based study. Lancet Oncol 2017;18:1445-53. [Crossref] [PubMed]
4. Ahmad A, Ahmad ZF, Carleton JD, et al. Robotic surgery: current perceptions and the clinical evidence. Surg Endosc 2017;31:255-63. [Crossref] [PubMed]
5. Tan A, Ashrafian H, Scott AJ, et al. Robotic surgery: disruptive innovation or unfulfilled promise? A systematic review and meta-analysis of the first 30 years. Surg Endosc 2016;30:4330-52. [Crossref] [PubMed]
6. Heemskerk J, Bouvy ND, Baeten CGMI. The end of robot-assisted laparoscopy? A critical appraisal of scientific evidence on the use of robot-assisted laparoscopic surgery. Surg Endosc 2014;28:1388-98. [Crossref] [PubMed]
7. Sheetz KH, Dimick JB. Is It Time for Safeguards in the Adoption of Robotic Surgery? JAMA 2019;321:1971-2. [Crossref] [PubMed]
8. Walker WS, Carnochan FM, Pugh GC. Thoracoscopic pulmonary lobectomy. Early operative experience and preliminary clinical results. J Thorac Cardiovasc Surg 1993;106:1111-7. [Crossref] [PubMed]
9. Cao C, Manganas C, Ang SC, et al. A systematic review and meta-analysis on pulmonary resections by robotic video-assisted thoracic surgery. Ann Cardiothorac Surg 2012;1:3-10. [PubMed]
10. Kent M, Wang T, Whyte R, et al. Open, video-assisted thoracic surgery, and robotic lobectomy: review of a national database. Ann Thorac Surg 2014;97:236-42; discussion 242-4. [Crossref] [PubMed]
11. Lacin T, Swanson S. Current costs of video-assisted thoracic surgery (VATS) lobectomy. J Thorac Dis 2013;5:S190-3. [PubMed]
12. Migliore M. Robotic assisted lung resection needs further evidence. J Thorac Dis 2016;8:E1274-8. [Crossref] [PubMed]
13. Louie BE, Wilson JL, Kim S, et al. Comparison of Video-Assisted Thoracoscopic Surgery and Robotic Approaches for Clinical Stage I and Stage II Non-Small Cell Lung Cancer Using The Society of Thoracic Surgeons Database. Ann Thorac Surg 2016;102:917-24. [Crossref] [PubMed]
14. Yang H-X, Woo KM, Sima CS, et al. Long-term Survival Based on the Surgical Approach to Lobectomy For Clinical Stage I Nonsmall Cell Lung Cancer: Comparison of Robotic, Video-assisted Thoracic Surgery, and Thoracotomy Lobectomy. Ann Surg 2017;265:431-7. [Crossref] [PubMed]
15. Agzarian J, Fahim C, Shargall Y, et al. The Use of Robotic-Assisted Thoracic Surgery for Lung Resection: A Comprehensive Systematic Review. Semin Thorac Cardiovasc Surg 2016;28:182-92. [Crossref] [PubMed]
16. O’Sullivan KE, Kreaden US, Hebert AE, et al. A systematic review and meta-analysis of robotic versus open and video-assisted thoracoscopic surgery approaches for lobectomy. Interact Cardiovasc Thorac Surg 2019;28:526-34. [Crossref] [PubMed]
17. Liang H, Liang W, Zhao L, et al. Robotic Versus Video-assisted Lobectomy/Segmentectomy for Lung Cancer: A Meta-analysis. Ann Surg 2018;268:254-9. [Crossref] [PubMed]
18. Worrell SG, Dedhia P, Gilbert C, et al. The cost and quality of life outcomes in developing a robotic lobectomy program. J Robot Surg 2019;13:239-43. [Crossref] [PubMed]
19. Oh DS, Reddy RM, Gorrepati ML, et al. Robotic-Assisted, Video-Assisted Thoracoscopic and Open Lobectomy: Propensity-Matched Analysis of Recent Premier Data. Ann Thorac Surg 2017;104:1733-40. [Crossref] [PubMed]
20. Emmert A, Straube C, Buentzel J, et al. Robotic versus thoracoscopic lung resection: A systematic review and meta-analysis. Medicine (Baltimore) 2017;96:e7633 [Crossref] [PubMed]
21. Zhang L, Gao S. Robot-assisted thoracic surgery versus open thoracic surgery for lung cancer: a system review and meta-analysis. Int J Clin Exp Med 2015;8:17804-10. [PubMed]
22. Nakamura H. Systematic review of published studies on safety and efficacy of thoracoscopic and robot-assisted lobectomy for lung cancer. Ann Thorac Cardiovasc Surg 2014;20:93-8. [Crossref] [PubMed]
23. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol 2009;62:1006-12. [Crossref] [PubMed]
24. Schulz KF, Chalmers I, Hayes RJ, et al. Empirical Evidence of Bias: Dimensions of Methodological Quality Associated With Estimates of Treatment Effects in Controlled Trials. JAMA 1995;273:408-12. [Crossref] [PubMed]
25. Wells G, Shea B, O’Connell D, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses [Internet]. Ottawa, ON: Ottawa Hospital Research Institute. 2000. Available online: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp
26. Augustin F, Bodner J, Maier H, et al. Robotic-assisted minimally invasive vs. thoracoscopic lung lobectomy: comparison of perioperative results in a learning curve setting. Langenbecks Arch Surg 2013;398:895-901. [Crossref] [PubMed]
27. Deen SA, Wilson JL, Wilshire CL, et al. Defining the cost of care for lobectomy and segmentectomy: A comparison of open, video-assisted thoracoscopic, and robotic approaches. Ann Thorac Surg 2014;97:1000-7. [Crossref] [PubMed]
28. Swanson SJ, Miller DL, McKenna RJ, et al. Comparing robot-assisted thoracic surgical lobectomy with conventional video-assisted thoracic surgical lobectomy and wedge resection: Results from a multihospital database (Premier). J Thorac Cardiovasc Surg 2014;147:929-37. [Crossref] [PubMed]
29. Paul S, Jalbert J, Isaacs AJ, et al. Comparative effectiveness of robotic-assisted vs thoracoscopic lobectomy. Chest 2014;146:1505-12. [Crossref] [PubMed]
30. Bao F, Zhang C, Yang Y, et al. Comparison of robotic and video-assisted thoracic surgery for lung cancer: a propensity-matched analysis. J Thorac Dis 2016;8:1798-803. [Crossref] [PubMed]
31. Novellis P, Bottoni E, Voulaz E, et al. Robotic surgery, video-assisted thoracic surgery, and open surgery for early stage lung cancer: comparison of costs and outcomes at a single institute. J Thorac Dis 2018;10:790-8. [Crossref] [PubMed]
32. Kalsekar I, Hsiao CW, Cheng H, et al. Economic burden of cancer among patients with surgical resections of the lung, rectum, liver and uterus: results from a US hospital database claims analysis. Health Econ Rev 2017;7:22. [Crossref] [PubMed]
33. Ahmed K, Ibrahim A, Wang TT, et al. Assessing the cost effectiveness of robotics in urological surgery - a systematic review. BJU Int 2012;110:1544-56. [Crossref] [PubMed]
34. ClinicalTrials.gov. Bethesda (MD): National Library of Medicine (US). 2014 Sep 3. Identifier NCT02292914. Prospective Analysis of Robot-Assisted Surgery. 2014 Nov 18 [cited 2019 Aug 5]; [about 4 screens]. Available online: https://clinicaltrials.gov/ct2/show/record/NCT02292914
35. ClinicalTrials.gov. Bethesda (MD): National Library of Medicine (US). 2015 Sep 22. Identifier NCT02617186, Robotic Lobectomy vs. Thoracoscopic Lobectomy for Early Stage Lung Cancer. 2015 Nov 30 [cited 2019 Aug 5]; [about 4 screens]. Available online: https://clinicaltrials.gov/ct2/show/record/NCT02617186
36. ClinicalTrials.gov. Bethesda (MD): National Library of Medicine (US). 2017 Apr 8. Identifier NCT03111797, Robot-assisted Lobectomy Versus Video-assisted Lobectomy. 2017 Apr 17 [cited 2019 Aug 5]; [about 4 screens]. Available online: https://clinicaltrials.gov/ct2/show/record/NCT03111797
37. ClinicalTrials.gov [Internet]. Bethesda (MD): National Library of Medicine (US). 2016 Jun 15. Identifier NCT02804893, Videothoracoscopic (VATS) vs. Robotic Approach for Lobectomy or Anatomical Segmentectomy (ROMAN). 2016 Jun 17 [cited 2019 Aug 5]; [about 4 screens]. Available online: https://clinicaltrials.gov/ct2/show/NCT02804893
38. McCulloch P, Taylor I, Sasako M, et al. Randomised trials in surgery: problems and possible solutions. BMJ 2002;324:1448-51. [Crossref] [PubMed]
39. Thoma A, Farrokhyar F, Waltho D, et al. Users’ guide to the surgical literature: How to assess a noninferiority trial. Can J Surg 2017;60:426-32. [Crossref] [PubMed]
40. Kao LS, Tyson JE, Blakely ML, et al. Clinical research methodology I: introduction to randomized trials. J Am Coll Surg 2008;206:361-9. [Crossref] [PubMed]
41. McCulloch P. Developing appropriate methodology for the study of surgical techniques. J R Soc Med 2009;102:51-5. [Crossref] [PubMed]
42. Nelson H, Sargent DJ, Wieand HS, et al. A comparison of laparoscopically assisted and open colectomy for colon cancer. N Engl J Med 2004;350:2050-9. [Crossref] [PubMed]
43. . Lin J. Robotic lobectomy: revolution or evolution? J Thorac Dis 2017;9:2876-80. [Crossref] [PubMed]
44. Prasad V, Cifu A. Medical reversal: why we must raise the bar before adopting new technologies. Yale J Biol Med 2011;84:471-8. [PubMed]
45. Prasad V, Vandross A, Toomey C, et al. A decade of reversal: An analysis of 146 contradicted medical practices. Mayo Clin Proc 2013;88:790-8. [Crossref] [PubMed]