Seeing Through the Lies: Innovation and the Need for Transparency

23 November 2016

Seeing Through the Lies:
Innovation and the Need for Transparency

Professor Martin Elliott

 “The first principle [of science] is that you must not fool yourself

and you are the easiest person to fool.”

Richard Feynman 1918-1988

theoretical physicist

Introduction

This paper is an evolution of the 2016 Robert Gross Lecture I delivered earlier this year in Boston, USA. In my early Gresham lectures, I introduced you to some of the pioneers of cardiac surgery, and discussed whether, in our current highly scrutinized and ethically monitored world, we would be allowed to do now what they did then.  Of course, the story of Robert Gross and his first PDA ligation was a classic case study in that context.  The very idea of waiting until your boss is away from town so that you could get on and do a ‘world first’ operation seems shocking now, and I am pretty sure that such an approach would, in most of the institutions in which we work, end our careers if we tried it now.  However, there remain medical practitioners whose moral compass is at least misdirected and who, because of both personal and institutional drivers, lie to themselves, their colleagues and the public at large.  In this lecture I will recount some stories of such individuals and consider, with particular reference to surgery, what might drive them to behave in such a way, and outline some ways in which we might be able to change the underlying culture which may even encourage such behaviour.

The Ethical and Moral Framework of Medicine

Modern medicine assumes that we care about the safety, effectiveness and patient experience in what we do.  It relies on us sharing the truth about risks and outcomes.  If we fail to deliver that truth, we lose the trust on which all good practice is based.

There can be little doubt that surgical innovators took risks based on a passionately held premise that they could improve the treatment of various disorders. The risks they were taking were reputational and professional. But the risks taken by their patients and their families were of a different order; not only were the patients’ lives on the line, but no one could predict what those lives would be like if they were to live. 

Over the centuries doctors have not exactly covered themselves in moral glory, as so many have been exposed as charlatans peddling unsuitable or unproven treatments to the weak and vulnerable, raising false hopes, causing pain and even death.

Patients are at their most gullible when most sick and most vulnerable.  Sadly, some physicians have always been willing to exploit that.  It represents a breakdown of trust. “Trust me, I’m a doctor”.  But how we have let down that trust. 

I have written previously (http://www.gresham.ac.uk/lectures-and-events/the-ethical-challenges-of-new-treatments-in-children-could-we-do-now-what-we-did) about the deep involvement of doctors throughout history in the design, supervision and maintenance of torture.

To quote from another Gross1, “like Elie Wiesel, we are apt to believe that a medical degree ought to ‘shield’ doctors from evil”.  The acts of the Nazi doctors in the 2nd World War clearly demonstrated that it does not. These doctors carried out, in concentration camps, a series of medical (so-called) ‘experiments’ on large numbers of prisoners, mainly Jews (including children) from across Europe, but also Romani, ethnic Poles, Soviet POWs and disabled non-Jewish Germans.  Prisoners were forced to participate; they did not volunteer, there was no consent. Data collection and analysis were infantile, biased or absent, or indeed all of these. Typically, the experiments resulted in death, disfigurement or permanent disability.  This was true medical torture; designed and performed by doctors.

Times change, and we must be aware that the ethical and moral principles that govern our decisions are a product of their time.

Sir John Laws (https://en.wikipedia.org/wiki/John_Laws_(judge)) has argued[1] that these principles are forged over time by reasoned argument, and only by such argument, on the core principles of the human condition.  These state that 1) man is rational; 2) that man has free will; and 3) that he lives in community with his kind.  Thus we must recognise that values, once held dear, may change and be replaced by other, previously unacceptable ideas.

To repeat, ethics are a product of the time in which we live and the company we keep. Our era, long after the Helsinki Declaration, has not been immune from scientific fraud, quackery, cruelty or deceit.  Here are some well-known examples.

In the UK, the most famous is Dr. Andrew Wakefield who, it turned out, was paid by lawyers to undertake a study flawed in every possible way to prove a link between MMR vaccination, autism and inflammatory bowel disease.  His paper has been retracted, but not before huge falls in immunization rates and a renewed incidence of death from measles, worldwide.  Disgraced in Europe, he made his way to the USA and continues to campaign against immunization, the latest outing of which beliefs is in the self-directed film “Vaxxed”, which has been panned in the media [https://en.wikipedia.org/wiki/Vaxxed] and withdrawn from this year’s Tribeca Film Festival.

In the USA in 1981, John Darsee falsified data in 5 important papers from Braunwald’s cardiovascular lab in Harvard, Boston.  Thirty papers had to be retracted.

In 1987, there was the famous case of Stephen Breuning of Pittsburg USA was found guilty of ‘knowingly, willfully and repeatedly engaging in misleading and deceptive practices in reporting results of research relating to drug therapy in severely retarded patients.

In 2009, Hwang Woo-Suk from Seoul in Korea was found to have fabricated most of his data relating to Somatic Cell Nuclear Transfer in stem-cell cloning experiments. 

Scott Reuben, at Tufts University, Medford, MA, USA fabricated data in clinical trials into post-operative orthopaedic pain.  The trials reported in 21 papers were never actually done. Scientific American called Reuben ‘A Medical Madoff’.

In 2010, another anaesthetist, Joachim Boldt in Giessen, Germany, fabricated data in 10 of 91 papers relating to colloid based resuscitation.

In 2012, Diederik Stapel of Tilburg University, The Netherlands, fabricated data in 58 social psychology studies, and in the same year Yoshitaka Fuji, from Toho University, Japan was found to have been fabricating data over 19 years relating to 183 papers about post-operative nausea and vomiting.

These were the ones that were found out.

King et al, from Boston University published an interesting abstract this year2 analysing 186 retracted articles in the surgical literature from 1991 thru 2015.  Almost a quarter (22%) of the retracted articles related to cardiac surgery.  Anesthesia related topics were found in 32%.  The USA accounted for 22% of retractions, Germany 19% and Japan for 16%.  Data duplication or falsification accounted for almost half the retractions.

I want to tell you a story which started with one patient, required me to acquire new ‘company’, initiated some very productive research and which gained much positive publicity.  Sadly, that company had, let us say, ‘flaws’ and the trust on which patients and science both rely was broken.

I direct the National Service for Severe Tracheal Disease in Children in the UK. My team manages most of the patients with long segment congenital tracheal stenosis in the UK. I want to introduce you to Ciaran. I have known him and his family for almost 19 years, since shortly after he was born with long segment tracheal stenosis, < 1mm in diameter.  The mortality for such severe lesions at the time was huge, yet he was one of the first ten people in the world to be put onto ECMO before being transferred to Great Ormond Street (GOSH) for surgery.  In fact, he has been one of the first ten people in the world to have most of the treatments to which he has been subjected.  He underwent the last primary pericardial patch tracheoplasty that I did. This form of repair carried at least a moderate mortality and an uncertain future.  He did well, although suffering mild brain damage due to ECMO problems at his referring hospital.

The pericardial patch shrank and scarred (as they do) and, at 3 months of age, he needed a Palmaz metal tracheal stent.  Again, he was one of the first 10 patients in the world to get such a stent.  Long-term results were unknown.

At the age of three, his stent eroded into his aorta, and, as an emergency, I had both to fix his aorta and replace the stented portion of the trachea. I used a tracheal homograft, the largest ever used at the time and one of a handful worldwide, most of which our team had done, working with Claus Herberhold from Bonn and with Jeff Jacobs in Florida.  No one knew what the outcome would be. Actually, it turned out not to be so bad, with the grafts becoming covered in normal airway lining (ciliated epithelium) and survival being significantly better (at 60% at 12 years) than expected. No one has repeated this study, and we have yet to discover how these patients are, so many years later.

Ciaran needed another stent, but was well for 10 years, until once again his stent eroded through his trachea into his aorta.

Because of the BSE crisis (mad cow disease) in Europe, the homografts we had used previously were under a moratorium, and we had to find a new way to treat him; he was lying sedated in a Belfast hospital at high risk of bleeding to death.  We now ran the largest tracheal service in the world, and had a strong network.  We had been working to ‘grow’ tracheas in the lab with Martin Birchall at University College London, Mark Lowdell at the Royal Free Hospital, Augustinus Bader in Germany and Paolo Macchiarini in Barcelona and Florence.  Paolo had inserted a small stem-cell supported transplant in a lady in Barcelona in 2008, and this formed the basis of our new strategy for Ciaran.

Using techniques developed at all these sites, together we performed the world’s first stem cell supported tracheal transplant in a child3. We had little idea what the future held. Yet Ciaran is alive and well nearly seven years later, 25kg heavier and with a good quality of life. He is on no anti-rejection therapy and continues to play the drums in show bands.  He has a stented, slightly narrow trachea with little evidence of growth (increase in width), but excellent cilial function despite that. 

Ciaran has done well, but were we right to do it?  His life has been lived in the ‘innovation space’, and as a group we have continued to innovate.  At each stage, we had gathered evidence that the approach we were to take was, at least, reasonable.  There was some animal work out there, but no or minimal human experience.  Often the science was in its very early stages, and we were seeking, with his family in complete accord, to offer him the best chance we could by taking bigger leaps than, as scientists, we would naturally have been comfortable doing.  The alternative was an unpleasant and rapid death for Ciaran.  He was what Kate Bull calls an ‘n = 1’ patient.  It is known as ‘compassionate use’ of a new technique.

In theory at least, innovation should be in the interests of everyone, patients and doctors alike.  Humans generally want to live disease free, for longer and with a good quality of life.  Who wouldn’t support an innovation in patients for whom there is no available effective treatment for their condition and who otherwise might look forward to distress and suffering, in addition to a shortened life?  The rewards for the patient if the innovation proves to be successful are obvious. There are significant rewards for the medics involved.  These may be reputational, financial, promotional or just related to self-worth.  Without care, they can become a conflict of interest.

Let me return to my story.  We received immense publicity throughout the world for this procedure, and thanks to Ciaran’s ready smile, good humor and drumming skills, continue to do so. We did one more transplant shortly after Ciaran, but very sadly that child died, after a bronchoscopy at another hospital.  Autopsy was refused and we will never know the fate of the donor graft.

We were successful in attracting stem-cell scientists from around the world, and obtained substantial grant income to continue the work.  We have clinical trials about to start on tracheal, laryngeal, diaphragmatic, oesophageal and (soon) small bowel transplantation as a result.

It was a good example of the benefits of incorporating ‘compassionate use’ into the framework of research.  We had gone from bench to bedside and back to the bench, re-testing optimal scaffolds, bioreactors and reagents.  Everything we did, every step we took, was published4-7.

To avoid the need for de-cellularised donor human tissue to act as a scaffold upon which recipient stem cells could be bonded, we had been working with the nanotechnology group at University College London to manufacture a synthetic scaffold. The materials tested were very promising, and mesenchymal stem cells would stick to them for a while in vitro, but not reliably.  Given the ‘success’ of the experience with homograft transplantation, we thought that it would be wise to test out in the lab which would be the optimal scaffold for use in human trials, before operating again, unless for compassionate use. 

Meanwhile, Paolo Macchiarini was continuing a peripatetic clinical and academic life, ultimately settling at Karolinska University in Stockholm as Professor of Thoracic Surgery.  He was performing a number of similar airway transplants in different locations around the world, but he became specifically enamoured with concept of the synthetic scaffold (using a different product to the one we had been testing in London), and after a small number of individual cases, obtained a grant in Russia to undertake the operations in multiple centres there, but based out of Karolinska in Sweden.  Macchiarini has huge charisma and rapidly became a ‘celebrity surgeon’, and was the subject of a number of interviews and later full-scale documentaries both in Europe and the USA.

However, black clouds began to loom on his horizon. First because he was accused of financial fraud in Italy relating to the mode by which he was obtaining private patients and how much he was charging them.  Second, rumours began to circulate in the tracheal community that all was not well with the patients he operated on, despite his upbeat public statements and initial papers.  And thirdly, his nemesis, Pierre Delaere from Leuven in Brussels launched a vitriolic attack on Machiarini’s work, publications and style. Delaere employs an alternative and very interesting mode of tracheal transplantation, sadly beyond the scope of this lecture.  Nonetheless, the Macchiarini façade was beginning to crack.

It was clear that public arguments like these, and lack of clarity over outcomes had the potential seriously to damage other workers in the field.  Accordingly, with other interested parties, I organised a workshop (held in Paris) to bring together everybody who was active in tracheal replacement research to define what the outstanding research questions were, how we might collaborate and what our priorities should be.  Macchiarini would not attend.  Delaere came under duress, but we did publish a consensus8.  Delaere has been quoted as saying the stem-cell biology and tissue engineering are little more than witchcraft, or black magic.  And he regards the practitioners with little more respect than witch doctors.  He does not think the field will ever prove beneficial.

What happened next can only have strengthened those opinions.  As I indicated, Macchiarini is very charismatic; he was described, by my colleague Professor Martin Birchall, as being something of a Pied Piper in the way he could attract people to work with him.  Macchiarini was the subject of two documentaries.  One was from a German crew and the other from a Swedish crew.  The German one heaped paeons of praise on Macchiarini – their resulting 2013 documentary was called “Cells that heal: The Wonder of Krasnodar” in its German original. Bosse Lindquist’s Swedish one (http://www.svt.se/dokument-inifran/experimenten-in-english) was an out and out investigation into what had become bigger news, namely that both ethical and outcome questions were being raised with Karolinska about Macchiarini’s outcomes and research practices. Data manipulation, absences and inconsistencies with published papers were exposed, and the ethical and consent processes questioned. As Bosse Lindquist points out (personal communication), “the German crew’s footage provided important evidence to substantiate that Macchiarini knew his [tracheal] matrices had been faulty in previous plastic trachea surgeries, and still were ‘erroneous’ when he implanted them into subsequent Russian patients. Their footage also showed how he and his team duped the patient into believing the procedure was safe and well-known. Finally, their footage also showed how the truth was revealed during the first bronchoscopy of Julia post-op. The team could clearly see that the synthetic trachea had become bent and partially collapsed. And that she had difficulties breathing.”

Karolinska launched a formal review, which initially he got through relatively unscathed, but persistent media criticism of both him and the process of the review resulted in the dismissal of Macchiarini and the resignation of his chiefs in Karolinska. 

As this was going on, lurid tales of a third documentary, this time commissioned by NBC in the USA emerged in an article in Vanity Fair9.  The article came out in January this year and was called “The Celebrity Surgeon who used Love, Money and the Pope to Scam and NBC News Producer”.  It is a long, fascinating but ultimately depressing and tragic story.  But nonetheless worth a read!  Macchiarini met the producer, Benita Alexander, in February 2013, shortly before he was due to do an interview with Meredith Vieira for a 2-hour special called “A Leap of Faith”.  After spending increasing time together through early 2013, when he inserted a synthetic trachea into a child in Peoria with Richard Pearl, Benita and Paolo went for ‘an incredibly romantic weekend’ in Venice and their relationship, shall we say, blossomed.  He proposed to Benita in June 2014, having eaten and loved their way around Europe in previous months.

Whist she described him as ‘secretive’ at times, he eventually told her that he had operated on Bill and Hilary Clinton, Emperor Akihito of Japan and Barack Obama.  His texts to her were a litany of famous names, e.g. “I just left a meeting with PF [Pope Francis]”.  Benita was Episcopalian, and Paolo a Catholic, but he said he would talk to this friend the Pope, and by October 2014 he claimed to have resolved the issue and in fact Pope Francis had agreed to officiate at the wedding, which would be held at the Pope’s summer residence at Castel Gandolfo.  He also said that Putin, the Obamas, the Clintons, Sarkozy and the opera singer Andrea Bocelli would be attending.

A concerned friend had come across the Pope’s forthcoming schedule, and it was clear that he was to be in Latin America at the time of the wedding. She sent the schedule to Benita. It slowly dawned on her that everything she had been told was an elaborate charade.  Ronal Schouten, director of the Law and Psychiatry service at Massachusetts General Hospital is quoted in the Vanity Fair article as saying the following “We’re taught from an early age that when something is too good to be true, it’s not true,” he said. “And yet we ignore the signals. People’s critical judgment gets suspended. In this case, that happened at both the personal and institutional level.” 

“Macchiarini is the extreme form of a con man. He’s clearly bright and has accomplishments, but he can’t contain himself. There’s a void in his personality that he seems to want to fill by conning more and more people.” When asked how Macchiarini stacks up to, say, Bernie Madoff, he laughed and said, “Madoff was an ordinary con man with a Ponzi scheme. He never claimed to be the chairman of the Federal Reserve. He didn’t suggest he was part of a secret international society of bankers. This guy is really good.” 

So why am I telling you all this gossip.  Further to humiliate Macchiarini? To embarrass Benita Alexander?  To drop names myself?  No; the reason I wanted you to hear this is to emphasise the absolute importance in medicine and in science of truth and trust, both of which have been massively eroded by Macchiarini’s narcissistic behaviour.  He has done immense damage to the field of tissue engineering, however right his underlying hypotheses have been.  Units and teams around Europe with whom he has worked have had full-blown academic investigations into ethics approvals, patient consenting processes, lab books, hard results and publications.  People have been distracted greatly from their proper jobs, and highly talented youngsters have been scared off.  Grants have either been suspended or become more difficult to obtain, and trials harder to get off the ground.  Morale has been badly damaged.  And he has fed the fire of Delaere’s ire, who himself launched attacks on social media and via the medical literature on those doing careful lab-based work in tissue engineering.  It has all been very destructive.

We have to face the fact that doctors tell lies.  And it is no good pretending, as one does, that ‘it would not happen here’.  Recent work10 by Kim Serota reveals that whilst most people say the don’t lie, lying is frequent.  And almost half the total number of lies are told by only 5% of subjects.  A few prolific liars can taint the rest of us.

And it has made me think about truth and lies in surgery.  And there is no better introduction to this part of my talk than these words from the PLoS Medicine Editorial team, in a 2009 editorial called; ‘An unbiased scientific record should be everyone’s agenda’

“It is the responsibility of everyone involved to ensure that the published record is an unbiased, accurate representation of research.” 

Despite all the advances that we have made in cardiac surgery over the last 70 years, I feel some shame.  Shame because I think that we have not used our data as well as we could, despite the training and encouragement of the great clinical scientist, John Kirklin.  Shame, because of the data lost from patients who have had procedures in places that do not contribute either to research or registries.  Shame because journals tend only to publish positive results, and often only that which is fashionable.  Most of all I feel shame that almost none of the data about patients treated during that time is poolable or even retrievable.

I do not think our current system is fit for purpose.  It conspires against proper accumulation, sharing and analysis of data.  And it facilitates fraud.  It is all about the data, and our peers in other disciplines like physics, maths, geography or economics do it much better.  Let me first explain why we publish as we do and how we publish.

I am probably best described as an academic surgeon.  My status in that world and indeed both my employability and fundability, depend on the number of peer-reviewed papers I have published in scientific Journals (of which there are over 11,000 according to the Web of Science), the impact factor [2]of the Journal which publishes the paper and the citation index[3] relating to the paper itself.  It is in my interests both to publish and to publish in high impact journals, which incidentally largely focus on drugs, genes and basic science rather than surgery or highly specialised fields.  It is in the journals interests to maintain this position.  As George Monbiot pointed out in the Guardian in 2011 “Academic publishers make Murdoch look like a Socialist” (http://www.theguardian.com/commentisfree/2011/aug/29/academic-publishers-murdoch-socialist).  Journal subscriptions are high, and account for >65% of university library budget.  Individual article reprints are extortionate for non-university employees (including many of those working for the NHS), and operating margins are high.  In 2011, Wiley, Elsevier and Springer published 42% of journal articles.  They control the scientific world, and their business model succeeds.  Once again from Monbiot; “The big publishers have rounded up the journals with the highest academic impact factors, in which publication is essential for researchers trying to secure grants and advance their careers.”

I often wish to share papers with my patients.  As members of the laity, they are functionally excluded (unless they are rich) either because it can cost up to £40 to download the full text of an article, or I break the copyright rules.  The Journals hold the copyright, not the creator of the work.

Let me illustrate this with a surgical example, a new operation.  Not a complex clinical trial, a new drug or a piece of cellular engineering.  Just an operation.

So, I have a proposal.  To illustrate that proposal I want you to imagine that we invent a new operation.  I hope you will agree with me that most ‘new procedure’ papers have small numbers, variable datasets, fairly simple outcome measures and very short follow up.  That may be enough if you want to acquire an eponymous operation, or to increase the weight of your CV (still incidentally a goal for most residents, knowing that it will help them get a job).  And it does mean that it heads off some potential criticism for not remembering Buxton’s Law11: “it is always too early (for rigorous evaluation) until it is suddenly too late”. Sadly, most evaluation is far from rigorous.

Let’s take go along the pathway of fame for my innovative procedure which, for today, I have modestly called the Elliott operation.  I think it up, perhaps have some animal work to support it and (perhaps) get clearance from the Ethical Committee to carry it out.  I have conviction in my idea having thought it through carefully perhaps for years; I am confident or vain enough to try it;  and keen to publish, which as you have heard I must do to maintain my promotion prospects in my University or hospital. 

I do the first few operations and am, of course both convinced and not surprised that the outcome is good.  I want to publish it.  If I don’t, I am sure that someone will copy my idea and get there first, and that would be no good would it?  My vanity (I will enjoy the press conference) and the CV pressure again. 

So I send a paper to a Journal as early as I think I can get away with it.  But what is that paper in reality?  It is actually a narrative that I write, usually wrapped up in a conventional ‘scientific’ format of introduction, methods, results and discussion, but almost certainly with a ‘positive bias’. The paper will include a description of the operations, summary tables of information and often histograms, pie charts or simple statistical summaries.  There will be no raw data sent to the journal. I retain ownership of that and thus what I send to the journal is actually a précis, written by me, the self-interested one.   Without access to the raw data and the data definitions associated with them, the reviewers are at a significant disadvantage.  They only see my selection and interpretation of the data.  As Robert MacCoun & Saul Perlmutter have pointed out (Blind analysis: Hide results to seek the truth.  Nature 526, 187–189 (08 October 2015) doi:10.1038/526187a), “many motivations distort what inferences we draw from data. These include the desire to support one's theory, to refute one's competitors, to be first to report a phenomenon, or simply to avoid publishing 'odd' results. Such biases can be conscious or unconscious”.

Why should the journal publish my paper?  Because they love innovative ideas, they also want to be first, my results look promising so there is +ve publication bias and they may get papers from the people who do it next.  All of these are good for readership, advertising and subscription figures.

The Journal is pretty well obliged to send my paper out to peer review.  The reviewers may feel it is a duty to perform the review.  They may be flattered.  Or they may be really glad to read about my new idea and want to do it next, to improve their reputation.  Unfortunately, there is a great deal of evidence12 that peer review is far from perfect. In 1998, Dr. Fiona Godlee, editor of the British Medical Journal, sent an article with eight deliberate mistakes in study design, analysis, and interpretation to more than 200 of the journal’s regular reviewers[4]. None of the reviewers found all the mistakes, and on average, they spotted fewer than two. And another study by the BMJ showed that experience was an issue, not in improving quality of reviewers, but quite the opposite. Over a 14­year period assessed, 1,500 referees, as rated by editors at leading journals, showed a slow but steady drop in their scores. 

Despite that, many feel that peer-review, like democracy, may be the least-worst option!  Remember though, I could have been lying, and anyway, I’ve only sent them summary data, to my design.

Let’s now imagine that the Journal has published my brilliant paper on my brilliant new operation.  Unsurprisingly, a second wave of surgeons and units want to try it.  They must, otherwise surely they will be judged old fashioned by their peers?  And aren’t there always new procedures by which we weigh our cojones (or whatever the female equivalent is)?The Senning, the Jatene, the Norwood, the Nikaidoh etc.; they have all followed this path.

So our second wave of surgeons starts to do the operations, based on my optimistic paper of summary findings and early results.  If we are lucky, they too get good results, but that rarely happens as the hidden tricks I’d thought about for years weren’t in my brief paper.  If their results are less good, it is really not in their interests to publish.  And even if they did submit a paper for publication, many Journals will be uninterested in ‘negative’ results.  The second wave submission has everything against it.  It is not innovative but derivative, it has negative outcomes, and the follow up is as short as mine.  The peer reviewers are unlikely even to be asked for an opinion.  Worse still, the data associated with these second wave operations are lost forever. Most of us know this to be the case. Think again of the operations I listed and consider how many unreported deaths and complications there must have been in centers trying the procedures.  Anecdote I am aware, but I doubt if I am far from the truth.

I would love to disrupt this system, and make useful data available to the clinical, and scientific communities and to anxious patients, desperate to make the correct choice.

The first part of my proposal for system change relates to the ownership of the data associated with the procedures.  At present either I, as the inventor of my eponymous operation or my institution effectively ‘own’ the data associated with it. There will of course be arguments declaring that the ultimate ownership lies with the patient (it is after all their body upon which I operate). I am concerned that it is I, or those working with me, that collect and hold data associated with the operation.  I have a clear conflict of interest, from the design of the blank dataset onwards.  It is in my best interests to show my operation in the best light, or bin the results if everything looks as though it is going pear shaped.  It is also in my best interests to retain those data under my ownership to retain my eponym and perhaps any associated IP deals that might be negotiable.

It is also in my interests to work on the presentation of those data to reveal my procedure in its best light, either via presentation at meetings, at other centers or in submission to Journals.  Much can be hidden in well-designed bar charts and pie diagrams, and it is, in my experience as a reviewer, rare for surgical papers to have had good advice from statisticians in advance.  As Sir Ronald Fisher (1890 – 1962), the great British statistician put it;

“To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination.  He can perhaps say what the experiment died of.” 

The raw data stay at home, on a metaphorical USB stick on my key ring.  The data cannot be added to by others, analysed by others or challenged by others. They are MINE.

The data associated with an innovation are critical to understanding its value.  To quote Michael Porter, value = outcome/cost, and in healthcare both outcome and cost estimation should extend over the life of the individual.  Modern digital systems both commercial and domestic allow us to begin to address this equation in our work.  And we should remember the words of Martin Buxton in what has become known as Buxton’s Law11: “it is always too early (for rigorous evaluation) until it is suddenly too late”. Sadly, most evaluation is far from rigorous, although surgeons in Bristol led by Jane Blazeby are convinced13 that such evaluation is possible using randomized trials from the beginning.

Thus, for innovative procedures; 

Proposal 1; 

Datasets must be well designed in advance and open to review, criticism and contribution before data are entered into them. They should contain clear data definitions, be held securely, be publically accessible (by a system of permissions) and where relevant have public APIs to permit easy sharing between data systems. There may be a role for permanent, independent data monitoring committees to support this process.

Proposal 2;

Datasets must address Porter’s value equation.

Proposal 3;

The raw data populating the datasets should not be owned by the operator or unit, but be held by an ‘honest broker’ such that the data can be seen, analysed, added to and challenged by other teams, statisticians, regulators and institutions. An ‘honest broker’ might be a professional body, or other trusted institution. This is the only way to achieve the necessary transparency.  Further, and as Professor Jane Blazeby from Bristol has pointed out (personal communication) the accumulation of large quantities of high quality data associated with relevant and well defined cohorts of patients will provide the basis for the development of more sophisticated and appropriate clinical trials. Indeed, work from Bristol has established methods to teach surgeons how to recruit into RCTs and these are being widely applied successfully. In addition, the recent investment into RCTs in surgery by the Royal College of Surgeons is leading to a gradual change in the culture of surgical research because surgeons are demonstrating that they can collaborate to deliver large scale trials14 - something which hitherto has not been possible.

Proposal 4;

No unit or surgeon should be allowed to perform the innovative procedure without agreeing to submit complete and validated data to the dataset held by the ‘honest broker’.  Proposals 3 and 4 mean that not a single patient’s data are lost, from the first patient on.  The learning curve can be defined, and confounding variables such as centre or surgeon can be included from day one.  Available data will accumulate rapidly and be open to further units and more sophisticated analysis.  No longer could deaths or complications be hidden, and no longer would the acceptance of great procedures be held back.  Modifications, by innovative ‘followers’ in the second wave, to the procedures or data required could be incorporated into the dataset as variables, and policed by the data monitoring group.

Good examples of honest brokers exist in the form of some Journals (e.g. Nature), professional bodies, and now secure cloud services [Open Science Framework, Open Science Data Cloud].  The NIH has a recommended list of data sharing repositories on its website [https://www.nlm.nih.gov/NIHbmic/nih_data_sharing_repositories.html], but sadly I could not identify any relevant to surgery. The time is right for change.

Proposal 5;

No patient should be allowed to have the innovative procedure, unless they agree to the use of their data in this way.

This may appear a little Stalinist to some, but seems to me to be ‘fair’ to society at large, and would ensure that ALL patients are included in the analyses.  I have yet to meet a patient or family unwilling to share their data in a research study.  They readily recognise the value to wider society of sharing their anonymised data.  They are very often interested in what the data are used for and would like access to the findings as data accumulate. 

Professor Sir Julian LeGrande of the London School of Economics brought to my attention the works of Sir Richard Titmuss, who published a wonderful book called “The Gift Donation” (George Allen & Unwin, London 1970.  ISBN 0 04 301027) arguing for the social benefit of donating blood rather than trading it in a market.  LeGrande has proposed that we see data in the same way; as a social good.  People are likely altruistically to donate their data to a safe and policed depository, as long as it is used for the benefit of others.  I am very attracted by this concept of data donation and I hope that this talk will encourage you to support the movement that is starting around it.  My experience is that patients and their families are very comfortable with the sharing of research data, and that it seems to add confidence to their interactions with us.  It suggests we are being honest in all aspects of our work.

Our peers in other disciplines are way ahead of us with regard to such matters.  As long ago as 2009, your National Academies of Sciences and Engineering, and the Institute of Medicine published an excellent report15 entitled “Ensuring the integrity, accessibility and stewardship of research data in the digital age”.  In that report they defined three core principles, which I summarise here.  They will build trust.

The Data Integrity Principle 

Researchers themselves are ultimately responsible for ensuring the integrity of research data

Data Access & Sharing Principle

Research data, methods, and other information integral to publicly reported results should be publicly accessible

Data Stewardship Principle

Research data should be retained, documented, referenced and indexed for future uses and that others can find them and use them accurately and appropriately.

I don’t have time or space to go into the potential benefits of ‘big data’ analytics, but just remind you that Formula One teams collect, analyse and use over 243 Tb of data (more than the entire National Library of Congress) from one two hour race.  They use these data for continuous iterative design change, strategy decision support and prediction of component failure.  The technology exists, the benefits are clear, we just need access to the data, which must be of good quality.  If we make strong, complete datasets available, the opportunity to benefit from their analysis is obvious.

I am not alone in thinking that biological science publishing is no longer fit for purpose.  Ben Goldacre, who campaigns for openness and better use of data in medicine has written about the need to make raw data available (Make journals report clinical trials properly. Nature 2016.530:7). 

And I reproduce in full here a letter, entitled “Preprint Servers: Vet reproducibility of biology preprints” published in Nature16 from Sir Roy Calne FRS, outlining another approach to transparency.

Posting preprints in online repositories is common practice in the physical sciences and

mathematics. Publication of biological-sciences preprints has been less satisfactory, largely because many are of inferior quality. Addressing the credibility of these submissions could bring free preprint servers such as bioRxiv (founded in 2013; see bioRxiv.org) more into line with arXiv, which has been running successfully for 25 years. Introducing a staging process for submission could rectify the credibility problem. This would establish priority for new work and allow time for it to earn a seal of approval — helping to speed publication after formal submission to a journal.

Initially, an abstract would be assessed by the editors of the preprint server to confirm suitability for posting. Submission of a short form of the paper would follow, with added details of raw data, materials and methods; this would allow other researchers to confirm, refute or comment on the results. Six months later, say, the authors could submit a revised preprint in response to this feed-back, together with evidence to support their preprint’s credibility and reproducibility. 

Alternatively, they could withdraw it with dignity.

Ladies and gentlemen, the current methods of publishing surgical results are outdated, inefficient, incomplete and in most cases little more than stamp collecting.  I hope my proposals might nudge those of you with power to change the system, beginning on the easy territory of innovation, where motivations are strong.  Competition will not be inhibited by these proposals, but as a profession we need to find a way to facilitate career progression based on collaboration and transparency, and not on the weight of ones CV.

Medical publishing is threatened, however, and it should be.  Richard Smith, former editor of the British Medical Journal has written eloquently about this (http://blogs.bmj.com/bmj/2016/04/19/richard-smith-what-are-medical-journals-for-and-how-well-do-they-fulfil-those-functions/). He states “I can’t see journals continuing to make 30% profit margins. Because there are money and jobs to lose and because universities and others continue to misguidedly use publications as a means of judging the performance of academics journals may take some time to fade way.”  He goes on to describe in a later blog http://blogs.bmj.com/bmj/2016/06/15/richard-smith-what-will-the-post-journal-world-look-like/ a ‘post-journal’ world, in which “research will start with the publication on an open website of a funded grant proposal. Anybody will be able to comment, and to avoid repetition I should make clear that everything I mention will be posted online with anybody free to comment. This is open science”.

I wish to make it clear that there are very few really serious liars in medicine, but the current system does encourage economy of truth, and most importantly, as a result of the cycle of reward associated with publication, loses potentially valuable raw data within which there are likely to be many important observations waiting to be made. As the great American jurist, William O. Douglas (1898-1980) said[5], “Sunlight is the best disinfectant”.  We need to heed his words and make our research data accurate, available and analyzable. We would find it much more difficult to fool ourselves and others.

Thank you, ladies and gentlemen, it has been an honour.  The paper will be online for you to tear apart!

References

1. Gross ML. Bioethics and armed conflict: moral dilemmas of medicine and war. Cambridge, MA: MIT Press 2006.

2. King EG, Oransky I Sachs DB, et al. Analysis of Retracted Articles in the Surgical Literature. Academic Surgical COngress. Jacksonville, FL, USA, 2016.

3. Elliott MJ, De Coppi P, Speggiorin S, et al. Stem-cell-based, tissue engineered tracheal replacement in a child: a 2-year follow-up study. Lancet 2012;380(9846):994-1000. doi: 10.1016/S0140-6736(12)60737-5

4. Hamilton NJ, Kanani M, Roebuck DJ, et al. Reply to: "Recent Advances in Circumferential Tracheal Replacement and Transplantation". Am J Transplant 2016;16(4):1336-7. doi: 10.1111/ajt.13736

5. Culme-Seymour EJ, Mason K, Vallejo-Torres L, et al. Cost of Stem Cell-Based Tissue-Engineered Airway Transplants in the United Kingdom: Case Series. Tissue Eng Part A 2016;22(3-4):208-13. doi: 10.1089/ten.TEA.2015.0283

6. Hamilton NJ, Kanani M, Roebuck DJ, et al. Tissue-Engineered Tracheal Replacement in a Child: A 4-Year Follow-Up Study. Am J Transplant 2015;15(10):2750-7. doi: 10.1111/ajt.13318

7. Birchall MA, Elliott MJ, Lowdell M, et al. Stem-cell-based, tissue-engineered tracheal replacement in a child - Authors' reply. Lancet 2013;381(9861):113. doi: 10.1016/S0140-6736(13)60044-6

8. Weiss DJ, Elliott M, Jang Q, et al. Tracheal bioengineering: the next steps. Proceeds of an International Society of Cell Therapy Pulmonary Cellular Therapy Signature Series Workshop, Paris, France, April 22, 2014. Cytotherapy 2014;16(12):1601-13. doi: 10.1016/j.jcyt.2014.10.012

9. Ciralsky A. The Celebrity Surgeon Who Used Love, Money, and the Pope to Scam an NBC Producer. Vanity Fair. New York: Conde Nast, 2016.

10. Serota KB, Levine TR, Boster FJ. The Prevalence of Lying in America: Three Studies of Self-Reported Lies. Human Communication Research 2010;36:2-25.

11. Buxton MJ. Problems in the economic appraisal of new health technology; the evaluation of heart transplants in the UK. Economic appraisal if health technology in the European Community, 1987:103-18.

12. Ferguson C, Marcus A, Oransky I. The Peer=Review Scam. Nature 2014;515:480-82.

13. Blazeby J. Is Buxton's law still true?  Evaluating evolving surgical techniques within pilot and full RCTs. In: Blazeby J, ed. Oxford: University of Oxford IDEAL Collaboration Conference, 2016.

14. Hamdy FC, Donovan JL, Lane JA, et al. 10-Year Outcomes after monitoring, surgery, or radiotherapy for localized prostate cancer. N Engl J Med 2016;375:1415-24.

15. (US) CoSEapp. Ensuting the integrity, accessibility and stewardship of research data in teh digital age. Washington , DC: The National Academies Press 2009.

16. Calne R. Preprint servers: Vet reproducibility of biology preprints. Nature 2016;535:493.

Special Thanks to:

Mr Amman Abid

Professor Jane Blazeby FRCS (Bristol)

Professor Martin Birchall FRCS (London)

Professor Sir Roy Calne FRS (Cambridge)

Professor Pedro del Nido MD (Boston)

Mr Harry Evans (Ipsos Mori, London)

Mr Ewen Harrison FRCS (Edinburgh)

Professor Tian-Yen Hsia MD (London)

Professor Marshall Jacobs MD (Johns Hopkins, Baltimore)

Professor Sir Julian LeGrand PhD, DLitt, FRSA (LSE, London)

Professor Mark Lowdell FRCP (London)

Mr Bosse Lindquist (Stockholm, producer of the documentary “Experimenten”)

Tom Sasse, (Open Data Institute, Bristol)

Dr Richard Smith (former editor of the BMJ)

Peter Wells, (Open Data Institute, Bristol)

© Professor Martin Elliott, 2016

[1] At a University of Buckingham Human Rights Seminar  “On Extremism and Freedom” 18th May 2016 held at the Reform Club, London, UK

[2] The impact factor of a Journal is the number of citations received by articles published in that Journal during the two preceding years, divided by the total number of publications published in that Journal in the same period.

[3] https://en.wikipedia.org/wiki/Citation_index

[4] quoted by Lawton, J.S., AATS Thoracic Surgery News March 24 2016

[5] He may have been quoting Louis Brandeis (1856 -1941) a previous Supreme Court Justice