More concerns on the “Sputnik” vaccine

FOR THE PTS INTEGRITY GROUP:

Prof. Enrico Bucci – Temple University (USA)
Prof. Raffaele Calogero – Turin University (Italy)
Dr. Piero Carninci – RIKEN Center for Integrative Medical Sciences (Japan)
Prof. Pellegrino Conte – Palermo University (Italy)
Dr. Andrea Grignolio – CNR (Italy) 
Prof. Luigi Marchionni – Weill Cornell Medicine (USA)
Dr. Angelo Parini – INSERM (France)
Prof. Gianluca Sbardella – Salerno University (Italy)

Abstract.

• The last paper published on the Sputnik vaccine boasted high efficacy and safety; however, the paper suffers from several shortcomings
• Access to the full data set supporting the published research papers related to the clinical development of the Sputnik vaccine has been so far refused to several independent researchers worldwide (including Russia). The corresponding author Logunov declared that “There are seven billion people on earth, and it’s impossible to present every data point to everyone.” However, as far as it is known, there is actually not a single data access request that has been complied with.
• The peer-review process by The Lancet appears far from accurate, especially in light of errors in Sputnik’s papers that were easily avoidable; far worse, however, is the fact that such a prestigious journal does sacrifice data accessibility, for the sake of publishing a few attractive papers
• In light of the above, while there is a broad expectation that the Sputnik vaccine will show some degree of efficacy and safety, any comparison with other products is precluded, as well as any solid foundation for its evaluation by the scientific community based on the papers published by The Lancet, until a true data disclosure will occur

Introduction.

Very recently, a new paper was published on “The Lancet”, reporting a new set of “ad interim” results for the phase III trial of the Russian “Sputnik” vaccine.

There are several new concerns on this paper, concerns which are widely shared with different colleagues all over the world, which we asked for an independent opinion.

Before entering into details, however, and to avoid becoming a target for Kremlin-controlled media, let us say that we do not think the Sputnik V vaccine’s development and authorization procedure is representative of the general regulatory procedure adopted in Russia; and we must add that this view is shared by the authors of the paper, as will be evident in the following. Thus, we are not criticizing the Russian biomedical science as a whole, but only a specific episode, which should be regarded as an exception – albeit not a justifiable one, not even in a pandemic. For several reasons, connected to an agenda which is not a scientific one, the Russian government decided to gamble on a vaccine before phase III data were available, disregarding regulatory procedures, people safety and scientific transparency and accuracy. Whether this will pay off in terms of epidemic containment – and we think it will, given the overall data on adenoviral vaccines – it is utterly irrelevant to the fact that such a procedure is dangerous and not acceptable, neither in Russia nor elsewhere (there are other few cases).
Moreover, it is evident that Russians were not alone in stretching their data and cutting corners: the truth is that the reviewing process of both papers describing the efficacy of the Sputnik vaccine failed spectacularly. In particular, beside errors and inaccuracies which passed through the peer review process (a fact that could have been still tolerated, if errors were fixed), it is completely unacceptable by The Lancet to plainly allow the authors denying access to the original/raw data for the Phase 1/2 study. As we will see in the following, this peculiar tolerance for lack of transparency seems to be even worse in the last study.

Some open questions and concerns on the last paper.

1. There is a gross lack of transparency. We (and several others in the world, including some scientists from Russia) never got any answers to our data access requests for the previous paper related to the Sputnik V vaccine; in the new article, we found that a “security department” will scrutinize data access requests. Such a declaration is outrageous, both on the side of the Gamaleya and The Lancet.

2. There are four deaths reported. However, sufficient details are given only for two individuals vaccinated and deceased after getting COVID-19. What about the other two deaths?

3. In the paper, we read that the authors enrolled “21,977 adults” but also that the study included “21,862 participants”. Which figure is the correct one? Is this related, at it seems, to the subtraction of some individuals from both the placebo and the treated groups, for reasons which are not entirely clear?

4. As derived from a subsample analysis, 15% of participants in the placebo group had RBD-specific antibodies on day 42, implying asymptomatic COVID-19. How were they considered for the efficacy calculations? Please note that the answer is crucial for calculating the true efficacy of the vaccine, which could easily decrease if those cases were improperly handled, inflating the COVID-positive individuals in the placebo group.

5. In the paper, three preceding interim analyses are cited. In these analyses, efficacy calculations rely on “over 16,000” (11 Nov), “more than 19,000” (24 Nov) and over 26,000 (14 Dec) people vaccinated with two vaccine doses. Why do we find in the published paper less than 15,000 volunteers described as receiving a double dose?

6. In the paper, we read that: “From 21 days after the first dose of vaccine (the day of dose 2), 16 (0·1%) of 14 964 participants in the vaccine group and 62 (1·3%) of 4902 in the placebo group were confirmed to have COVID-19”. However, in figure 2, we read that at day 20, there were 61 confirmed COVID cases among the vaccinated and 30 among the controls. How do these numbers match? And, more importantly: looking at the figure, we find that there are data for 15,717 vaccinated people at day 20, but only 15,338 at day 10. How is it possible that hundreds of people whose data were available at day 20 were not included in the analysis at day 10?

Please note that in light of these objections, the authors’ efficacy estimate cannot be taken at face value, before the actual data and appropriate clarifications are given. Indeed, we learned that about 500 participants in the Phase III trial disclose information online to check independently for vaccine efficacy. About 75% reported antibodies development, after having their blood tested by private labs. If they were all vaccinated, and if 25% were equally tested and did not show enough antibodies, this figure would closely match the overall efficacy of 73.1% calculated by the authors when including all the data, i.e. without discarding all infections occurred before the second vaccine shot.

As for the safety, more details should be given on all deaths during the trials, although it is improbable that there are significant mortality issues with the Sputnik vaccine. Instead, the problem is that, unless we have a valid justification for the very apparent exclusion of several thousands of volunteers, there could be any sorts of issues – including safety issues – not evident until all data are fully available.

More concerns, coming from Russia.

We were not alone to find some discrepancies in the paper published by The Lancet. Vasily Vlassov, Olga Rebrova and Valerii Aksenov, all from the Russian Society for Evidence-Based Medicine, released on February 05, 2021 a commentary (in Russian) which broadly overlaps with our analysis.
They kindly provided us with the following translation, reported below (unedited).

A commentary on the publication of preliminary results of the Sputnik-V vaccine phase 3 trial

Vasily Vlassov, Olga Rebrova, Valerii Aksenov – Russian Society for Evidence-Based Medicine

February 05, 2021

The authors declare that they have no conflicts of interest regarding any COVID-19 vaccines.

Despite the announcement of the preliminary results of a randomized controlled trial (RCT) of the first Russian vaccine in November 2020 and the promise of a future full publication in an international journal to take place in November, the publication took place only on February 2, 2021 (1). Given the tremendous social significance of this development and decisions regarding citizens of this country, we considered it necessary to publish the following commentary on this article.

It should be recalled that the vaccine received Emergency Use Authorization in Russia after reduced phase 1-2 trials involving an unacceptably small number of healthy young volunteers. The publication of these trial results raised doubts about the study’s quality among specialists in Russia and abroad (2). The published corrections to the article, not accompanied by explanations of the origin of the text’s errors, only exacerbated the distrust of the report (3). That is why the results of the phase 3 RCT have long been expected by Russian society.

It is important to emphasize that the other vaccine developers who received temporary approval for their use simultaneously as the developers of Sputnik-V not only have already released the results of the preliminary analysis of their phase 3 RCTs but have also released the trial protocols. The Russian vaccine developers have limited themselves to press releases from the trial sponsor, and the report we briefly comment on is the only source of information beyond the limited RCT registration data on clinicaltrials.gov.

Only from this article do we learn that the protocol reflected in the RCT summary on clinicaltrials.gov (NCT04530396) was revised on November 5, 2020, which is not reflected in this register. We are only told that at that time, it was decided to conduct three interim data analyzes. This article includes the analysis results as of November 24, 2020.

From Fig. 1 (flow chart), we learn that, for various not entirely clear reasons, 74/16501 (0.45%) patients were excluded from the number of those who received the first dose­—in the vaccine group, and 41/5476 (0.75%) patients, in the placebo group. The number of those excluded at this stage is small and could hardly have influenced the result. However, it should be taken into account that the total numbers of participants planned, screened, included in the study, and this analysis are very different. That is, 40,000 were planned for inclusion in the trial; 35,963 were screened; 33,758 were actually enrolled (reported on clinicaltrials.gov, ‘actual enrollment’); 21,977 were randomized; 21,862 were analyzed for the primary outcome (named as such in the register) and for serious adverse events (SAEs); 19,866 for another outcome (quoted in the article as the primary one); 12,296 for adverse events. Thus, the published results are obtained in selective groups, and there are signs of unjustified exclusion of cases.

The article provides data on the immunogenicity of the vaccine. They are derived from an analysis of blood samples from RCT participants, who can be called a ‘convenience sample.’ The authors provide no evidence that this sample is representative of all RCT participants. Accordingly, we refrain from the discussing these results.

Discussing the RCT limitations, the authors note that cases of COVID-19 were detected by self-observation of patients, assessed by a doctor, and confirmed by PCR. Thus, asymptomatic cases were not detected. It can be assumed that there were more such cases in the vaccinated group than in the placebo group. The result of such a measurement of incidence may be an underestimation of the incidence in the vaccinated group to a greater extent than in the placebo group and thus an overestimation of the protective effect of the vaccine. The size of this bias in the vaccine efficacy evaluation cannot be estimated. The problem with choosing clinically detectable disease as the primary outcome is present not only in this trial but also in trials of other COVID-19 vaccines. This is not only a problem of estimation of accuracy. If vaccination leads to a significant reduction in the disease’s severity, then vaccinated people who have a mild infection or those asymptomatic can be active carriers of the infection. In other words, vaccination may accelerate the infection spread rather than slow it down. In any case, like the RCTs of other vaccines already approved on temporary conditions for use, this RCT did not aim to assess the effect of vaccination on the infection spread.

Cases of common cold classified by doctors as COVID-19 were recognized as such and included in the RCT statistics as the primary outcomes only if confirmed by a PCR test. The article does not indicate which tests were used for these purposes and what their operational characteristics are.

The analysis of the results was carried out with a deviation from the protocol, in the part in which we are familiar with the latter from the report on clinicaltrials.gov. The protocol prescribed an analysis of outcomes within 6 months after the administration of the first dose. When analyzed for the timeframe from the administration of the first dose, the effectiveness of the vaccine is 73.1% (as calculated by the authors of the article). The effectiveness of 91.6% is declared and presented in the article’s conclusion as the main result is obtained if the days before the second dose are excluded from the analysis. There are reasons for this variant of the analysis, but these reasons were known as well during the protocol development. The principal analysis should be performed according to the protocol. Deviation from the protocol during the analysis is acceptable, but the results of such an analysis are usually considered as exploratory ones, requiring additional confirmation in a separate experiment.

In this study, the follow-up period for participants varies greatly. Therefore, only the survival analysis can be considered correct, but its results are not presented in the article. Only Figure 2 is shown. Our data reconstruction shows a statistically significant difference between the groups in the log-rank test (P < 0.001). The measure of the magnitude of the effect, in this case, should be the hazard ratio. The calculation of efficacy given in the article appears to be incorrect.

The vaccine is presented as safe. The presented frequencies indicate approximately the same incidence of SAEs with the administration of the vaccine and placebo. The incidence of SAEs (0.3%) is only 10 times lower than the disease incidence (2%), which cannot be considered unremarkable but should not be overestimated since 2% is the cumulative incidence over the analysis period. Over the year, the number of cases should increase, and an improvement in this ratio can be expected.

34% of cases were excluded from the safety analysis. The authors explain this by delaying obtaining information, which does not seem entirely convincing, given the information technologies used and the importance of the issue.

The data on deaths are insufficient. In particular, the ages of the two patients who died from COVID-19 in the vaccination group were not specified. The authors regard these cases as ones included in the trial while in the incubation period of the disease. In our opinion, such a simple interpretation is insufficient. A much more detailed description of these two cases is required.

The follow-up period for the trial participants is too short — the median is 48 days. With the negligibly small sample size of phase 1-2 trial and the phase 3 RCT actually have been completed, this means that the vaccine’s safety beyond one and a half months remains unknown. Before the present epidemic, adenoviral vector vaccines had not been approved in the world, and an abridged trial of a fundamentally new technology intended for mass use does, indeed, cause concern.

When assessing the risk of bias of the results using the Cochrane Risk-of-Bias tool, we find that due to missing outcome data and the authors having a marked conflict of interest, the overall risk of bias of the study results is high.

The article’s text also contains numerous other errors, ambiguities of the text, possible misprints, which we are not discussing here. For example, in Fig. 2, the number of people observed in the vaccination group on the 20th day is greater than that on the 10th day (15,717 and 15,338). It is impossible, just as it is impossible to become a twenty-year-old without previously having been a ten-year-old.

The authors have fulfilled the journal’s condition and declared their readiness to provide access to the original data. However, this time they have locked out the access to the raw data, setting so many conditions, including the approval by the ‘security department,’ that no one will likely turn to them for the raw data. This is rather unfortunate since the lack of readiness for constructive interaction with scientific communities provides grounds for the most serious suspicions. Up to suspicions of data falsification because of unexplained discrepancies or failure to report essential methods or results.

The authors offer readers an interim report for November 24, creating an illusion of the trial being ongoing and the final report to appear on schedule. Meanwhile, it has been known for at least 50 days that the trial is, in fact, interrupted, the placebo group is partially unmasked. This means that no data on comparison between the vaccinated and unvaccinated groups will be obtained any longer. The next report can only contain information about the disease incidence among the vaccinated. This information will be useful, but it should be understood that this interim report is, in fact, the final report on the phase 3 RCT of the Sputnik-V vaccine.

Conclusion

The feasibility of using any medical intervention is determined by the balance of benefits and harms from its use, i.e., its efficacy and safety ratio. An analysis of the phase 3 trial data reported in the publication and the RCT registry leaves both these aspects of this intervention highly uncertain.

References

1. https://doi.org/10.1016/S0140-6736(21)00234-8

2. https://trv-science.ru/2020/11/covid-19-logunov-retraction/

3. https://doi.org/10.1016/S0140-6736(20)32681-7

Hiding the data.

As serious as the criticisms on this and on the preceding Lancet paper on the Sputnik vaccine are, it seems that Russian scientists are unimpressed by the lack of transparency so obvious in the entire clinical development of the Sputnik vaccine.

Apparently, Logunov and their colleagues claim that the exceptional pandemic is a reason sufficient to depart from the standard procedures. He said in a recent interview that “If we have something that is proved to be safe and that has the chance to save a person, it’s unethical not to try and do so“.

However, this is exactly the same fallacious argument which is used in pseudoscience when approaching incurable diseases: given the lack of a cure, anything goes. Logunov seems not to understand that his vaccine was not at all “proved to be safe“, as well as effective, when its use was approved for starting mass inoculations in Russia; Phase III was far to be completed (and will not be, according to the our colleagues from the Russian Society for Evidence-Based Medicine). Proving safety and efficacy is exactly the aim of the standard clinical trials procedures, which require also the proper documentation of all the steps; instead, both the Russian government and the Gamaleya scientists were gambling on the vaccine success, putting at unnecessary risk their own people in the process,by deviating so much from the regulatory procedures.

That is why, in the very same article reporting his words, we found also the words of Svetlana Zavidova, the head of a trade group that represents multinational pharmaceutical companies working in Russia. “Like in a slalom race, you have to pass through certain gates along the way. We decided to just zoom straight downhill so as to save time. We cut past and then said, ‘Now give us a medal.’ ”

Logunov and colleagues seem to believe that having obtained a working vaccine is all what is needed to accept a lack of transparency and a serious disregard for the scientific standards in the vaccine clinical development process during a pandemic.

Indeed, in the same press article, we read the following:

They declined to provide the raw data. Logunov told me that to give such information to anyone who asked for it would be a distraction, and a violation of the norms and practices of modern pharmaceutical development. “There are seven billion people on earth, and it’s impossible to present every data point to everyone,” he said. “No one works this way.”

Now, the fact is that refusing to share the data with colleagues all over the world, when faced with objections to what has been published, is precisely the sort of gross violation of scientific integrity and research ethics that should lead at least to a retraction of any published papers. Of course, no one wants to abuse the right to data or stalk the authors; but several concerns, from several researchers have emerged, and this cannot be hidden. In light of that, it is astonishing that The Lancet not only did not retract the Phase 1/2 paper for not providing access to the data requested to assess several important points, but also accepted another paper from the same authors whose data will likely never be accessible to external researchers. Does The Lancet intend to renovate the old tradition of the infamous Wakefield’s paper on vaccines, which took more than 10 years to be retracted? Is vaccinology a special field of science, one in which researchers are not requested to show their data, if there is well-documented evidence for potential problems?

So what does it all means?

As said by Konstantin Andreev (Northwestern University) when referring to our analysis of the first Sputnik V paper on The Lancet, “We weren’t saying whether the vaccine is good or bad, safe or unsafe. Our objection wasn’t really to the vaccine per se but to how the researchers carried out their study. At minimum, it was sloppy; at most, it was manipulated.”

Uncertainty on the reported statistics, refuse to share the original data, obvious mishaps and potential manipulations plague all the documents describing the results obtained with the Sputnik vaccine. It is all too bad that Russian scientists and the Gamaleya Institute in particular, but also The Lancet, show such a disregard for the very foundations of the scientific procedures. Without data, there is no way to repeat and check the calculations performed by the authors, hindering the replicability of such an important piece of science, and leaving as the only option for scientific replication a duplicate clinical trial.

The Sputnik vaccine will surely work; but the science behind its eventual success cannot remain in a black box. The only hope is that, should the Gamaleya Institute seek the approval of EMA or FDA, regulatory agencies will have access to better data than the scientific community had.