Behind the hype – design flaws, poor execution, and questionable publication
The title for this post comes from an important letter which has been just published in Russia, once more showing how flawed was the study published by The Lancet on the Phase 1/2 trial for the “SputnikV” vaccine.
The authors are Vasily Vlasov, Doctor of Medical Sciences, Professor of the National Research University Higher School of Economics, Olga Rebrova, Ph.D., Professor of Pirogov Russian National Research Institute of Medical Sciences, and Valery Aksenov, Candidate of Medical Sciences, scientific editor of “Bionica Media” publishing house. Vlasov and Aksenov are Vice-Presidents of the Russian Society for Evidence Based Medicine.
As in the case of our multiple requests for access to the primary data of the study (which was rehiterated 4 times, as of today), the authors were stunned by the lack of any response by Logunov et al.; I must here add that several other researchers from all over the world experienced exactly the same lack of transparency and answers by the Gamaleya Institute, including some high-profile scientists affiliated to the Russian Academy of Sciences.
Interestingly, Vlasov, Rebrova and Aksenov add further concerns to what was already pointed out and asked The Lancet for a retraction of the paper by Logunov et al.
At this point, it is very difficult to understand the lack of any meaningful editorial reaction by The Lancet, which should follow the COPE and its own procedures in dealing with the lack of access to the primary data of a study; it is dismaying to see that the editorial board seems not to be concerned.
In the following, I attempted a translation of the letter by Vlasov, Rebrova and Aksenov (originally in Russian here); we should all help the Russian scientific community, in its efforts to re-establish the scientific ethic standards needed for vaccine development.
EDIT, December 2020: I HAVE NOW A MUCH BETTER TRANSLATIO OF THE LETTER; PROVIDED BY THE AUTHORS THEMSELVES, WHICH REPLACED MY ORIGINAL TRANSLATION IN THE FOLLOWING.
Behind the hype are design defects, low study quality, and questionable publication
(published by Troitsky Variant – Science (TrV-Science) on November 16, 2020 https://trv-science.ru/2020/11/covid-19-logunov-retraction/ )
In October, in a letter to the Lancet, we requested a retraction of the article by D. Yu. Logunov et al.  that reported a study on a COVID‑19 vaccine developed by the Gamaleya Research Institute. The reason for our letter was that the authors had not responded to our request for sharing individual patient data of their study, as was promised in their data sharing statement in the article. Of course, they could have refused to provide the data, but we have not had the privilege of receiving any answer. We consider this a gross violation of publication ethics that, according to the Lancet’s editorial policies adhering to COPE guidelines , warrants a retraction of the article. Our letter has not been published, while events related to vaccine development are unfolding rapidly. Vaccination of the population in our country has been started outside of clinical trials based on the results of , which requires explaining our concern. Below we provide an analysis of the study’s flaws based on the published report and its supplementary appendix.
- The inclusion criteria for patients in the study were negative tests for antibodies and virus RNA. The product РЗН 2020/10393 was named as a test system for determining IgG, but the other two products were not designated. Probably, the virus RNA was determined using the Gamaleya Institute РЗН 2020/10550 product. No test for IgM determination is reported. The diagnostic accuracy of individual components of the test is unknown. Also, the overall diagnostic performance of the test suite is unknown. The preanalytical parameters (probe material, biomaterial, sampling technique) are also unknown while affecting the test’s diagnostic accuracy. Therefore, it is not known how many test results could have been false-negative. When re-tested, such cases could appear to be true-positive.
- As a result of screening, out of 120 volunteers, 100 were found to be eligible, but only 76 were included in the study. No comments nor the full study flow chart are presented. The methods of group formation in both phase 1 and phase 2 are not described. This can cause selection bias.
- The primary outcome of efficacy is surrogate (seroconversion). Its association with the clinical outcome (protective properties of the vaccine) is unknown. The required seroconversion level has not been set. The result obtained for this outcome is 100% seroconversion. However, for sample sizes of 20 patients, 95% CI is 83%-100% and is unacceptably wide.
- The primary safety outcome is the number of patients with adverse events (AEs). AEs are not specified, and how many (or what proportion of) patients with AEs will be sufficient to prove vaccine safety is not declared. There is no result of the primary safety outcome in the article, only rates of AEs are provided, which vary from 0% to 100%. But the most important is that the probability of missing serious AEs in samples of 20 patients reaches 17%.
- The text describing the calculation of the sample size is incomplete. The sample size was calculated for the expected seroconversion of 99%, but other parameters are not provided. With a calculated patient population of 16, the 95% CI for 99% seroconversion is 83% -100%, and such an estimate seems unacceptably uncertain.
- The sample of 4817 patients, whose convalescent plasma was used for external comparison (with mild or moderate illness, plasma was taken one month after “recovery”), has not been adequately described. The number of seronegative and seropositive cases is not indicated.
- There are issues with data analysis, among them:
- Tables 1, S9-S14 present parametric descriptive statistics (means, SD, SEM) for small samples although the distributions are obviously not Gaussian;
- If one reproduces raw data based on Fig. 2 and Tables S3-S6, then 14 of 46 geometric means will differ from presented in Tables S3-S6 and the text. For example, for the Phase 2 trial of frozen preparation, the GM of IgG titer on Day 28 is 3442 in the text and similar in Fig. 2-B.
- Repeated measures ANOVA has not been performed in analysis, as would be preferable for such design, and no corrections for multiple comparisons applied.
8. The authors discussed several limitations in the article. Among them, young participants’ age (means from 25 to 31, no information on maximum ages in the publication). The declared inclusion criteria for age was 18-60. No explanation is presented why the older subjects were not included. Unethical inclusion of military employees who are simple to recruit is the probable cause. The next discussed limitation is the small sample size. It contradicts the authors’ statement that they calculated a sufficient sample size when planned the study.
We believe that low methodological quality, defects in the design, and errors in the application, analysis, interpretation, and reporting data analysis call into question the study conclusions about the vaccine’s safety and efficacy. Our findings are based on a review of the published article, while an independent analysis of the raw data could provide answers to some of the issues discussed above. However, the authors do not provide access to them contrary to their stated obligations. This is an additional argument for the article’s retraction. The fact that The Lancet does not retract the paper and limits its criticism may be due to this journal’s recent painful experience related to the publication of a report on hydroxychloroquine’s effectiveness, describing an analysis performed on an extensive database that did not exist . That article had to be retracted, and the editorial policy’s inadequacy was recognized, including concerning obligations to provide access to primary data. And now the second fiasco of the Lancet this year …
When we were drafting this article for TrV-Science, Pfizer and BioNtech reported the first interim efficacy analysis of their ongoing COVID-19 vaccine trial. The analysis evaluated 94 confirmed cases of COVID-19 in trial participants and showed the vaccine efficiency of 90%. One day later, the Sputnik V vaccine developers announced that their product had an efficiency of 92% . This report is also based on an interim analysis of data from an ongoing trial. However, it is essential that an interim assessment of efficacy was not stated when the trial was registered. Besides, an efficiency of 92% was obtained by calculation based on a total of 20 cases. At the time of analysis, the study participants received only the first dose of vaccine. This is completely unacceptable, even ridiculous, political action to impose competition between vaccine developers in violation of the most critical research standards. The creation of a race atmosphere makes the Sputnik V developers hostage to their careless statements and threatens the integrity of research on vaccines, and, therefore, poses a threat to the health of Russians.
Olga Rebrova, Ph.D., Professor of Pirogov Russian National Research Institute of Medical Sciences.
Valery Aksenov, Candidate of Medical Sciences, scientific editor of “Bionica Media” publishing house.*
*The authors are Vice-Presidents of the Russian Society for Evidence Based Medicine (osdm.org).
- Publishing Excellence https://www.thelancet.com/publishing-excellence
- Denis Y. Logunov, et al., Safety and immunogenicity of an rAd26 and rAd5 vector-based heterologous prime-boost COVID-19 vaccine in two formulations: two open, non-randomised phase 1/2 studies from Russia», The Lancet. Volume 396, Issue 10255, P887-897, September 26, 2020 (Published Online September 4, 2020). https://doi.org/10.1016/S0140-6736(20)31866-3. DOI 10.1016/S0140-6736(20)31866-3.https://www.thelancet.com/article/S0140-6736(20)31866-3/fulltext Accessed 13.11.2020
- The Editors of the Lancet Group Learning from a retraction. Volume 396, ISSUE 10257, P1056, October 10, 2020 September 17, 2020DOI:https://doi.org/10.1016/S0140-6736(20)31958-9 (https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)31958-9/fulltext). Accessed 13.11.2020
- The effectiveness of the Sputnik V vaccine against coronavirus was 92% during the first interim analysis of data from phase III clinical trials in the Russian Federation. https://sputnikvaccine.com/rus/newsroom/pressreleases/effektivnost-vaktsiny-sputnik-v-protiv-koronavirusa-sostavila-92-v-khode-pervogo-promezhutochnogo-an//. Accessed 14.11.2020
4 pensieri su “Behind the hype – design flaws, poor execution, and questionable publication”
Davvero interessante, grazie mille. Purtroppo ormai la polarizzazione della discussione su “vaccini sì” o “vaccini no” (come, ahimè, il tifo da stadio su qualsiasi argomento) lascia poco o nullo spazio ad analisi dei dati fatte con competenza e discussioni produttive, ma mi delude molto vedere come il Lancet, che dovrebbe essere avulso da dinamiche di questo tipo, si stia comportando in questa situazione.