Update on recent case reports of ST including comment on ‘Conundrums in neurology: diagnosing serotonin syndrome – a meta-analysis of cases’
I have long reasoned that the continued publication of case reports of supposed serotonin toxicity (ST), or serotonin syndrome, is almost certain to be scientifically valueless (1). This is because the reliability and information content of case reports is low compared to the validated and replicated data that we already have. I had been considering the task of updating my comments about case reports to confirm the above view — things seem to have become even worse since my editorial 10 years ago (1)— when this (July 2016) meta-analysis of case reports by Werneke et al. (2) landed on my desk. One cannot recommend this paper, but at least it has done some of the work of collating recent reports for me and saved me the tedium of trawling through them all — I confess I have ceased to ‘log’ all the ST reports in my bibliography database for the last few years because it became such an unproductive use of time and energy. It is also dispiriting — some good work has been published, but it becomes lost in the tsunami of mediocrity and has so little effect on the collective knowledge-level and understanding evinced in case reports.
The ‘meta-analysis’ (it may be stretching the point to justify calling this a meta-analysis) by Werneke et al. seeks to use more recent case reports (2004 to 2014) to ‘challenge’ the ‘textbook knowledge’ about ST (more on their misuse of words below). However, this exercise merely demonstrates that applying ‘meta-analysis’ to case reports, all of which are of low observational quality, simply compounds the errors, of which there are many. Bad data cannot negate good data. Meta-analysis is not capable of transmuting base lead into gold. If everyone was adequately educated in the scientific method, objective and rational that would be the end of the discussion. Evidently, they are not.
My concise advice to readers is do not waste your time reading the Werneke meta-analysis – you will not learn anything useful from it: rather, you will be mis-informed and confused. Instead, read this excellent recent update on ST by a group of informed and experienced toxicologists (3). The full free pdf of this paper is here
If you wish to understand more of how not to do science, and maybe use Werneke et al. as an example of bad science if you are teaching, then please read on. There is sufficient to criticise in this to keep a tutorial group of honours students busy for a month.
Not only is the Werneke et al. paper pointless, but worse, it adds to the errors and confusion in the literature that in turn has adverse consequences for ordinary doctors trying to engage in sensible patient management of cases that they encounter. This paper just adds to the mass of misinformed opinion and makes it more difficult to find that which is good (ironically, finding ‘that which is good’ is exactly what these authors have conspicuously failed to do).
There are so many misconceived and incorrect presumptions and statements in this report that there is an embarrassment of choice about where to start one’s comments in order to help general readers understand how deeply flawed it is.
This ‘meta-analysis’ audaciously claims to contradict much of what has become established over the last twenty years. It must perforce receive some attention beyond insouciant dismissal, which is all it warrants.
The first thing for readers to be mindful of is that the Hunter criteria — the only consequential target of their ‘challenge’ — have been developed from an enormous consecutive series of overdoses (of all sorts and causes) presenting to a regional toxicology service and all examined by experts in toxicology. Although classified as ‘overdoses’ a small but significant proportion of the cases (in the reference cited) would be definable as ‘high therapeutic doses’ rather than overdoses.
Werneke et al. make various key errors in both their reporting and understanding of the ‘Hunter’ research data. These errors invalidate their criticisms. I will start with these examples:
‘Yet, the purported HC superiority is based on one study only.’
‘One concern regarding validity is that HC was derived exclusively from SSRI overdoses’.
… a proportion of the cases used to derive HC was then also used to validate HC.
These statements are erroneous: one can hardly suppose Werneke et al. actually read the paper they cite (4), because it states clearly:
‘A learning dataset of 473 selective serotonin reuptake inhibitor (SSRI)-alone overdoses was used to determine individual clinical features predictive of serotonin toxicity by univariate analysis. Decision rules using CART analysis were developed, and tested on the dataset of all serotonergic overdose admissions.’
So, not ‘derived exclusively from SSRI overdoses’.
In fact, derived from all different classes of drugs in small and large doses and all degrees of severity of ST starting from the odd shake and twitch through to near-fatal cases requiring IC admission and care. This discrepancy between Whyte’s publication and the impression and account that Werneke et al. give of it is extensive. Can it possibly be due solely to careless scholarship?
The Hunter toxicology group, formed by Prof Whyte, has been keeping a detailed prospective database of all toxicology cases for some 30 years. This has enabled a series of seminal papers on many aspects of toxicology, not just ST.
For Werneke et al. to appear to diminish or dismiss this massive achievement with the ill-informed comment that it is ‘one study only’ is breath-taking hubris.
The ‘Hunter’ publications about ST (there are a number that Werneke et al. do not cite — and have probably not read) encompass all ranges of severity of ST, including potentially fatal toxicity from combinations of MAOIs and SRIs. Werneke et al. have obviously not read and understood the oeuvre of Prof Whyte’s ‘Hunter’ group. Their scholarship is lamentably deficient for those who make such presumptuous refutations.
‘Our findings challenge four commonly made assumptions about SS’
I think not.
Whyte’s paper (4)also clearly states:
Six patients were intubated solely for worsening serotonin toxicity. All of these patients had a high fever [> 38.5_] and multiple features of serotonin toxicity. Review of these life-threatening cases showed that progressive rigidity compromising respiratory function was the precipitating event for intervention in these patients. The preceding signs were a high fever (> 38.5_) and increasing (particularly truncal) rigidity and peripheral hypertonicity (5)***.
*** Needless to say, these were all MAOI/SRI interactions, but Werneke et al. clearly did not understand that point and did not look at the reference(5) to the other publication of Whyte et al. Some scholarship. Some understanding.
That paper, Isbister 2003 (5), reports in more detail on those severe cases, and others, in a larger series of severe cases of ST specifically caused by an MAOI/SRI interaction.
So, it is assuredly not the case, as these authors carelessly and mistakenly contend, that the ‘Hunter’ criteria have been derived from a specialised subset of patients (‘derived exclusively from SSRI overdoses’) and that they therefore do not represent the drugs, combinations and degrees of severity, that have been shown to precipitate ST. These points are crucial in understanding ST and for their argument: they have got it badly wrong.
My ‘MB exemplar’ review also contains a summary of Hunter data illustrating degrees of severity of ST seen with different drug classes and combinations, see especially Fig. 3 (6).
Also note, the toxicologists who developed the Hunter criteria have seen and cared-for many other cases of ST caused by MAOIs and SRIs and also many cases of neuroleptic malignant syndrome. They are experts who are fully conversant the whole range of severity of presentation of both these conditions, so their opinions are to be taken seriously. You might wonder how the clinical experience of ‘Werneke et al.’ stacks up in comparison?
Few scientists who understand clinical medicine will give weight to case reports (most authored by doctors, and even non-medical people, of ‘uncertain’ expertise and experience), and the conclusions drawn from them by persons inexperienced in the field, in comparison to the Hunter groups’ data and expert experience: bad data cannot negate good data.
Severe life-threatening cases of ST are caused (almost exclusively) by the co-ingestion of SRIs in conjunction with a monoamine oxidise inhibitor. That fact has been exhaustively document over two decades, yet these authors (and the referees) appear quite oblivious to that. Such cases are now rare (cf. the MB story (6)): but they are predictably severe and life-threatening.
It may be noted that the discovery of the MAOI properties of methylene blue (MB) was entirely due to my confidence in the predictive validity of the spectrum concept of ST that allowed me to persuade the biochemists to find the research money to assay MB in order to establish its’ MAOI potency (7). And indeed, the same process of logic has more recently established the MAOI properties of metaxalone, this time using in silico methods: see here for that story.
Contrary-wise, overdoses of SRIs (combined with almost any drug, other than an MAOI) causing ST are only of mild to moderate severity and are not life-threatening. Therefore, the following paragraph from Werneke et al. can be seen to exemplify a profound failure of understanding:
‘Clinically, particularly when a condition is life threatening, it may be better to err on the side of caution and temporarily withdraw a purported offending agent, until the differential diagnosis is clarified and appropriate action can be taken. The alternative of refusing*** to take into account symptoms because they do not meet HC and continuing a potentially harmful agent seems less safe.’
NB *** ‘refusing …’ Who do they suppose is doing this refusing? This is a classic straw man argument, albeit a rather pathetic one. And the ‘gold standard’ is the diagnosis of a clinical toxicologist, no-one is a slave to research diagnostic criteria.
If any readers thought these authors were serious intellectual disputants, I hope I have disabused them of that idea by now.
The idea that less severe cases of ST precipitated by drugs such as SRIs can somehow mysteriously progress to life-threatening ST is a misunderstanding emanating from ignorance of basic facts, exhibited by many authors. Severe ST precipitated by SRIs does not occur and has not been reliably documented. Hence becoming concerned that mild-to-moderate ST cases (precipitated, typically, by SRIs) represent some kind of incipient danger demonstrates a fundamental misunderstanding of the whole ‘spectrum concept’ of ST and of the ‘ceiling’ effect exhibited by each drug class (see ‘MB exemplar’ paper). This is why it is useful to understand that the consequences of serotonin elevation by drugs are more usefully considered as a toxidrome, not a syndrome (i.e. it is not idiosyncratic, it is a predictable dose-related phenomenon).
A further key point is that the whole concept of ST, and the relationship between severity of signs, degree of elevation of serotonin and the potency & interactions of the drugs causing that, has been well-established in a large number of experiments using in vitro Human Cloned Receptor (HCR) assays, animal models, as well as human data (of various kinds). This enables very confident and clear statements concerning drugs which can and cannot raise levels of serotonin in the brain and therefore which drugs are, or are not, capable of inducing substantial serotonin elevation, or even toxicity. This is what gives the construct of ST an almost unassailable level of external and predictive validity. The comparison Werneke et al. make to a psychometric rating scale is incomprehensible and speaks to their level of understanding of science.
One cannot induce strychnine poisoning with vitamin C, and likewise one cannot elevate brain serotonin levels with drugs that do not affect serotonin. End of story.
We are talking about science, not ghost hunting, which is what these authors appear to be engaged in with their spotting of supposed ST cases detailed in their supplementary list of references, the diagnostic reliability of which is exceedingly low (no matter what ‘criteria’ one retrospectively applies).
I know many of these cases, I refereed some of them, and I can verify that a substantial proportion of them do not meet the criteria for ST — quite a number of them do not even involve drugs with serotonin elevating properties (e.g. triptans, ‘setrons’, trazodone, mirtazapine etc.) so they are, without question, false positives [Quick reminder — all three criteria specify ingestion of a drug with known ‘serotonergic’*** properties]. Indeed, only a fraction of these cases could be rated as ‘definite’ ST. The methylene blue story illustrates all these points very well and I strongly recommend to those who wish to learn about ST that they read Gillman 2011 (6), and or the introduction to ST here.
*** NB ‘serotonergic’, is a much misused word.
Werneke et al. actually cite a paper that is a good example of how common false positives are: the example involves the old drug nefazodone (8). Like trazodone, it has no SERT potency and has never caused ST, or even serotonin-mediated side effects, in overdose. Yet, in this paper it was ‘shown’ to cause more ‘SS’ than venlafaxine or several SSRIs — clearly utterly nonsensical.
The whole scientific basis of ST, from the pharmacology of the drugs involved, the magnitude of their effect on SERT in vitro, serotonin levels in the animal brain, and the symptoms associated with that in both animals and humans, are all firmly scientifically established (9). To imagine that a series of uncontrolled case reports, that by their nature are selected, retrospective and of variable, usually poor, reliability, can possibly contradict all of this is illogical and unscientific.
It is difficult to comment on the effort of these authors without provoking discombobulation. The referees’*** poor and perfunctory reviews of the paper reveal the deficiencies in their own understanding, and in their degree of application to the task they voluntarily undertook.
*** It is relevant to be aware of the background (scientific and medical) of doctors who publish scientific papers. I will therefore make two brief comments: the main author of the paper is a doctor who has no apparent experience of seeing or caring for ST/NMS cases in a hospital or ICU setting, is not a toxicologist and who has no expertise in the area of pharmacology or ST. Also, the journal publishing the Werneke et al. paper engages in open peer-review and one can see the comments of the reviewers. One of them (Prakash) is the author of the following paper (concerning ST). Anyone with a simple understanding of scientific method can see from the link provided to this paper that it is of minimal value and certainly does not qualify Prakash to referee other works on this subject as an ‘expert’.
Referee 1— Prakash. Here is the reference to this referee’s paper about ST (10); txt at:
The fact that the journal editor — who needs to think more about ethics and perspicacity — selected such an author to referee this paper illustrates that many journals have descended into a parody of the refereeing system where the blind are leading the blind. It is very clear that many journal editors make little effort to ensure the referees who they recruit to review papers have appropriate expertise in the field (you really should see the many ridiculously inappropriate requests that I get). Editorial standards have been massacred by the maw of commercialism.
There are too many journals publishing too much third-rate material refereed by people who are not adequately expert in the fields concerned: that is turning much of the scientific publishing enterprise into a farce [but sure is generating increasing profits for publishing companies and forcing libraries to pay more money for less quality].
These authors repeat mistakes made in the body of their text in their conclusions. I will pick just one example (I am wearying of this task) of their careless and faulty thinking to illustrate my point (the text below is not a mistake, it is exactly as rendered in their paper).
‘Fever is considered a hallmark of SS and hyperthermia. To be more precise, a temperature > 41.1 °C, a hallmark of severe SS (11).’
This is confused English and confused thinking, to the point of being devoid of useful meaning.
Fever (pyrexia) and hyper-thermia are quite different (other people confuse them too, but that is no excuse). A similar confusion appears earlier in their text, so this cannot be put down to a typo. A more detailed examination of elevated temperature, and the distinction between fever and hyper-thermia is in various sources e.g. Gillman 2010 (12). At a true core temperature of 41°C irreversible cell damage is in well progress and death is imminent (12): their figure of 41.1°C has, in this context, an absurdly false degree of precision. Serious hyper-thermia does not have a universally accepted definition but it has been argued that 39°C or higher is appropriate. Here is what they say earlier in their text:
We defined fever as a temperature > 38 °C (100.4 °F) (13) and hyperthermia as a temperature > 41.1 °C (106.0 °F) (14).
One cannot ignore this odd statement, so I must give some space to explaining temperature measurement, since it is such a vital defining feature of ST and is the ultimate cause of death in ST. Yet it is measured in the most casual and unscientific way in almost all reports, except those that involve patients in intensive care units. The site, type of instrument used, and the number of elevated measurements (and over what time period) are almost never presented (15), which of course they should be. So much for ‘science’. In fact, I cannot remember seeing or refereeing an ST report in the last 15 years that reported temperature properly.
These authors take the abuse of temperature considerations to a new level by inventing their own definitions of fever and hyper-thermia and justifying them with altogether inappropriate references from Sclar & Sternbach (13, 14). Sternbach is a misplaced reference, it says nothing about hyperthermia being > 41.1°C (106°F), even if it did that would not be relevant, because it is not a paper that considers that question in any depth. The Sclar reference does not even mention temperature! Gillman 2010 discusses this, with appropriate references (12).
Werneke et al. contains a number of other instances of misquoted or misinterpreted references. That is a very serious academic failing. Repeatedly citing papers that do not support the material they relate to, or are irrelevant, shades, at some point, from carelessness to deceit and fraud. Some might say ‘J’accuse’.
However, I will simply note these errors obviously reflect on their general level of poor scholarship.
Incidentally, the Hunter database now has more than 5,000 (five thousand) SRI overdoses documented in it, the last published update (not cited by Werneke) was in 2015 (16). None of these cases have developed a temperature greater than 38.5°C, or been rated as more than mild, or occasionally moderate, severity [Prof Whyte, personal communication: 27/7/2016].
There are few paragraphs of their manuscript that do not invite significant criticism. I will just add one last comment (I have to stop somewhere) on the section sub-titled ‘Is there a gold standard for diagnosing SS?’ which opens:
Rather than being a tangible physical quantity such as body temperature or blood glucose, SS is an abstract construct made up of various conceptual, elements (items). In this way, the three classification systems are similar to a psychometric scale that might measure a construct such as quality of life. … In the case of SS, we measure CNS hyper-excitability and try to relate this to a purported drug-induced serotonin excess.
A ‘purported’*** drug-induced serotonin excess? Have we slipped into an alternative post-modernist reality?
There is nothing purported about it. The fact of elevated serotonin and its consequences is reliably established by a lot of good science (as outlined above), so it is nothing remotely like a psychometric rating scale: it has massive and indestructible external validity, predictive validity and objective signs and …). Werneke et al.’s above paragraph is complete and utter nonsense. ‘Rather than being a tangible physical quantity such as body temperature …’. That is exactly what severe ST is — a potentially fatal hyper-thermic state.
*** One must observe that in their paper they repeatedly use words in a value-laden or misleading way — ‘assumptions about SS’ (six occurrences), to describe deductions and conclusions based on good evidence. Like-wise their use of ‘purported’ (six occurrences), ‘claim’, ‘refusing’. I will leave the examples there, but check them out, you may get the impression of low objectivity and immature attitudes, as I do.
I can only suppose that these authors do a different kind of science to me, and I hope, most of my readers.
And, perhaps worst of all, the referees, who should be ashamed of themselves, have not picked up on any of the above, nor indeed on the many other problems with this paper.
I suggest you remember this paper as a supreme example of ultracrepidarian bloviation. And that you remember the admonition ‘caveat lector’.
November 2016: Incidentally, I should add an explanation about the publishing journal for this article, B M C Neurology.
Previously they had a facility, as any proper scientific journal must have, for post-publication comment about papers. I tried to respond to this paper on their website just after it was published, the ‘comment/response’ facility was non-functional. I emailed them about this and they promised it would be restored with their updated website within a short time.
Here we are six months later and it is still not possible to make a comment about this paper. Many scientists would consider that fact disqualifies the journal from describing itself as a scientific journal because the whole process of science is built around refutation of published work and comment and criticism thereof.
There are 1001 ways in which the commercial imperatives for publishing enterprise are corrupting the values of good science and it becomes wearying to repeatedly comment about them. Nonetheless, it behoves all readers to appreciate the seriousness of this problem and how it diminishes the value of so many published papers.
1. Gillman, PK, Extracting value from case reports: lessons from serotonin toxicity. Anaesthesia, 2006. 61: p. 419-422.
2. Werneke, U, Jamshidi, F, Taylor, DM, and Ott, M, Conundrums in neurology: diagnosing serotonin syndrome–a meta-analysis of cases. BMC Neurology, 2016. 16(1): p. 1.
3. Buckley, NA, Dawson, AH, and Isbister, GK, Serotonin syndrome. BMJ, 2014. 348: p. g1626.
4. Dunkley, EJC, Isbister, GK, Sibbritt, D, Dawson, AH, et al., Hunter Serotonin Toxicity Criteria: a simple and accurate diagnostic decision rule for serotonin toxicity. Q. J. Med., 2003. 96: p. 635-642.
5. Isbister, GK, Hackett, LP, Dawson, AH, Whyte, IM, et al., Moclobemide poisoning: toxicokinetics and occurrence of serotonin toxicity. Brit J of Clin Pharmacol, 2003. 56: p. 441-450.
6. Gillman, PK, CNS toxicity involving methylene blue: the exemplar for understanding and predicting drug interactions that precipitate serotonin toxicity. J Psychopharmacol (Oxf), 2011. 25(3): p. 429-3.
7. Ramsay, RR, Dunford, C, and Gillman, PK, Methylene blue and serotonin toxicity: inhibition of monoamine oxidase A (MAO A) confirms a theoretical prediction. Br J Pharmacol, 2007. 152(6): p. 946-51.
8. Mackay, FJ, Dunn, NR, and Mann, RD, Antidepressants and the serotonin syndrome in general practice. Br. J. Gen. Pract., 1999. 49(448): p. 871-4.
9. Gillman, PK, A review of serotonin toxicity data: implications for the mechanisms of antidepressant drug action. Biol Psychiatry, 2006. 59(11): p. 1046-51.
10. Prakash, S, Patel, V, Kakked, S, Patel, I, et al., Mild serotonin syndrome: A report of 12 cases. Ann Indian Acad Neurol, 2015. 18(2): p. 226-30.
11. Boyer, EW and Shannon, M, The serotonin syndrome. N. Engl. J. Med., 2005. 352(11): p. 1112-20.
12. Gillman, PK, Neuroleptic Malignant Syndrome: Mechanisms, Interactions and Causality. Mov. Disord., 2010. 25(12): p. 1780-1790.
13. Sclar, DA, Robison, LM, Castillo, LV, Schmidt, JM, et al., Concomitant Use of Triptan, and SSRI or SNRI After the US Food and Drug Administration Alert on Serotonin Syndrome. Headache, 2012. 52(2): p. 198-203.
14. Sternbach, H, The serotonin syndrome. Am J Psychiatry, 1991. 148: p. 705-713.
15. Gillman, PK, Neuroleptic malignant syndrome, poor science and inaccurate measurements. J Psychopharmacol (Oxf), 2010: p. 20 May 2010, 10.1177/0269881110367461.
16. Buckley, NA, Whyte, IM, Dawson, AH, and Isbister, GK, A prospective cohort study of trends in selfpoisoning, Newcastle, Australia, 1987–2012: plus ça change, plus c’est la même chose. Med. J. Aust., 2015: p. https://www.mja.com.au/journal/2015/202/8/prospective-cohort-study-trends-self-poisoning-newcastle-australia-1987-2012-plus.