The Nature of Scientific Evidence

The questions that I get asked via the website indicate that some information and discussion concerning the nature of scientific evidence and how to acquire it will be generally useful. The pursuit of scientific truth is a noble endeavour, but it is beset by the human foibles that affect all the areas of our lives. Elsewhere I have recommended the excellent book by Lewontin called “Biology as Ideology” {literal}{Lewontin, 1991 #9171}{/literal} which sets out, inter alia, the misconceptions and myths surrounding the human genome project. Lewontin mounts a powerful argument that the dominant social and economic forces in society determine what scientists do and how they do it. Stephen Jay Gould described Lewontin as “the most brilliant scientist I know” and said that his work embodies the very best in genetics, particularly in the way it debunks the reductionism of genetic determinism. That phrase 'the reductionism of genetic determinism' means, for those not familiar with the language of logic and medicine, the overly simple idea that your genes determine what happens to you. It's very important, because most of the genetic tests, now being offered directly to the public in the USA today, are of limited use. Indeed, most are, for all practical purposes, useless. The assumptions underlying them are reductionist and deterministic in a way which invalidates them. That is why most genetic tests for the way you metabolise drugs are uneconomic, unreliable, and a waste of time and money.

Science: a work in progress

Science is about ongoing investigation; it has always been, and will always be, a work in progress. Almost nothing is right or wrong, or ‘proved’. Whenever you see the word proved being used in science a red flashing light should come on in your mind. If you find an example of it in my writing let me know, because I should almost certainly expunge it.

Science tries to make increasingly accurate representations of, and predictions about, the world, as interpreted by the observations that we are able to make, imperfect as those may be. The group of people that are now generally regarded as cranks, and called the “flat-earthers” are only wrong in a limited sense. We all operate on the assumption that the earth is flat for most of our everyday activities. The block of land on which a house rests, and its construction, operate on the flat-earth principles. We do not take into account the curvature of the earth. It is only when considering things on a larger scale that the curvature of the earth becomes important.

Medical science is a difficult enterprise. I am often critical of the science in medicine, and in psychiatry in particular. That it often reflects (among other things) the enormous difficulties of research in the medical sciences which are so dependent on variables and parameters that are difficult to assess and measure.

Accessing Scientific Information

It is now easier for non-medical and non-scientific people to access scientific information. This may come in many forms, from information on educational and government websites through to scientific papers that may be available free on the Internet. Many scientific journals require payment for reprints of articles (usually $25-40 per item, even if it is only a short letter), an increasing number of papers are free, and some journals make selected papers available free because they perceive it to be in their interests. For instance, my review of the anti-migraine drugs (triptans) and their ability to precipitate serotonin toxicity was published a year or so ago, and has just been made freely available on the journal website. The stated reason is because it is a frequently highly downloaded paper. I would like to believe that is true.

Wikipedia is often a useful source of information, even though it must not be regarded as gospel. Below are a couple of links demonstrating how much effort is going in to coordinating information about open access publishing in science. In this connection one must also mention the open access journal called on the Internet called PLoS Medicine (PLoS= Public Library of Science) which was founded by several influential people, including Nobel prize winners.

http://en.wikipedia.org/wiki/List_of_open-access_journals

http://www.freemedicaljournals.com/index.htm

Two sources of information concerning papers published in the peer-reviewed scientific journals are particularly helpful. The first is the National library of medicine database that is freely available to anyone worldwide. This database registers an enormous number of scientific journals in the medical and biological sciences (but not things like geology and astronomy etc). It is possible to go on to that website and search using all of the details that one would normally expect in a database: by that I mean of the year, volume, page number, keywords, journal names, words in the abstract etc.

http://www.ncbi.nlm.nih.gov/pubmed/

It does not take much imagination and enterprise to learn how to find quite a lot of information. However, it is well worth remembering that even sophisticated searches on such databases, performed by experienced librarians, can only be expected to find about half the relevant information that exists. That means that if somebody does a search for serotonin toxicity on such a database, and then compares what they have found with an expert such as myself, they will find I have about twice as many references. It also means that if somebody does a search for my papers they have to be a little bit careful, because my initials are PK, but I use my middle name, Ken. The inflexibility of many of the journal websites, in relation to entering your details as the author of a paper, mean that kind of irregularity is not coped with very well. As a result of this my name appears in various different ways in the database, meaning you have to search for Gillman, K Gillman P Gilman PK etc. Then of course also you find papers by people who aren't actually me. That is because, unfortunately this system still has no unique identifier for authors.

The second database that has been more recently available is Google scholar. If you look at the various subsections of Google you will find a special section ‘Google scholar’ which is an invaluable resource and covers many scientific journals not included in the NLM database.

Bibliography databases

Organising and accessing large numbers of scientific studies is much facilitated by using a dedicated bibliography database. I could not manage without one. There are various commercial ones available (I use 'Endnote'), and a Google search will soon show the variety available. Such tools allow one to interact easily and seamlessly with databases such as the NLM, and to store and organise information. (That means less fiddling around with cutting and pasting information into fields, because it’s all done automatically. A great help with my bad neck). For those who use Firefox as a browser there is a fantastic add-on called Zotero, which is free and performs as much, and more, compared to be commercially available dedicated bibliographic databases like endnote. Downloaded references come complete with an abstract of the paper (if one exists) which by itself often contains valuable information which allows assessment of the likely value of the paper concerned. Other information provided includes the authors address and affiliations, allowing direct contact with the author, which means you can request a copy of the paper if needed. However, a note on strategy is relevant here. I think the journals, mostly owned by big international companies like Walters Kluwer, Elseiver etc, are beginning to get less than straightforward. Author contact information is not always so easily visible as it used to be (and should be). Obviously there is a major financial advantage to them if you pay $25 - $40 to them to get a copy of the reprint, rather than write to the author directly, who can usually supply you with a free copy. I do suspect that publishers are too fat and too greedy and getting an unduely large a slice of the pie in relation to scientific publishing revenue. You only have to look at the share value and expansion of some of these companies to appreciate my point. Perhaps my retirement fund would be a little larger if I had purchased some of their shares long ago.

These financial considerations are having substantial impact on research, especially independent research done by people like me. Obviously I can't afford to buy reprints at $30 a copy, especially when 99/100 of them will, on reading thoroughly, prove to be of no use. That would mean I would be paying roughly $3000 for every useful paper that I find. Even major university libraries are having to cut back on the number of different journals and databases they subscribe to because of the cost.

Nevertheless, going to the journal website and looking carefully usually gives the result (author address), if the address is not registered in the NLM database. If all of that fails the NLM database will produce a list of all other papers by the same author at the click of a mouse. It is usually possible to find a different recent paper with an address. Failing that, go to the institution website of one or more of the authors of the paper and it is usually possible to find an e-mail address.

Lastly, one can always have a guess at the e-mail address because many organisations ask people to format their e-mail address in a standard way. So if you know their name & initials you can guess their e-mail address. In the past I've formatted half a dozen different e-mails addresses in slightly different ways to discover which one works. Tedious, but where there is a will, there is a way!

If anybody else has got any other clever techniques they know of in relation to this let me know, I will add them to this commentary.

Assessing the Quality of Information

It is also useful to have some idea of how to assess the likely quality of the information that you are accessing. Things posted on the Internet, like the document you’re reading now, that have not been published in a peer-reviewed journal or other source of quality-controlled information, may have 1001 hidden agendas, biases, inaccuracies and downright lies and misrepresentations. This is why so-called “peer-reviewed” journals are generally accorded higher credibility, sadly not always justified.

Peer Review and ‘Refereeing’

The term peer-reviewed refers to the filtering processes that scientific work usually has to go through before it is accepted for publication in a reputable journal (this does not generally apply to books). When a scientific article is submitted to the editor of the Journal for consideration of publication the editor sends it to 2 or 3, sometimes even more, experts in the area of study concerned. They comment on it, and recommend to the editor whether or not it should be published. Almost always, if it is recommended for consideration of publication, this will be only after amendments and revisions in response to the comments that the referees will have made. I act as a referee for journals and have a modest amount of experience with this process. This process is far from perfect, but is important as a quality control mechanism for what is published. Nevertheless it is extremely important from readers to appreciate that just because something is published does not necessarily mean it’s good true or reliable. I will not enter into a diatribe about the imperfections of the peer review system. One only has to do a Google search about peer review to see that many eminent people have various criticisms of the process, many of which I heartily agree with. The selection of referees to do peer review for journals is pretty haphazard, even for supposedly prestigious journals. I am quite sure that many editors are under pressure to get papers published quickly, and because everybody is so busy now the number of researchers prepared to donate their time for nothing to undertake the time-consuming (and usually thankless) process of criticising papers is limited. I get all sorts of strange e-mails from editors requesting me to peer-review papers that I am quite clearly not qualified or competent to do. Since the most basic research and knowledge, on their part, should have informed those editors that I am not qualified it is obvious that the system is seriously flawed. Unfortunately, although I refuse all offered assignments for which I'm not qualified, that is clearly not the case for everyone. Obviously being asked to be a referee is quite an ego boost for many people and their own subjective judgement about whether they are sufficiently expert to referee a particular topic might well be the disputed by others.

Another example, which may give a little insight into the vagaries of the system maybe useful. Unless the paper that one has submitted is rejected outright by the journal concerned, the authors are sent comments from the various referees, often two or three, and asked to amend their paper explaining precisely how they have addressed the criticisms raised by the referees. Often there are useful comments, correction of errors and suggestions for addition of important references and material. However, equally often there are comments that are less than perspicacious. Most academics are busy and need to pay attention to the number and prestige of the papers they get published in scientific journals. Often their chance of getting research grant money etc. is crucially dependent on getting good papers published in good journals. That means most of them will bend over backwards to placate referees, to the point of being obsequious, rather than argue a point, waste precious time, and risk having the editor lose patience and reject the paper. That leads to unsatisfactory outcomes. Perhaps those explanations will give some small insight into the difficulties and weaknesses of the peer review system. Nevertheless it seems to be the best we have at the moment, although I am sure it can, and should be, improved on. Time will tell.

Meanwhile it is important to understand that even articles published in the most prestigious peer-reviewed journals (which have their fair share of bad refereeing ) cannot, and should not, be accepted without criticism and reservation. Further elaboration concerning the biases in scientific publishing can be found in my essay on ‘Why antidepressants are ineffective and how drug companies have deceived us’.

Internet searches

Generally speaking unrefined searches on the Internet for medical information will lead to so much bad and biased misinformation that most people will not be able to see the wood for the trees. One has a certain sympathy for ordinary family doctors who are bombarded with questions provoked by information acquired on the Internet. My advice is this: start off by limiting your search to educational establishments, government websites, and ‘no-profit’ organisations. It is easy to do this by using the Google search feature that finds only certain suffixes, such as '.edu' or .org or .gov (enter e.g.- 'site:.edu')

It is also possible to search for particular file types, such as Adobe PDF files etc. Generally speaking people who post quality material on government and educational websites are most likely to do it in the form of PDF files rather than word-processing documents. Therefore limiting your search to PDF files is likely to produce a more refined and useful subset of results.

Remember that in many instances someone else will have done the work already for you, it's just a question of finding it. So for instance, if one was trying to find out quality information concerning diet, exercise, weight and the health implications relating to heart disease, stroke, diabetes etc. one could start by searching for the recommendations and guidelines produced by the multitude of organisations and agencies with a finger in that pie. If your search has been limited to educational sites (.edu) then you will largely avoid blatantly partisan information produced by interested parties such as those who sell beef, cheese or sugar. From those documents it is possible to learn and extract the key terms and phrases that are used in that field, and note the prominent researchers that are quoted frequently in the bibliographies of such reports. One can then put those names into the appropriate databases to find whatever else they may have published more recently.

e.g.

Google web search for heart/cardiac info, enter:-

filetype:pdf site:.edu cardiac guideline

Which leads, along with much else, straight to a paper “Conflicts of Interest in Cardiovascular Clinical Practice Guidelines” (free pdf available!)

(www.bioethics.upenn.edu/documents/COICardio-ArchIntMed.pdf )

That paper contains a list of most of the significant guidelines produced recently. All done in a matter of less than a minute.

Then put the title into Google scholar and you get the result

“Conflicts of interest in cardiovascular clinical practice guidelines. TB Mendelson, M Meltzer, EG Campbell… - Archives of internal …, 2011 - ncbi.nlm.nih.gov
BACKGROUND: Clinical practice guidelines (CPGs) serve as standards of care in practice, quality improvement, and reimbursement. The extent of conflicts of interest (COIs) in cardiology guideline production has not been well studied. Herein, we describe the scope of COIs in CPGs. ...
Cited by 4

Click on ‘cited by 4’ and you get a list of subsequent papers.

I think that illustrates my point, and the process.

That illustrates the tactic I call a “forwards search”. If you put the name of a particular prestigious or recommended research paper into Google scholar you will notice that it tells you how many other papers have cited that paper, i.e. how many papers have used it as a reference in their bibliography. If you click on that it will list all of those papers. In other words you are looking forwards in time, from the publication you put in, to see who was subsequently used it. That can be a very valuable way of finding good and useful papers that are more recent.