Thursday, December 1, 2016

Applied critical thinking: Expert testimony

We often take the testimony of experts for granted. When reading the newspaper or watching a documentary program, it seems reasonable to accept certain claims, provided that they are stated by individuals who have the right sort of credentials. But for areas of genuine controversy, it would be unwise to just accept expert testimony at face value. If we are to take sides in an area of controversy, we ought to be able to explain why a given expert is right and why others who disagree are wrong. There are plenty of instances where experts strongly disagree. For instance, there are doctors who believe acupuncture is an effective treatment for muscuoskeletal pain, and others who believe it doesn’t work, nothing more than an elaborate placebo. In such cases, it might not be obvious whose testimony we ought to trust, especially if one knows little to nothing about medicine. Figuring out who to trust can be complicated and will likely take some time. Through personal experience, I have encountered many friends and family who are disposed to throw their hands up in the air whenever areas of controversy are brought up. How can we—as nonexperts—ever decide who is telling the truth? How can we know? Instead of adopting agnosticism regarding all areas of controversy, most of us, in practice, do listen to some experts and ignore others. For instance, we are more likely to accept the testimony and advice of experts who share our own views. Instead of trying to confirm the beliefs we already hold, or engage in wishful thinking, we ought to critically evaluate competing expert testimony to the best of our ability. In what follows, I will expand upon a proposal developed by the philosopher Alvin Goldman, aimed to help one decide which experts to trust.

First step: Sift out the pseudoexperts
Before discussing the problem of how to choose between experts, we need to determine who has expertise in the first place. The thought is that once we eliminate all of the phony experts, we can move on to the harder questions of how to decide between the genuine experts. Perhaps a sufficient condition for being an expert in some domain of study would be the possession of an advanced degree—in the relevant field—awarded by a recognized academic institution. For instance, an expert in physics would be expected to have a PhD in physics. However, the knowledge set of your typical PhD is likely to be highly specialized. For instance, a scientist with a PhD in physics may be an expert regarding particle physics, but know very little about astrophysics or applied physics. In some cases, it may be unclear which area of study is the most relevant to the issue at hand. Here’s one example. For dietary advice, one might consider a nutritionist to be the most relevant authority to consult. While nutritionists might know a fair amount about dieting and nutrition, a better source would be a registered dietician. Dieticians tend to have much more training in science and medicine than a nutritionist, and have to pass a comprehensive exam to become certified. Overall, they are more qualified to be making judgments about dietary claims than nutritionists. Therefore, with regard to claims about dieting, the relevant experts are dieticians, not nutritionists. Individuals who purport to be experts on certain matters, but whom lack the relevant qualifications and/or training should raise red flags. In summary, before assessing expert testimony, one must try to answer the following questions:

What does X’s expertise consist in?
Is X’s expertise in any way relevant to the issue at hand?

Once one establishes that they have found a genuine expert and more importantly, one whose expertise is relevant to the question at hand, one must determine whether or not this expert is trustworthy. Are there reasons to doubt his or her testimony?

Steps for analyzing the testimony of true experts

It can be unhelpful to look at the testimony of one expert in isolation. From the point of view of a layperson (nonexpert) most experts tend to be highly persuasive. To get a better sense as to how reliable their testimony is, try to find an expert who disagrees with them, preferably, an expert that has had comparable training and experience. After finding two experts that disagree, it’s time to compare what each of them has to say. The philosopher Alvin Goldman proposes five ways to determine which expert is more trustworthy. I will deal with each one in turn and list some of the problems these guidelines face.

“(1) Read or listen to arguments and counter-arguments offered by the two experts, whether in a published exchange of views, an oral debate, or separate defenses of their respective positions.” (1)
Difficulty 1: The evidence and/or arguments discussed may include esoteric terminology. One may try and listen to the arguments and counter-arguments but fail to understand or even misunderstand them. Goldman makes a distinction between esoteric and exoteric terminology. Esoteric terms are not only unfamiliar to non-experts; they are inaccessible to them. This may because they involve unfamiliar concepts and theories. Exoteric terms are unfamiliar to non-experts but can that can be learned and understood by novices without any specialized training. Grasping exoteric terms may involve a subject to do some extra reading on the subject, whereas understanding esoteric terms may require one to become an expert in that field.

Difficulty 2: Superficially convincing arguments could be made to support one side of the debate, but these arguments may turn out invalid or contain false premises. If one tries to assess the arguments and counter-arguments of two disagreeing experts, one better have a decent working knowledge of informal logic. The expert committing more logical fallacies possesses fewer solid reasons for their belief. But a valid argument is not necessarily a sound one. Going back to the first difficulty, non-experts might not be able to tell whether a premise is true or not. To an expert, a given premise may be obviously false and contradicted by lots of evidence they are aware of. But to a layperson, it may seem to be plausible.

“(2) Find out what the opinions of other (putative) experts on the topic in question.    If most of them agree with expert A, then identify A as your best guide.  If most choose expert B, identify B as the more trustworthy one.  In short, go with the numbers to guide your choice of favored expert.”
Caveat: When it comes to issues where the vast majority of relevant experts are in agreement, going with the numbers is a good rule of thumb. For instance, the vast majority of climatologists accept that the planet is currently going through a warming trend and that this is primarily due to recent human activity. Should we accept the testimony of climatologists solely on the basis of consensus? In short, no. There are some possible scenarios where it would be rational to doubt consensus opinion (e.g. Nazi scientists during WW2). As long as the consensus position seems to be backed by valid arguments and independent sources of evidence, it is rational to side with the consensus.

“(3) Consult "meta-experts" about experts A and B.  Try to find out which of them is the superior expert by asking people in a position to compare and contrast them.  Or people who trained them or have worked with them.”

The idea is looking for additional experts other than the two you initially found. They might be in a good position to tell whether or not experts A or B have compelling arguments, especially if they have nothing to gain or lose in the debate. Where can one find meta-experts? You’re likely to find tons of meta-experts if you look through the peer-reviewed literature, reputable periodicals, or even personal blogs. In today’s age, many scientists blog to try and educate the general public about their work. One could even directly contact meta-experts at a local university or one could reach out to experts through email or on specialized web forums. And if you're lucky enough, you may personally know some meta-experts that could weigh in on the debate.

 “(4) Obtain evidence about the experts' biases and interests, which might lead them to self-serving answers of dubious veracity (whatever their underlying competence).”

In some cases, there is obvious bias and conflicts of interest at play with a given expert. Whether it is some political or religious agenda, these factors need to be taken into account. But one should be cautious when discrediting certain experts. It could be the case that some of the testimony is perfectly valid, or that their bias or conflicts of interest played no role in the formation of their beliefs about the issue at hand. Furthermore, an expert may be extremely biased but turn out to be right. Therefore, before one discredits an expert on grounds of bias or a conflict of interest, one must have some independent reason to think that their claims are wrong. Conspiracy theorists frequently misuse such a guideline when they discredit all experts who testify against their favored theory. Finding a potential source of bias (e.g. government funding) for some expert, and then completely disregarding everything they have to say is intellectually lazy and dishonest. The implication is that all such experts are lying or saying misleading things. If it can be shown that either is the case, that would be the reason to seriously doubt the expert’s testimony, not from just from the possibility of a conflict of interest.  

Another source of bias can be uncovered by carefully studying the behavior of a given expert. Experts who dismiss alternative positions out of hand should raise red flags. I say this because most experts tend to exhibit a certain kind of psychological profile. They tend to be fairly humble, well-integrated into their epistemic communities, and they seem to be genuinely interested in the truth. Experts tend to be very cautious when making controversial claims and admit that their own favored hypotheses could be mistake. They tend to actively engage their peers in academic journals and at conferences, try and test their hypotheses, and compare the success or failure of their own predictions to rival theories. Commenting on the small group of scientists who endorse the 9/11 conspiracy theories surrounding the tower collapses, Noam Chomsky has noted that, “They are not doing what scientists and engineers do when they’ve think they’ve discovered something" (2). Having studied the phenomenon of conspiracy theories quite extensively, I can vouch for Chomsky. Many of the “experts” who promote conspiracy theories lack the psychological profile found amongst genuine experts, and seem to share a number of opposing psychological traits. For instance, many manifest an excessive degree of pride or intellectual superiority, they tend to keep within a closed circle of peers who share their views, and they don't even attempt to convince the general scientific community about their "findings". These individuals might even possess the relevant knowledge to assess claims of interest, but their psychological profile and behavior casts serious doubt upon their capacity to seriously engage criticism and to critically evaluate their own positions. It is for this reason that their testimony should be taken with a grain of salt.

“(5)  Gather evidence of their past track-records and apportion trust as a function of these track records.”

Gathering evidence of an expert’s track record may turn out to be a challenge. Good track records of experts may come in the form of accurate and specific predictions. One might also try to look through their publication history or get a sense of their reputation amongst peers. Another way would be to look at an expert’s track record is to see if they have subscribed to controversial or fringe views in the past. Some experts can just be contrarians, while others seem to be suffering from crank magnetism. For instance, James Fetzer, a well-respected philosopher of science who taught critical thinking for most of his career, believes just about every conspiracy theory. He not only believes that 9/11 was an inside job, but that no children were killed at Sandy Hook, that we didn’t land on the moon, and that Paul McCartney died in the 60’s and was then replaced by someone with the same physical appearance, personality, and musical talents. In the case of Fetzer, there seems to be some kind of systematic misapplication of critical thinking going on (at least when he is theorizing about certain historical events). Those like Fetzer not only routinely appeal to pseudoexperts, but they accept many demonstrably false or highly questionable claims, make a number of unwarranted assumptions about human nature, and are apparently unskilled at making inferences to the best explanation. Since critical thinking is really a set of skills, knowledge of certain concepts and strategies related to the subject is not sufficient for knowledge how.

I believe that if one closely follows all of the advice outlined above, one will be more likely to acquire true beliefs about the world. But it is one thing to propose a strategy that makes sense in theory and quite another to have the strategy actually work for most people. Certain facts about our psychological limitations (e.g. confirmation bias, cognitive dissonance) may prevent us from being objective enough to really follow through with such advice. Nonetheless, we ought to at least try to be as objective as we can. Given the complexity of the world around us, we all need to appeal to experts at some time or another, whether it's to find out about our personal health, or how the world works. If we are going to appeal to experts, and are genuinely interested in discovering the truth, then it's a good idea to try and track down the right ones.

 1. Goldman, A. I. (2001). Experts: which ones should you trust?. Philosophy and Phenomenological Research, 63(1), 85-110.

2. Tuskin, B. (2013). Noam Chomsky has no opinion on building 7. Retrieved December 05, 2016, from

No comments:

Post a Comment