Home | Feedback | Feedback-General |     Share This Page
Media Studies

A conversation about the scientific standing of media studies

Copyright © 2008, Paul LutusMessage Page

Reliability of Evidence I | Reliability of Evidence II

(double-click any word to see its definition)

  Reliability of Evidence I
Hello Mr. Lutus, great to see you're are still updating content for inquisitive students. Did you get a chance to sail to Alaska this summer? Sure did: Alaska 2008 I'm writing to ask a question that I personally already have an answer for, but I still could use a little insight. Awhile back I wrote to you asking if it would be possible to apply essentially the same argument you made against Psychology to another obscure social science like Media Theory. You never wrote back, so I imagined the question either annoyed you, or you were just too busy to respond. It's just that I get quite a lot of inquiries on that topic, and I can't answer all of them. I think the answer is obvious, but I'll explain further below. Anyhow, I decided for myself, based on a review of several sociology textbooks, and more recently, my mass media theory textbooks that virtually all social research has such severe experimental limitations and can't be trusted. But I may have overstepped. For some reason, none of the professors agree with me that limited predictability equals limited reliability. Most likely the professors are not scientists and do not understand basic science. They may not care that basic scientific standards cannot be met. They may be arguing that science doesn't matter, or something else. But they cannot argue that an untestable, unfalsifiable theory represents science in anything but name. I made this assertion in class, and it fell short with the other students and the Professor. I tried to argue that these theories are lacking predictability, or as I see it, they cannot convincingly claim that media have more negative than positive effects on society and vice verse. This is not to say that the media doesn't have an effect good or bad, just that the science doesn't hold enough weight either way. As to science, it's not basically a matter of predictability, that is more a symptom than a fundamental issue. A theory that lacks sufficient scientific properties might lack predictability, but that isn't the determining factor that separates scientific theories from other kinds.

The primary property that identifies scientific theories is falsifiability. If a scientific theory is put forth, it must make testable predictions. The tests must be practical, and if they fail, the theory must be discarded. If any element in this sequence is not present or practical, the theory is not part of science.

In media studies, people like to hold political discussions that have a superficial resemblance to scientific debates.

Let's say that a particularly violent TV program is followed by a gruesome murder. Someone says the program caused the murder. How can we establish this in a scientific way?

Well, we can't, for several reasons. One reason is there is just one data point, and one data point is never enough for science (except in the special case of an immediate falsification — the so-called "black swan" case).

To establish a positive correlation between a cause A and an effect B, we need several things that are almost never part of media studies. Among those things are:

  1. A control group.
  2. An unambiguous test case that guards against hidden assumptions.
  3. A prospective, not a retrospective, experimental design.
This is just the barest outline. Real-world media cannot ever be used for serious scientific work because it fails all three criteria.
Much of my thinking on this topic stems from the fact that my great aunt and uncle were murdered by my cousin. His father was killed in a car accident when he was 8, and he never got the necessary help he needed to recover and grow. I knew him as a fun loving ten-year-old, and he was not insane then. But because of some little understood reasons, he became psychotic by age 20. He was given three separate diagnoses by different psychologists. That by itself would have made me psychotic. Oh, sorry, I guess this isn't very funny from your perspective.

You need to realize that psychology is a collection of opinions, not diagnoses. Psychology is closer to astrology than it is to oncology, with the complication that psychologists are sometimes confused with scientists in courts of law, and people sometimes lose their freedom based on psychologists' opinions.
All of which were considered to be personal illnesses that were not dangerous to others. He was diagnosed as manic depressive, then schizophrenic, then schizo-affective — whatever the heck that means. Schizophrenic means you're crazy. Schizo-affective means you feel crazy. Some people feel crazy but aren't, while others ... oh, I'm sure you get it, four possibilities altogether:
  1. Doesn't feel crazy, isn't crazy.
  2. Feels crazy, but isn't crazy.
  3. Doesn't feel crazy, but is crazy.
  4. Feels crazy, is crazy.
And anyone who believes what psychologists say needs to have his head examined.
I might be making a big leap here, but because of the number of contradictory theories in media science, I'm now wondering what good media research is to the general public. Without strict experimental controls, media research ends up being a playground for anyone's pet theory, and nothing is resolved in a reliable, systematic way. This happens to be the present state of the field.

Media studies is a very highly politicized field, and even if it were much more scientific than it is, because of all the special interests with a stake in the outcome, it would still be a minefield in which to try to conduct serious research.

One side of the most common media issue (portrayals of violence) might say:
  1. Violence in the media harms young, impressionable minds, and even though this can't really be demonstrated scientifically, "everyone knows it's true".
  2. Society has a responsibility to guard the welfare of the young and innocent, thus the state has a mandate to act with respect to this issue. State regulation is both appropriate and necessary.
The other side might say:
  1. People must accept responsibility for personal actions. They should not be allowed to blame others for personal choices and/or crimes, and to try to blame the media is to throw open the door to declaring everyone a victim and absolving people of anything resembling personal responsibility.
  2. To try to control the media is to meddle in very basic constitutional issues, dabble in censorship, and if such a program were to begin, it would only drive certain kinds of media portrayals underground, where they might thrive in a more insidious and dangerous form (as in the example of illegal drugs).
I think you will see that both these positions have merit. Like any public issue worth discussing, this one has at least two legitimate sides.
I chose to study the media partly because I'm generally intrigued and somewhat frightened by what people with political motivations believe As you get older, you will probably mellow with regard to what positions people take on public issues. There's nothing like experience and seasoning to help you see why people say the things they do. (I'm also terrible at Mathematics). Now that is a shame. If I were you, I would do my best to acquire those kinds of skills. Given your present interests, knowing how mathematics works could save you a lot of heartache while trying to sort out the very issues we're discussing. To begin with, you would discover how an experiment can be designed to appear to support a view that it cannot actually support. I'm just starting to work in the broadcast field, but also have a little training in marketing, advertising, and public relations strategies and tactics. The only thing I've figured out is that communicators don't necessarily know why people respond to media tactics but they know how to push emotional buttons, and some media people do so usually for sake of profit or propaganda. I see media theory as a very dangerous "science" as it is currently pursued. In other words, we can push a bunch of buttons, but we don't quite know what will happen. Actually, old hands in the field know what will happen. Propaganda is hardly a scientific undertaking, but there are some broad, general principles that can be relied on to effect public opinion in particular ways.
Reliability of Evidence II
Thanks for your earlier reply. I recently read The Black Swan, The Impact of the Highly Improbable by Nassim Nicholas Taleb. I recently read this book also. He argues, to my understanding, that Fractals are a superior form of mathematics for identifying Black Swans. The problem with Taleb's thesis is that Black Swans are unpredictable by definition. That is what makes them "outliers," to use his term. As he defines them, Black Swans lie outside any model — their appearance in a presumably controlled system like state economic policy only proves they are Black Swans, nothing else. If you create a mathematical model that takes yesterday's Black Swan into account, tomorrow's Black Swan will pop up in a place outside the new model's purview. Otherwise, it isn't really a Black Swan, is it?

Also, fractals are a very fashionable topic right now. I would want to see an example where a fractal approach produced a reliable predictive model of real-world phenomena in a way that sheds more light than heat. At the moment, fractals are more a plaything than a tool, and are more often used to model an existing data set than make a prediction.

There is an essential contradiction in claiming that Black Swans exist, defining them as Taleb does, and then expecting to be able to forecast their appearance. In fairness, he doesn't do this in an overt way, he only hints at it.
He takes a harsh tone against scientists who rely heavily on Gaussian Statistics in developing mainly economic theory. There's plenty of truth in that position. It is well-established that bell curves are often misused, so this is perhaps not quite so original as he makes it seem.

Students of statistics are cautioned that a straight bell-curve analysis can often be a waste of time and can lead to a false sense of security.
He points to the stock market crash of 1984, as evidence of the Gaussian Curve's shortcoming. He uses many other examples as well, but you've probably read at least one of his books already. Yes, one can always point to a real-world event to show that a conventional statistical analysis failed to predict something. Remember Einstein's remark, "As far as the laws of mathematics refer to reality, they are not certain, as far as they are certain, they do not refer to reality."

As to the 1987 (not 1984) stock market crash, it's important to understand that the behavior of free markets is not predictable even in principle — a large number of independent conscious entities are trying to maximize their gain, propelled by logic, strategy or emotion. Predicting a free market is identical to trying to analyze our brains while using our brains as the research instrument. This is a classic problem where, due to self-reference, Gödel's Incompleteness Theorems place strict limits on what can be predicted.
I'm guessing that you would agree, partly, that Gaussian statistics cannot be used properly in any social science, let alone economic theory. I would not say "any." That goes too far. If we're predicting the outcome of a series of coin tosses or a similar activity, a straight Gaussian approach is quite reliable. There are many real-world problems that can be reduced to a pile of fairly tossed coins, but the stock market isn't one of them. My main questions are: 1.) Are these problems with Gaussian statistics more a result of the people using them and the context in which they are using them? That's the problem — a Gaussian analysis is very easy to apply, so it gets applied to a data set for which it is inappropriate. This problem is emphasized in any worthwhile statistics course. 2.) Do you agree with Taleb that Mandelbrotian fractals are a useful alternative for developing social scientific theory, or at least predicting Black Swans? No. And he didn't give any examples for a reason — it's very difficult to meaningfully correlate a fractal with a real-world data set.

In fairness, the other reason there aren't any examples is that books meant for a popular readership tend not to have any technical detail, and equations are aggressively removed by editors. Supposedly there's a rule in publishing that each additional equation cuts your readership in half. I was going to include a simple equation that shows this relationship, but I don't want to cut my readership in half.

And further, with respect to predicting a free market, if a particular model's existence becomes known, its effectiveness is immediately destroyed. This is the self-reference problem again, and it's why people who try to sell you "winning market strategies" (methods to consistently stay ahead of average market indices) are all frauds — without exception, and whether or not they realize it.
My guess is that the limitations of stringent experimentation supersede the use of either types of calculation, as they apply to social theory building. That's true, and remember the classic pitfall of applying statistical methods to a pre-existing data set. Someone might apply a series of methods until one seemed to fit, without realizing the result is more likely a coincidence than a correlation. 3.) Have I missed the mark entirely? Not at all. I'm probably screwing up Nassim Taleb's actual position, and I'm sure you'll correct me. No, actually, Taleb's book contained a lot of unsupported claims, "hand-waving," and I think it was intended more as an essay on human gullibility and mental laziness than a scientific analysis with solid supporting evidence. I also understand that you likely view me as an idea consumer, and I can't really defend that, but I'll try. I view the ideas I come across as life preservers, giving me at least a chance for independence. We all have areas where we appreciate a little additional buoyancy, and ... any port in a storm.

Also, your inquiry has a very favorable property — skepticism. That's an important starting point for evaluating a new idea. Remember the importance of the "null hypothesis," the idea that a claim should be assumed to be false until evidence supports it.

For most people, an idea is assumed to be true until proven false. For a scientist, it's the reverse — an idea is assumed to be false until and unless evidence supports it.
 

Home | Feedback | Feedback-General |     Share This Page