Flaw in the Hockey Stick

Global Warming Bombshell
A prime piece of evidence linking human activity to climate change turns out to be an artifact of poor mathematics.

By Richard Muller
Technology for Presidents
October 15, 2004

Progress in science is sometimes made by great discoveries. But science also advances when we learn that something we believed to be true isn’t. When solving a jigsaw puzzle, the solution can sometimes be stymied by the fact that a wrong piece has been wedged in a key place.

In the scientific and political debate over global warming, the latest wrong piece may be the “hockey stick,” the famous plot (shown below), published by University of Massachusetts geoscientist Michael Mann and colleagues. This plot purports to show that we are now experiencing the warmest climate in a millennium, and that the earth, after remaining cool for centuries during the medieval era, suddenly began to heat up about 100 years ago–just at the time that the burning of coal and oil led to an increase in atmospheric levels of carbon dioxide.

I talked about this at length in my December 2003 column. Unfortunately, discussion of this plot has been so polluted by political and activist frenzy that it is hard to dig into it to reach the science. My earlier column was largely a plea to let science proceed unmolested. Unfortunately, the very importance of the issue has made careful science difficult to pursue.

But now a shock: Canadian scientists Stephen McIntyre and Ross McKitrick have uncovered a fundamental mathematical flaw in the computer program that was used to produce the hockey stick. In his original publications of the stick, Mann purported to use a standard method known as principal component analysis, or PCA, to find the dominant features in a set of more than 70 different climate records.

But it wasn’t so. McIntyre and McKitrick obtained part of the program that Mann used, and they found serious problems. Not only does the program not do conventional PCA, but it handles data normalization in a way that can only be described as mistaken.

Now comes the real shocker. This improper normalization procedure tends to emphasize any data that do have the hockey stick shape, and to suppress all data that do not. To demonstrate this effect, McIntyre and McKitrick created some meaningless test data that had, on average, no trends. This method of generating random data is called “Monte Carlo” analysis, after the famous casino, and it is widely used in statistical analysis to test procedures. When McIntyre and McKitrick fed these random data into the Mann procedure, out popped a hockey stick shape!

That discovery hit me like a bombshell, and I suspect it is having the same effect on many others. Suddenly the hockey stick, the poster-child of the global warming community, turns out to be an artifact of poor mathematics. How could it happen? What is going on? Let me digress into a short technical discussion of how this incredible error took place.

In PCA and similar techniques, each of the (in this case, typically 70) different data sets have their averages subtracted (so they have a mean of zero), and then are multiplied by a number to make their average variation around that mean to be equal to one; in technical jargon, we say that each data set is normalized to zero mean and unit variance. In standard PCA, each data set is normalized over its complete data period; for key climate data sets that Mann used to create his hockey stick graph, this was the interval 1400-1980. But the computer program Mann used did not do that. Instead, it forced each data set to have zero mean for the time period 1902-1980, and to match the historical records for this interval. This is the time when the historical temperature is well known, so this procedure does guarantee the most accurate temperature scale. But it completely screws up PCA. PCA is mostly concerned with the data sets that have high variance, and the Mann normalization procedure tends to give very high variance to any data set with a hockey stick shape. (Such data sets have zero mean only over the 1902-1980 period, not over the longer 1400-1980 period.)

The net result: the “principal component” will have a hockey stick shape even if most of the data do not.

McIntyre and McKitrick sent their detailed analysis to Nature magazine for publication, and it was extensively refereed. But their paper was finally rejected. In frustration, McIntyre and McKitrick put the entire record of their submission and the referee reports on a Web page for all to see. If you look, you’ll see that McIntyre and McKitrick have found numerous other problems with the Mann analysis. I emphasize the bug in their PCA program simply because it is so blatant and so easy to understand. Apparently, Mann and his colleagues never tested their program with the standard Monte Carlo approach, or they would have discovered the error themselves. Other and different criticisms of the hockey stick are emerging (see, for example, the paper by Hans von Storch and colleagues in the September 30 issue of Science).

Some people may complain that McIntyre and McKitrick did not publish their results in a refereed journal. That is true–but not for lack of trying. Moreover, the paper was refereed–and even better, the referee reports are there for us to read. McIntyre and McKitrick’s only failure was in not convincing Nature that the paper was important enough to publish.

How does this bombshell affect what we think about global warming?

It certainly does not negate the threat of a long-term global temperature increase. In fact, McIntyre and McKitrick are careful to point out that it is hard to draw conclusions from these data, even with their corrections. Did medieval global warming take place? Last month the consensus was that it did not; now the correct answer is that nobody really knows. Uncovering errors in the Mann analysis doesn’t settle the debate; it just reopens it. We now know less about the history of climate, and its natural fluctuations over century-scale time frames, than we thought we knew.

If you are concerned about global warming (as I am) and think that human-created carbon dioxide may contribute (as I do), then you still should agree that we are much better off having broken the hockey stick. Misinformation can do real harm, because it distorts predictions. Suppose, for example, that future measurements in the years 2005-2015 show a clear and distinct global cooling trend. (It could happen.) If we mistakenly took the hockey stick seriously–that is, if we believed that natural fluctuations in climate are small–then we might conclude (mistakenly) that the cooling could not be just a random fluctuation on top of a long-term warming trend, since according to the hockey stick, such fluctuations are negligible. And that might lead in turn to the mistaken conclusion that global warming predictions are a lot of hooey. If, on the other hand, we reject the hockey stick, and recognize that natural fluctuations can be large, then we will not be misled by a few years of random cooling.

A phony hockey stick is more dangerous than a broken one–if we know it is broken. It is our responsibility as scientists to look at the data in an unbiased way, and draw whatever conclusions follow. When we discover a mistake, we admit it, learn from it, and perhaps discover once again the value of caution.

Richard A. Muller, a 1982 MacArthur Fellow, is a physics professor at the University of California, Berkeley, where he teaches a course called “Physics for Future Presidents.” Since 1972, he has been a Jason consultant on U.S. national security.

Copyright 2004 Technology Review, Inc. All rights reserved

And to be fair, here’s the rebuttal.

Saving the ecology

The most expensive project supported by Kentucky’s tobacco-settlement money is changing that. The farmer-owned Commonwealth Agri-Energy ethanol plant in Hopkinsville started quietly last winter but will stage its grand opening today. It covered nearly $10million of its $32.8million cost with grants and loans from the Kentucky Agricultural Development Board….

Commonwealth Agri-Energy plans to convert more than 7million bushels of corn into 20million gallons of ethanol a year. General manager Mick Henderson said half of the ethanol will go to Louisville, the only Kentucky city requiring gasoline mixed with an oxygenate additive. The rest is headed to markets such as New York and St. Louis, which also require reformulated gasoline.

The grant mentioned in the article was funded in large part by the EPA, particularly Bush’s Clear Skies initiative of 2003. Under Clear Skies, Louisville had to add additives to their gasoline. The Hopkinsville plant will be the supplier. They are also supplying St. Louis and New York due to the same regulations. There was an accompanying article that I can’t find now that this location has already secured a private customer that is going to add their ethanol to their gas in all of their gas stations, thereby compelling the plant to produce even more ( 40 million I think it was ).

This is great, but it’s not unusual right now. There are a lot of plants opening and under construction right now. Besides the obvious ecological benefits of cleaner burning gas, because of the sudden surge in gas prices, it makes economic sense. Kentucky used some of the tobacco settelement to develop a new crop base for farmers, corn. That’s a win-win situation for Kentucky. Other areas have similar reasons.

But, none of this would be happening if it were not for the huge lure of federal funding under Clear Skies. It needs a lot more credit than its being given. Reducing domestic usage by even 10% will have a massive impact on the politics of OPEC. They need to know we’re serious about reducing our dependency on foreign oil. Ethanol is the key to getting that message across NOW.

Cool summers and hurricanes

NOAA REPORTS COOL SUMMER, SEVENTH COLDEST AUGUST ON RECORD ACROSS THE LOWER 48 STATES;
RECORD NUMBER OF NAMED ATLANTIC TROPICAL CYCLONES FOR AUGUST

I’m in that “Much Below Normal” category. This was the coldest summer I can ever remember. Tourism was down here, we rely on boating. I only made a couple of trips to the lake and wore jackets both times.

The contiguous United States experienced its 16th coolest summer (June-August) on record and seventh coolest August, according to scientists at NOAA Climatic Data Center in Asheville, N.C. While much of the West, including Alaska, remained warmer than average, the majority of the nation had a cool summer, with Minnesota having its coldest August on record. Meanwhile, eight named tropical cyclones in the Atlantic basin broke a record for named storms during the month of August.

Meanwhile? I think there’s a connection.

Regional Global Warming?

Warming May Be Less Severe in Central U.S.

ST. LOUIS (AP) — Anticipated global warming by mid-century may be less severe in the central U.S. than elsewhere in the country, researchers said Tuesday.

Scientists at Saint Louis University and Iowa State used a detailed regional climate model to determine that estimated summer daily high temperatures will not climb as much in the area centered around the Missouri-Kansas border as in other parts of the country….

This doesn’t surprise me. Not only are there fewer urban areas in the Midwest, it’s not growing either. This has been the coolest summer in my lifetime. And, scientists have already noticed that most global warming occurs in cities, where the most weather monitoring is done.

I’ll place a standing bet that if the NOAA monitors very rural areas of the US, they’ll find no significant change in temperatures by 2040. In fact, over the last five years here, it’s been cooler than normal.

Warped Space and Time

First of all, Einstein’s Theory of Relativity is not one simple equation, it’s a book. Secondly, although it poses a bunch of theories, it also leaves room for a lot of “what ifs”. Scientists have been exploring those “what ifs” since he wrote it. NASA is addressing one of those “what ifs” right now via the Gravity Probe B. The intent of the GP-B is to measure Earth’s impact on space-time. In other words, it will try to measure how space-time is warped by the presence of Earth ( a large body of mass ).

Always the type to ponder questions, I immediately had this question, if indeed Earth does warp space-time, then how do we know for certain by using the doppler shift how far away things are. I know they account for objects, but if they aren’t sure how large bodies of mass ( huge planets, large stars, etc.. ), then how do they know how much to account for the warped space-time?

Earth and Sky offered this great explanation: ” gravitational lens is a clump of matter that bends space — and thereby distorts light coming from a source behind it. Instead of seeing one star or galaxy through the lens, you might see two to five stars — or arcs of light. Gravitational lenses are hard to find because an enormous amount of matter is needed to create any noticeable effect on light. In 1979, the first gravitational lens — an entire galaxy — was discovered. Here’s Anthony Tyson … they found two quasars — radio quasars — right next to each other and when they looked at the spectrum, they discovered it was the same identical quasar seen two times, separated by six arc seconds on the sky. And that was the beginning of a hugely productive mining of the sky for gravitational lenses. It started in the late 1970s. There are many, many gravitational lenses now.”

That did well to explain how they know that what we see has been warped, but it really didn’t answer my question of how much space-time has been warped.

A friend continued to appease my ignorance, and further added this:

the amount of space-time bending is inversely proportional to the distance from the massive object. Since the massive object is very far away, the hidden object must appear to be directly behind the massive object in order to get the light-bending effect.

And to answer another question. The time it takes the photon to reach the earth increases due to the bending of space-time.”

I thought that was very insightful, but still didn’t answer the basic question. We went round and round and round and finally another poster, 0x6a74, summarized with an even lengthier explanation of how space is not measured “in” time, but “by” time. And once it was all said and done, finished with this:

by the time one’s looking at these guys, billions of ly away, that something massive in between may have warped space enough so that something that looks to be several thousand ly more than it is …. not much percentage error against all the other possible errors.

In other words, we’re just making best guesses based on what we do know. GP-B could re-establish the age of the universe, again.

The Natural History of ADD, part 3

There was a medical report on tv tonight, amid all the bad football, I lost track of who ran it. I thought it was Fox, but I can’t find it now. The jist of the story was that ADD/ADHD is horribly overdiagnosed. However, in trying to find it, I found this:

There is a clinical study out now, along with many others that support the same results and probabilities, which revealed 74% of the 261 HYPERACTIVE CHILDREN (ADHD) studied, had a reaction to food additives. REACTIVE HYPOGLYCEMIA causes a burst of adrenaline, AKA “FIGHT OR FLIGHT”, a natural innate human behavioral/physiological instinctive response to adrenaline, apparently triggered by an unnatural exposure to consumed or digested amounts of chemicals or food additives. I believe most of us are also familiar with the dynamics of symtomology related to allergies. Go to WWW.LEF.ORG, look up ADD/ADHD, and take it from there. To Julie, I research “ANOMALIES”, “SYNDROME’S” and “PHENOMENONS” related to child developement and human behavior, as they relate to the advanced and industrialized countries and societies of the free world, I focus on the extreme (geographic) mathmatical and statistical probabilities and improbablities, and the unusual per capita percentage differences and contradictions, this research produces both positive and negative results, depending on the individuals knowledge, perception and/or beliefs of the reviewer of the research information and materials. Keeping an open mind is essential to any research project, you have to meticulously research and dissect the pro’s and con’s for authenticity and accuracy. There is a subliminal message usually being projected among certain orginizations and lobbyist’s “IF YOU ARE SMART, YOUR BUYING WHAT I AM SELLING?” I have to look at this every day, so it is in my opinion, from 100’s of hours of research on and about ADD/ADHD, that it is “overdiagnosed” in the U.S., and should at least be near the same per capita % as the other advanced and industrialized countries, and not 9 times the rest of the entire world combined for consumption! Sincerely TOM

Pretty heavy conclusion if you ask me, nine times the rest of the world? Let’s look closer, shall we?

ADHD is often called “the benevolent diagnosis,” says Carolee Malen, a therapist at St. Aemilian-Lakeside, a residential treatment center on Milwaukee’s northwest side.

Malen, who treats children with extreme behavior problems, says she sees many who have been misdiagnosed.

“Ninety percent of the time, I’d say, that’s not really what the kid has,” she says

Basically what’s being said here is that ADD/ADHD is a way of covering up more serious “temporary” mental illnesses so the kid isn’t tagged with a worse label throughout their life. Problem is, that practice is sloppy and it’s being used against the argument of ADD/ADHD. The report I saw tonight said most kids could be treated for depression in the first place, temporarily, and the issue resolves itself as the kid matures. Problem is, schools use the ADD/ADHD issue as a cop-out to move kids into a lesser environment and tag them with that “can’t make it in school” label. If the kid has depression, treat the damned depression and get it over with. And, whatever you do, don’t let the school know.

Sheez.

The simple answer? If a kid has a mental illness, for sure, SOLVE OR ADDRESS THAT PROBLEM. If the kid is simply a hunter, deal with the hunter. ADD/ADHD is not necessarily a disorder or illness that has to be dealt with. Doctors lumping more serious problems such as depression into a catchall phrase like ADD is the problem.

The Natural History of ADD, part 2

Well, no sooner than I rant about ADD and ADHD not really existing, MSNBC informs me of this:

Long thought to be a condition of childhood that kids grow out of, recent research shows that up to 60 percent of children with ADHD continue having symptoms after they become adults.

It’s a common chronic disorder that affects about eight million grown-ups, or an estimated four to five percent of the U.S. adult population, according to researchers from Harvard Medical School and the World Health Organization. What’s surprising is that only about 20 percent of them realize it, experts say.

That’s just great. Before long we’ll have a diagnosis for every excuse people may have.

The Natural History of ADD?

I have just enough experience in this category to be dangerous. I took a couple of years of abnormal psychology in college. The topic depressed me too much so I switched to astronomy. I didn’t graduate in astronomy either.

I’d like to start this long-winded rant with a poll if you don’t mind. Think carefully about the question if you would.

If I were in school today

View Results

Loading ... Loading ...

Getting back to the subject, I was listening to Dr. Dean Adell today on our local talk radio station and he hit a topic, and a rant, that made some sense to me. He got cut off and didn’t fully elaborate his particular opinion, but I liked it so much I wanted to explore it here. The following is not Dr. Adell’s opinion, he just laid the very basic concept out, the opinions are purely mine. ( Unless he wants to share credit in this rant. )

Namely, ADD and ADHD are not actually disorders. This struck me as a radical thought, but frankly, a thought I’ve had for a long time. He just put it in a way I had never heard before, but hear me out, it makes sense.

For 99% of the history of mankind, he relied on two abilities only, hunting and farming. The instincts for farming are to be able to concentrate on the task at hand, to be very methodical, and, most importantly, to be very patient. The instincts for the hunter are the exact opposite. You have to be alert, you have to be aggressive, you have to have very quick responses, and, the task at hand had to be very brief. Those hunters that did not have those abilities would have succumbed to those that did. Evolution was not very generous for about 100,000 years. The process of the weak being eliminated eventually bred the fastest, most alert, and most adaptive hunters. This process evolved realistically until the 19th century.

Now, fast forward to the 1990’s. Kids who concentrate, are methodical, and patient are rewarded in school. Kids who are alert, aggressive, and can’t deal with a process that takes very long ( studying ) are punished. Now we’ve even given them titles to identify those traits, ADD and ADHD. We give them drugs hoping they can adapt to concentrate, be methodical, and patient. In other words, hoping the hunters become farmers.

In today’s society, our school systems can’t deal with the hunters. The education system is based on learning rote. We don’t do this in active ways, we do it in passive ways, reading. We expect them to sit quietly in class while information is pumped into their brains like a factory. The hunters are genetically not equipped to do this. And I, in agreement with Dr. Adell, don’t think it’s necessary that our hunters become farmers. The reason mankind overwhelmed the rest of the animal kingdom was because man was more adjustable to all climates and conditions. The reason this was true was because man could either hunt or farm. No other animal has that capability. It bought man enough time to develop his mental skills to develop his own environments and technologies that now separate us completely from the animal kingdom. And, the skills of the hunter and farmer both are equally as important today. The farmer has established his place in today’s society, but the hunter seems to be having a problem.

Not everyone needs to be book smart. In fact, most people don’t need to be. That does not mean they can not contribute to society. The hunters are the ones that build things, protect us, and do the jobs the farmers don’t have the skills to do nearly as well. But, society IMO is doing itself a total injustice by treating the hunters as if they have a mental problem. Rather than diagnosing kids as ADD and ADHD and doping them up so that they can sit in a classroom, let’s put them in vocational schools and treat those schools as respectfully as we do book schools. Today’s mechanics make as much as most lawyers. Hell, we need a lot more mechanics now than we do lawyers.

Think about this for one second. We’re giving amphetamines to kids for a “mental disease” that didn’t exist 20 years ago. We’re teaching these kids that the only way to be normal is to be hooked on drugs.  That’s wrong folks, that’s bad wrong. A lot of people are now finding “reasons” for the sudden blight of ADD and ADHD, blaming it on video games and such. It’s not the kids, all kids are addicted to excitement. It’s the adults. We’ve got too many farmers and not enough hunters in our school system.

ET Calling?

An unexplained radio signal from deep space could — just might be — contact from an alien civilization, New Scientist magazine reported on Thursday.

The potentially extraterrestrial signals were picked up through the SETI+home project, which uses programs running as screensavers on millions of personal computers worldwide to sift through the huge amount of data picked up by the telescope.

I participated in the Seti@home project for about a year. It was a blast but my stuff went nowhere. When I set up an internet server on my PC, SETI had to go.

This would be SO cool if it’s true! I have been advocating spending more money on SETI projects than going to asteroids and Mercury for a reason, radio waves travel a lot faster than man is going to for a long, long time. We can actually possibly contact other intelligent life out there via communications. It is fairly cheap to do, and the results could happen in our lifetime that proves there is life out there instead of spending decades at a time looking for microbes on Mars.

I’m not going to get my hopes up on ET calling, just yet anyway. But if it is what they think it might be, that would be the coolest, most exciting thing to happen to man since he started walking upright. The social ramifications to an Earthcentric religion based belief system would be shattered instantly. And for the sake of man, it needs to be.


Follow-up:

SETI@home Signal Story Sees Much More Than Meets the Eye
By Seth Shostak

Alas, this story is misleading. According to Dan Werthimer, who heads up the UC Berkeley SERENDIP SETI project, this is a case of a reporter failing to understand the workings of their search. He says that misquotes and statements taken out of context give the impression that his team is exceptionally impressed with one of the many candidate signals, SHGb02+14a, uncovered using the popular SETI@home software. They are not.

Pretty much what I expected, but I’m still hanging on.