Scientific Fact, Fiction & Repression [Updated]
Debate about global warming is likely to heat up in the next few days, and not just because the Copenhagen conference is about to begin. Someone hacked into the file server at the prestigious University of East Anglia Climate Research Unit and secured thousands of files and emails dating back to 1999 and then uploaded them to an anonymous FTP server.
Start reading here at the Air Vent, one of the blogs that broke this story.
Then here. (Intimidation and retribution – that’s the ticket!)
And here. (Paranoia or preemption? – Shoot first! Ask questions later!)
And here. (When the data doesn’t fit – Massage it! Hide it! Deny it!)
Examiner.com has been covering this as well. My favorite quote thus far is the one they selected:
One such e-mail makes references to the famous “hockey-stick” graph published by Mann in the journal Nature:
I’ve just completed Mike’s Nature trick of adding in the real temps to each series for the last 20 years (ie from 1981 onwards) amd from 1961 for Keith’s to hide the decline. Mike’s series got the annual land and marine values while the other two got April-Sept for NH land N of 20N. The latter two are real for 1999, while the estimate for 1999 for NH combined is +0.44C wrt 61-90. The Global estimate for 1999 with data through Oct is +0.35C cf. 0.57 for 1998.
[UPDATE:]
Bishop Hill has a good summary, with citations, of some of the most damaging emails discovered thus far in his post Climate cuttings 33. Much of this boils down to concerted, and seemingly successful, attempts to subvert legitimate Freedom of Information Act requests. Towards the bottom of the page, one of the commenters reprints an email sent after receipt of a FOIA request asking that all parties delete all emails referencing AR4, presumably so that they could reply that none exist.
Ambiance 
chasrmartin said,
November 20, 2009 at 2:33 pm
Don’t forget my piece on PJM.
jason said,
November 20, 2009 at 3:14 pm
I’ve spent the last twelve hours going through this data. It’s interesting–but not incriminating so to say. It’s all focused on paleoclimate models (tree rings, ice cores, etc.) as a means to predict future weather. There’s some embarrassing stuff in here, though, and that can’t be denied. Assuming it’s all true.
But is it a smoking gun against anthropogenic climate change? Hardly. It touches only on a fraction of the science that comes into play. Still, it’s going to do more harm than good.
(BTW, ‘trick’ in scientific parlance means “a shortcut”.)
jason said,
November 20, 2009 at 3:18 pm
I should add I’m still going through it. There’s a chance I’ll find the report/e-mail that says “We’re all lying!” or “It’s all a sham!” or whatever.
And gosh! Does anyone delete anything any more? There’s a lot of unrelated and/or personal stuff in this bundle.
PatHMV said,
November 20, 2009 at 4:21 pm
I agree with Jason that the”trick” quote itself may or may not mean anything. But one thing becomes clear quite quickly when you go through a bunch of the e-mails… just how much the “science” hinges upon massaging and interpreting some very dodgy raw data. And it’s crystal clear, an abundance of proof, that the organization fights very hard against releasing any of their raw data, hiding behind claims of confidentiality agreements with the scientists who provided it.
But since when does science provide confidentiality for raw data, like how many and what types of rings some ancient trees had? If the raw data isn’t released, the results can’t be replicated. And anything that can’t be replicated IS NOT SCIENCE.
The “hiding” and “correcting for the blip” messages will garner public attention, and appropriately so, but it’s this underlying fact which will doom Dr. Jones and his ilk. They’ve spent the last 10 years damning everybody who won’t accept their party line as unscientific “deniers,” but they won’t agree to follow the most basic of scientific practices themselves. If the science is really that conclusive, it can’t hurt to release the raw data. Undoubtedly, the raw data will show that most of the claims of global warming, with a causal connection to CO2 levels, is based on only assumptions and interpretations, rather than being clearly revealed in the actual data.
jason said,
November 20, 2009 at 4:45 pm
Pat: On the confidentiality of raw data, one need only look at commercial attempts to prohibit NOAA from freely releasing climatological data to the US public. When NOAA announced a few years ago that it would make readily and freely available all of its raw data (since we taxpayers already fund it, we should have access to it–so the logic went), a great uproar ensued with companies saying we the public didn’t deserve access to it and free access to it would harm commercial interests. I’d be less than assumptive with regards to nondisclosure agreements and science…
As a physics and mathematics double-major, let me point out that paleoclimate data DOES NOT reveal current trends, nor does it predict the future. It’s useful in terms of geologic trends with what happened–and guesses with what might happen, but it does not refute–nor even challenge–the clear physics of methane and CO2 and what those gasses do and are doing to the climate. That part of anthropogenic climate change remains wholly unchallenged by this data (as does the vast majority of the science). Even if we assume the hack contains 100% accurate information, paleoclimate disinformation in no way modifies current scientific understanding of climate change or the impact of our activities on atmospheric physics.
Disclaimer: This comment assumes the hack contains only paleoclimate data and is real data and is damning. So far, I’m assuming all three points–though there’s a lot of information and I’m still going through it.
PatHMV said,
November 20, 2009 at 5:54 pm
Jason, I do recall that debacle. But my point does not go away; I was not commenting on what the current practices of the scientific community are, but what is actually science. And science itself depends on sharing raw data. I’m not disputing that there really may be confidentiality agreements, just making the point that it stops being science when an experimenter refuses to release the raw data… especially when Phil Jones, one of the people who is refusing to release data, has said, on the record:
Certainly paleoclimate data does not tell us about the future, or the present, but it does tell us about the past. We know what CO2 concentrations do in small scale experiments and in closed systems, but we don’t actually know, other than by comparisons to the past or the use of unproven computer models, the effect it has on a global scale. We don’t know whether there are any self-corrective measures which may moderate the effects of CO2 and other greenhouse gases, for example. We don’t know what impact long-term sun cycles have on our climate and the global temperature. The people refusing to release the raw data are the ones telling us that the earth is hotter now than it ever has been since man’s been around, pretty much. They used the phony “hockey stick” (until its flaws became so evident that they had to publish a paper themselves discrediting it) to claim that there was a recent massive temperature increase of an unprecedented nature in history. They’re also the ones who say we can’t look at, say, the most recent 10 year period (which shows no warming), because we’re talking about a long-term trend.
Without paleo-data, we simply have no idea what the global temperatures 1,000 or 2,000 years ago were.The claim is that the introduction of massive amounts of CO2 in the atmosphere is the cause of significant warming over the past 100 years. But we only have global, truly accurate measures of temperatures going back maybe 50 or 60 years, and we know that climate is cyclical, as are short term weather patterns. So without the paleo data, we have nothing to compare recent temperature readings to.
Nobody’s had time to analyze the data that was released today; I’m not sure they even have figured out the format of the datafiles yet. But the e-mails themselves, as I noted, make it clear that there’s a lot of data massaging going on to make their models work and show the trends they claim.
PatHMV said,
November 20, 2009 at 5:59 pm
Oh, and before I forget, the e-mails also reveal a concerted effort, by AGW proponents, to pressure scientific journals to refuse to publish anything by scientists they considered to be skeptics or deniers. Very helpful to their argument that skeptics must be wrong because there work wasn’t appearing in peer-reviewed journals. And the proponents of AGW were mostly peer-reviewed by a network of people with whom they had co-authored articles.
jason said,
November 20, 2009 at 8:54 pm
First, as this no doubt will turn into a for and against argument with me portrayed as a “for” enemy of common sense, let me reiterate what I said at Annie’s blog more than two years ago:
jason said,
November 20, 2009 at 9:15 pm
That said, let me correct you, Pat. We have weather records going back centuries. Temperature data for England goes back to the 17th century, instrument data goes back to the 18th century, and the US has weather records going back 200 years. I’m not sure where that 50 or 60 years business comes from, but it’s scientifically and historically inaccurate.
I do owe you an apology, as I misunderstood your meaning on the nondisclosure thing. So, Pat, Please forgive me as I took that to be about nondisclosure as a whole–and on a broader scale.
However, scientists are people, too. Politics, religion and pressures come into play. I do accept, however, that all the e-mails in the hacked bundle are out of context; neither you nor I can speak to the meaning of individual messages–or even all the messages. I’ve reviewed them and can hardly link the majority to each other or to the surrounding conversations. Since I’ve worked in technology for decades and know legal investigations often run into the same problem, I can say without a doubt that these messages have been hand-picked–redacted if you will–based on their content rather than their meaning. What that quoted sentence means is dependent entirely on what came before it, and that data we don’t seem to have. That’s the case with all of the e-mails, by the way; they’re orphaned, tidbits of larger conversations offered up as though they mean something more than snippets of sound overheard sans context. Too bad as they could be important…
(Am I the only one who noticed each message that responds to another message lacks the original context–a practice that RARELY happens in the real world?)
As for the science of the issue, vast amounts of CO2 and methane as released by industry and agriculture DO make a difference. Closed system or climactic model notwithstanding, the physics of the thing is undeniable and irrefutable. How that works on this scale in a system based on chaos theory is anyone’s guess, but does that negate our responsibility to act, to change our ways, to clean up the mess we’ve made and are making? I hardly think so. Unfortunately, those who want to leave the whole thing to someone else will use this hack as justification to sit on their thumbs and pretend the world is perfect without taking responsibility. That’s the way most of our species works.
The data itself, after review, seems genuine. The e-mails are meaningless, as I’ve pointed out, since without context my own e-mails could mean I meant to shoot President Bush for torture or strangle President Clinton for the Oval Office affair. The e-mails in the hacked package are of the same sort: they stand on their own as selectively chosen specimens that really could be bad–and that’s precisely how they’ll be used by people who want to keep on consuming while leaving the mess for future generations–but ultimately the messages as they’re presented can’t be cleanly linked to intent, at least as far as I can see. Still, it looks bad–for a tiny portion of the science involved, a fact most people will fail to recognize or will conveniently ignore.
(And I admit I’m tired of looking through this data right now; perhaps I missed a link from one message to another; many of them are responses, not new content, hence I can’t tell what they’re responding to but I do know that responses without context are easily misused).
Tomorrow, I’ll finish with the reports. Or so I’m telling myself…
PatHMV said,
November 21, 2009 at 12:35 am
Regarding the weather data, I didn’t mean to suggest that we have NO measured temperature data from a couple of hundred years, of course. But what we have is, as I understand it, relatively isolated. Today we have thousands of weather stations, probably tens of thousands, around the world. Moreover, we have satellite measurements which can measure the average surface sea temperature with a great deal of exactitude. For 200 years ago, we have data in Europe, probably some isolated measurements in a few places in Africa, and readings along the major sailing routes, yes? As some of these e-mails themselves note, it is hard to integrate the old data with the new, because the measuring techniques, and the sheer amount of data, is so new.
At any rate, in 1999, the World Meteorological Organization [pdf] felt strongly enough about the importance of the paleo-data that it put it on the cover of its annual report on climate change.
In isolation, you are correct. It’s of course a good idea to examine ways we can minimize our impact on the environment. Even dogs know better than to shit in their own den. But, as you say, context is important. And the context with this political argument is that proponents, led politically by Al Gore, want to impose draconian measures, with massively increased governmental control, on our economy, as rapidly as possible. Without doing so, they tell us, New York may be flooded in less than 100 years.
I don’t have a problem with taking some action. But the fight is not over whether to take some measured action, but whether government should impose massive taxes on industry (which would merely result in more jobs going oversees, where a blind eye will be turned to pollution and CO2 emissions), ban incandescent bulbs and non-power-thrifty TV sets, impose mileage taxes which would require all of us to have GPS trackers installed in our car, you name it. If fighting against those measures (which would likely not actually reduce global carbon emissions, anyway) is “sitting on my thumbs and pretending the world is perfect without taking responsibility,” then yeah, I’m guilty. Show me some measures which will work and won’t force me to drive a Prius while Al Gore and other politically connected folks jet around the world to conferences and to their mansions in different countries, and I’ll be happy to support them.
I don’t think you and I disagree that much here, Jason. I did note, as you did, that while the e-mails will attract a lot of publicity, what will eventually have the most impact (and this is more a statement of hope than a definitive prediction) is the fact that so much of their data for paleo times, on which they themselves place a great deal of importance (if only because they think it helps them convince other people) is based on heavily manipulated and massaged data, which is based on raw data which they have consistently refused to release to any researcher not known to be correct in their politics on this issue.
Let me add that it’s not a matter of intent. I have no idea whether these guys have ill will or not, whether they are intentionally deceiving people or are truly trying to do the best they can. What matters is whether the data and the predictions based on the data are accurate. And that has not been at all demonstrated. And I’m quite certain that the individual scientists, such as Phil Jones, who are leading the movement, long since have lost objectivity and are now focused primarily on defending their theories rather than engaging in real discovery. That, too, is a very traditional part of scientific history. There’s nothing inherently wrong with it, but it means we should resist very strongly the demands that anybody who questions them must be excommunicated from “climate science” and labelled a “denier.”
amba12 said,
November 21, 2009 at 1:14 am
SUCH a good argument — and not against each other, but at a subtle angle to each other. Bless you both.
Here are a couple of contributions (the first more interesting than the second): a New Yorker article by Elizabeth Kolbert on paleoclimatic research which shows how frighteningly rapidly the climate can reach a tipping point (and flip into another ice age, e.g., which may be due to happen again); and a scienceblogger explaining some of the lingo in that specific e-mail that sounds so bad to the uninitiated.
Forrest said,
November 21, 2009 at 3:29 am
Jason,
In regards to your statement;
“On the confidentiality of raw data, one need only look at commercial attempts to prohibit NOAA from freely releasing climatological data to the US public. When NOAA announced a few years ago that it would make readily and freely available all of its raw data (since we taxpayers already fund it, we should have access to it–so the logic went), a great uproar ensued with companies saying we the public didn’t deserve access to it and free access to it would harm commercial interests. I’d be less than assumptive with regards to nondisclosure agreements and science…”
Could you actually provide me your sources to corroborate this accusation? I find this quite hard to believe and more along the lines of “deny everything, make counter accusations” rather than actual fact.
Thanks
Icepick said,
November 21, 2009 at 10:30 am
And gosh! Does anyone delete anything any more? There’s a lot of unrelated and/or personal stuff in this bundle.
If the IT staff did their jobs correctly, ALL of the email that comes through a mail server would have been saved. It doesn’t matter if people delete it from their personal machines or not.
Could you actually provide me your sources to corroborate this accusation? I find this quite hard to believe and more along the lines of “deny everything, make counter accusations” rather than actual fact.
Why is it hard to believe? It seems quite plausible, given the way that both sides in this argument have behaved. One side is preaching the End of the Earth if we don’t do something by the end of this year (they’ve been doing that for about 15 years now), and the other side says that nothing is wrong and nothing can ever be wrong. Any voices outside of these two camps get completely drowned out by the shouting.
DISCLOSURE: Personally I do believe that humanity can have an impact on global climate. Last I read evidence is emerging that humans have done so in the past. (For example, I remember a report that suggests that the global cooling trend from a few centuries ago may have been due to reforestation in the Americas caused by the die-off of the indigenous population and the related demise of large agriculture on these continents.)
However, I remain unconvinced by the AGW research to this point. Humans have been pumping out huge (though still increasing) amounts of carbon into the atmosphere for well over 100 years now. (I believe that the first person to propose that this carbon dump would lead to global warming actually made the prediction in the 1890s.) Why haven’t we seen a more pronounced effect? Over that span we’ve had two mini-cooling periods IIRC, and the hot periods don’t seem that much higher than the “norm”.
The science of climatology is still pretty new, as are many of the instruments. And the predictions to this point rely one rather shaky computer models. Frankly, the scientists need to keep studying the issue, and quit pronouncing that they know the truth of the matter. When their models have fewer assumptions and more knowns they can get back to us.
The biggest reason they need to quit the yammering on this subject is because there is only so much attention that we can expect people to pay to the environment, and there are more pressing matters than the Earth getting a touch warmer. I’d much prefer to see all this focus on the health of the oceans, where the effects of agricultural run-off and over-fishing are stark, irrefutable and directly related to human action. Action CAN be taken on such issues, but only if the policy & opinion leaders prioritize these matters.*
But that won’t happen, because that would be hard and the results would be measurable and people could be held accountable for their actions. That isn’t going to happen with the current lot running the world.**
* I firmly believe that dead oceans mean a dead planet. I’m not sure the evidence supports this position 100%, but it may. And it’s a hell of a lot more intuitive than believing that a temperature rise of a few degrees means the end of the world.
** The healthcare debate in the USA is a prime example of what I’m talking about. Healthcare costs did not wreck the economy, yet that has been the only focus of the Administration and Congress for months and months now. The non-universal healthcare Congress is currently pursuing won’t do anyone any good if the economy does not get better. But they will not focus on THAT topic until after they fuck-up another 1/12 of the economy and the angry mobs of unemployed people force them to take action.
Randy said,
November 21, 2009 at 12:45 pm
Jason: Your point about the probable meaning of “trick” in this case is well-taken. Thanks. At the same time, it appears that the authors did go on public record as denying that they had done exactly what they said they had done in the email. FWIW, somewhere amongst these emails, I understand that there some encouraging using the pretense of private interests’ objections preventing the release of publicly-funded data. Discovering which were real and which were manufactured by the authors may prove difficult as the authors exert so much control over both private and public research.
I don’t have a strong opinion on the debate over global warming but I do wonder about the millenarian overtones and the on-going organized suppression of dissent.
wj said,
November 21, 2009 at 12:56 pm
Ice, whether global warming is occurring due to human actions is actually irrelevant (or should be) to the discussion. Suppose, just for example, that global warming is due entirely to a change is solar activity. (Which I don’t think is the case, but just for discussion.) Well, to the best of my knowledge we can’t do anything about the cause. But perhaps something that we can do will still reduce the effect. Ditto for any other possible cause. Or even if we have no clue as to the cause.
There are really only 3 relevant questions:
1) Is global warming occurring?
2) Would global warming have, on balance, a negative impact?
3) Can decreasing the level of CO2 in the atmosphere, or any other action that human beings might be able to take, have an impact which reduces that warming?
If the answer to all of those is Yes, actions should be taken. Which action or actions, exactly, are a matter of feasibility, impact, etc. — all the parts of a standard cost/benefit analysis. But, if all those are true, they should be taken nonetheless.
So (working in reverse order), can we do something to reduce (or increase) global warming? The science there is pretty solid that we can.
Would global warming have an overall negative impact? I haven’t seen anything that actually goes thru all of the impacts, negative and positive, and evaluates the balance. Doesn’t mean that it hasn’t been done, just that I haven’t seen it.
Is global warming occurring? What data I’ve seen, admittedly nothing like an exhaustive review, inclines in that direction.
But those are the points of a useful debate.
Forrest said,
November 21, 2009 at 2:31 pm
Jason,
Once again in regards to your apparently fallacious statement;
“On the confidentiality of raw data, one need only look at commercial attempts to prohibit NOAA from freely releasing climatological data to the US public. When NOAA announced a few years ago that it would make readily and freely available all of its raw data (since we taxpayers already fund it, we should have access to it–so the logic went), a great uproar ensued with companies saying we the public didn’t deserve access to it and free access to it would harm commercial interests. I’d be less than assumptive with regards to nondisclosure agreements and science…”
I have been trying to corroborate this claim but have thus far been unable to find any information to support it; therefore I believe it to be a red-herring. You didn’t happen to glean this information from Green Peace or the Daily Kos did you?
Just asking.
PatHMV said,
November 21, 2009 at 4:34 pm
If I recall correctly, the NOAA controversy did not involve the sale of underlying raw weather data. NOAA has always made that available for free. But a few years back, NOAA proposed providing the same data directly to end users by building a pretty website with all sorts of weather data, maps, etc. This would have damaged the business models of Weather Underground and numerous other commercial, for-profit websites which relied on the free data and weather alerts and so forth provided by NOAA.
Their argument was essentially that there was no need for taxpayer funds to be spent developing a pretty front-end to the raw data, since there was a solid, functioning market out there providing the same service at no cost to the taxpayer.
I can’t find any old stories about the controversy, but I’m pretty sure my recollection is correct.
And there’s no need to levy personal insults at Jason, please. He’s engaging in civil debate, doing the best he can (as do we all) to remember old events. No need to accuse him of being a Kossack or Green Peacer.
Icepick said,
November 21, 2009 at 4:59 pm
There are really only 3 relevant questions:
1) Is global warming occurring?
2) Would global warming have, on balance, a negative impact?
3) Can decreasing the level of CO2 in the atmosphere, or any other action that human beings might be able to take, have an impact which reduces that warming?
No, these are not the only three relevant questions. Another one would be:
4) Is there anything that is a more pressing matter?
But those are the points of a useful debate.
Yes, the points of a useful debate are to start with you setting the terms (i.e. questions, and leaving one out), then answering the quesitons, and then determing that actrion should be taken. (Note that you claim action should be taken regardless of the results of the cost/benefit analysis.)
So as long as we all agree with you and do exactly what you say, we can have a meaningful debate. Outstanding.
Forrest said,
November 21, 2009 at 7:16 pm
PatHMV,
Whereas I do appreciate the fact that you see being accused of referencing the Daily-Kos as an insult (heh) I really didn’t mean to be overly critical of Jason, although I suppose it came across that way, my apologies. I greatly appreciate your illumination on this subject; it by and large makes much more sense then the usual “businesses are all evil” rhetoric. I am however, quite frustrated by the disinformation that is being propagated by the feverish supporters of AGW. If there is anyone who has been preventing the disclosure of our tax-payer funded research, it’s the AGW supporting climate scientists, the UN and associated left leaning governments as well as most media outlets, as further evidenced by this latest hacked data. I will admit that I’ve been a long time skeptic [of AGW] and have always found it very odd that all of the supporting research has left out the planets primary source of heat, good old Sol itself, especially since the non manipulated temperature data seems to trend rather nicely with solar flare activity…it’s a no-brainer (I do not dismiss greenhouse and oceanic effects, but I’d think that our sun plays a large part and should not be entirely dismissed.)
wj said,
November 22, 2009 at 12:27 am
Well, I would have said that “Is there anything that is a more pressing matter? would be an integral part of the cost/benefit analysis in question 3. But certainly it would be an issue in deciding what to do (if anything).
My comment about those being the relevant questions was directed more towards the arguments over whether human action was the cause, or at least the primary cause, of global warming. Because I don’t think that it matters whether it is the cause or not. All that matters, IMHO, is whether human action can do anything to counter global warming, should it be happening. Can you see some way in which it does matter what the cause is? If so, please enlighten me.
PatHMV said,
November 22, 2009 at 1:54 am
wj, the cause matters to the extent it illuminates whether our actions might have any impact or not. If the cause is solar cycles, and CO2 plays little role, then reducing CO2 emissions may not impact any warming which may be occurring.
My personal belief is that we are quite arrogant to believe we can fully understand a chaotic system as complex as the global climate.
For example, one theory suggests that data shows that the Clean Air Act actually increased warming, because it reduced the number of dark particulates in the air, and the particulates were blocking sunlight from heating the lower atmosphere. Consider this: suppose we learn how to completely manipulate the levels of CO2 in the atmosphere. What’s the “right” level? What’s the “right” temperature for the entire earth?
We can be pretty certain about the economic impacts of taxing carbon emissions and other of the massive regulations which Al Gore seeks to impose on us. The extent of any warming, the potential consequences of it, and all the other science of climate change, are massive unknowns, predictions based on a bit of really hard data and an awful lot of soft data which requires a lot of theories and interpretations to make sense of.
Assuming there is mankind-endangering warming afoot, we also need to know whether we can in fact stop the warming, or whether the warming has already reached a tipping point, and nothing we do will stop, say, 1/2 the polar ice caps from melting (I don’t in fact trust any of the data on such issues, so I have no idea whether the ice cap really is melting or not… and in large part this lack of credibility is precisely because it’s been obvious for quite some time how secretive and political people like Phil Jones had become). In other words, we need to know whether we should be spending our resources to try to stop the warming from happening, or to prepare to deal with the consequences. For example, suppose (as the most alarmist of the AGW “scientists” would have it) that we’re about 50 years away from New York City being submerged by melting ice caps. If we can’t stop the warming, then our resources would be better served by building a new city or two, further inland.
We can’t possibly know whether our actions will help or hurt until we actually understand the causes. That’s why it matters what the cause is.
Ennui said,
November 24, 2009 at 9:25 pm
For those of you who have downloaded the data, the HARRY_READ_ME.txt (in the Documents folder) should not be missed. It appears to be a log of “Harry’s” attempt to recreate published data from from “original data sets” (if by “original data sets” we understand “random gobbledygook”). Seriously, it’s as though the guy were trying to rewrite Varro’s Antiquities from the references contained in The City of God.
Somebody called “Asimov” at this place has written an entertaining series of posts on the file.
Here’s a representative quote from the file
This is pretty obviously the same station (well OK.. apart from the duff early period, but I’ve
got used to that now). But look at the longitude! That’s probably 20km! LUckily I selected
‘Update wins’ and so the metadata aren’t compared. This is still going to take ages, because although
I can match WMO codes (or should be able to), I must check that the data correlate adequately – and
for all these stations there will be questions. I don’t think it would be a good idea to take the
usual approach of coding to avoid the situation, because (a) it will be non-trivial to code for, and
(b) not all of the situations are the same. But I am beginning to wish I could just blindly merge
based on WMO code.. the trouble is that then I’m continuing the approach that created these broken
databases. Look at this one:
…
Here, the expected 1990-2003 period is MISSING – so the correlations aren’t so hot! Yet
the WMO codes and station names /locations are identical (or close). What the hell is
supposed to happen here? Oh yeah – there is no ‘supposed’, I can make it up. So I have :-)
If an update station matches a ‘master’ station by WMO code, but the data is unpalatably
inconsistent, the operator is given three choices:
You have failed a match despite the WMO codes matching.
This must be resolved!! Please choose one:
1. Match them after all.
2. Leave the existing station alone, and discard the update.
3. Give existing station a false code, and make the update the new WMO station.
Enter 1,2 or 3:
You can’t imagine what this has cost me – to actually allow the operator to assign false
WMO codes!! But what else is there in such situations? Especially when dealing with a ‘Master’
database of dubious provenance (which, er, they all are and always will be).
Ennui said,
November 24, 2009 at 9:46 pm
By the way, the larger point I draw from the file is this; we are being told to radically reorient almost every aspect of society, the world over … by technical incompetents.
Ennui said,
November 25, 2009 at 12:30 am
ESR weighs in with a “find,” here
;
; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,- 0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
;
yearlyadj=interpol(valadj,yrloc,timey)
This, people, is blatant data-cooking, with no pretense otherwise. It flattens a period of warm temperatures in the 1940s — see those negative coefficients? Then, later on, it applies a positive multiplier so you get a nice dramatic hockey stick at the end of the century.
All you apologists weakly protesting that this is research business as usual and there are plausible explanations for everything in the emails? Sackcloth and ashes time for you. This isn’t just a smoking gun, it’s a siege cannon with the barrel still hot.
I’ve checked. This is, indeed, in the file cited.