Data smoothing is one of the basic techniques of Chartmanship. Owing to an incidental citation, the attention of Number Watch was drawn to a typical recent discussion.
The basic problem is the Uncertainty Principle. If you use a truncated section from an infinite time series to represent the whole process there is a fundamental and irremovable error involved. If for example you are estimating the Fourier Transform, taking the sample section is equivalent to multiplying by a block function [U(t) –U(t-T), where U(t) is the unit step function]. The Fourier Transform of this function is the oscillatory Sinc function. The resulting estimate is convolved with this function, producing the long-known Gibbs Oscillations.
The oscillations are produced by the sharp cut-off at the beginning and end of the sample. The technique to eliminate them is to apply a smoother, tapering window function to the data. This means that different workers will produce different result from the same data, which is an indication that the taking of a block sample is an irredeemable corruption.
The same problem occurs in trying to estimate or illustrate the changes in local slope of a non-stationary process. All sorts of filters of various levels of complication are used to smooth the primary data. In the treatment of smoothing in these pages only the two simplest smoothing filters were considered, not only for clarity but also because in general the more complicated the filter the greater the misrepresentation. Furthermore, with different authors using different variations of a wide range of filters, it becomes impossible to compare their results, without going back to the source data (which nowadays are often withheld).
The biggest problem of all, however, occurs at the end points, as with the window problem. It is impossible to provide a reasonable estimate of the local slope at the last data point. It all depends on what happens next, which is the great unknown. Various methods have been applied, but in essence they are all cheats. Some of the dishonest tricks used are quite gross, as happened with the notorious “Hockey Stick” This predicted that temperatures were rising sharply, when in fact they were declining. Among other enormities was packing a smoothing routine beyond the end-point with manufactured data that assumed the desired outcome.
The moral is: treat any smoothed data with care and, if they are smoothed up to the end point, with disdain.
Sinister, or what?
"He gazed up at the enormous face. Forty years it had taken him to
learn what kind of smile was hidden beneath the dark moustache. O cruel,
needless misunderstanding! O stubborn, self-willed exile from the loving breast!
Two gin-scented tears trickled down the sides of his nose. But it was all right,
everything was all right, the struggle was finished. He had won the victory over
himself. He loved Big Brother."
Your bending author gave up most of his fellowships and memberships on taking early retirement as, even with pensioner’s reductions, they were not in the budget. The only one kept was The Royal Society of Arts, for which in a bout of enthusiasm a life subscription had been taken out. Alas it was soon taken over in a Greenie coup, as so many other institutions, and has simply become yet another propaganda front for global warming and related myths.
issue of the Journal goes a step even further. The front page has the
statutory polar bear on the cover, but the feature article is rather more
disturbing. It is entitled The
new rules of persuasion and is all about how computers can be used to
manipulate the behaviour and beliefs of people. Facebook occupies a central role in the narrative. Not long ago
society banned subliminal advertising, but that was in the bad old days before
... and furthermore
With your permission M’lud, with regard to the accusation that my client has libelled the society of which he is a fellow, I wish to draw the attention of the Jury to the photograph on another page of the very same journal in question. Unfortunately for those who have relied on the web version it is cropped and so small that the writing is difficult to discern. The scene is one of a committee of which one member is a polar bear. I ask the jury to observe and ponder on those paws and jaws. The reality is that, if such a scenario was ever enacted, the human participants would be mauled to death. The blood spouting from severed arteries would splatter the walls, perhaps even obscuring the mandatory reference to climate on the white board.
Nature is not accurately represented by the image of a boy and a bear dancing hand in paw through the Hundred Acre Wood. Bears are savage beasts which the wise do well to avoid. The image of the polar bear is frequently employed in reference to the myth that the population of these animals is under threat, maintained by those who do not even know where it lives.
Thus far, M’lud, I have followed my client’s instructions, but I would like to add a note in mitigation of sentence. My client is an elderly person, who was trained as a scientist when scepticism was the norm. He joined the said Society when it was a forum for free discussion of matters relating to human progress. He has found it difficult to adjust to modern times in which, for reasons of discipline and taxation, it is necessary for public discussion to be guided by those of superior knowledge.
I rest my case.
Link to this piece
Donne, I suppose, was such another
Who found no substitute for sense;
To seize and clutch and penetrate,
Expert beyond experience,
T S Eliot, Whispers of immortality
On the thirteenth of this month, Number Watch and its bending author share a birthday. Just in time, the author of the first ever article to be quoted in Number of the Month, Richard Wiseman, has come up with yet another article from his prolific output. We have occasionally referred to his progress in between. That first article was about the number thirteen (spooky, eh?). Members of our Forum might have something to say about his latest claims. Academic etiquette seems to have changed over the years. Time was when no professor would refer to himself as an “expert”; that was a word for lawyers and journalists. Now that marketing has superseded scholarship in academia, things are different.
Link to this piece
A further remark on filters
Further to the opening piece this month, over at Climate Audit there are erudite discussions about a fellow who was a bit naughty with triangular filters. The triangle is a convolution of two similar rectangles. It therefore follows that the triangular filter has the same response as the moving average applied twice. As we have remarked in our discussion of simple filters, there is no need to store any coefficients or intermediate values and the algorithm can be very fast. Furthermore, as the Fourier transform of convolution is multiplication, the oscillations in the frequency domain are similar except that the maxima are diminished. The more passes you make with the moving average, the closer you get to a Gaussian filter.
On a note of self-indulgent nostalgia, your bending author
(with two younger colleagues) was possibly the first in the
Link to this piece
The arrogance! Army generals think they know more about conducting warfare
than highly-trained Government ministers. Just because they have a few medals
and ribbons, they presume to challenge some of the finest minds in
Then there are the dedicated civil servants in the Ministry of Defence. They are so effective that 50 million pounds was set aside for bonuses to them, even in these straitened times. There are of the order of 100,000 of them. It must be very reassuring for our troops in the front line to know that each of them has twenty or more of these devoted desk-pilots behind them, seeing to their every need. It is a gratuitous insult to them to suggest that our soldiers are not properly equipped. After all, it was Tony Blair who promised that troops would receive all the equipment they needed, and everybody knows that he is a pretty straight kinda guy.
So let us all get behind the politicians and civil servants who are fighting this war and make sure that it is not lost on the home front. No doubt the Government machinery is already planning to provide every front-line soldier with a new kitchen and bathroom when he returns, but as such things are military secrets we cannot be told about them.
Footnote: “brown jobs” is RAF slang for the army.
Footnote 2: Correspondents have noted the deft switch from first person to third in The Minister’s biography. Those who are critical simply do not appreciate the poetic muse. After all, the alternative explanation is that the man is a total prat who cannot even organise his own biography, quite impossible in a trusted Minister of The Crown.
Link to this piece
Galileo in the wrong again
More great strides have been made by modern scientists, who
prove that Galileo (and, for that matter,
The engineers discovered that the laws of locomotion mean that they fall to the ground more quickly between each running stride or swimming stroke and therefore can outperform lighter competitors.
So the old boy was wasting his time dropping things off the
Link to this piece
How could it happen? Is it a freak accident or a light at the end of the tunnel? How did it get past the editorial censorship? In The Times, of all places, there is an article by Antonia Senior telling it how it is about the green extremists. It is entitled Blunt warning about greens under the bed. Read it before she is banished to the journalistic gulag.
Heard the one about …
… the Met Office? It is the joke of the year in the few
remaining pubs of
In April the Met Office forecast
a hot summer for the third year running, causing the media, in their usual
jolly way, to announce the barbecue summer. In
May the Met Office told us that we are now going to roast because of the
urban heat island effect, which up to now has been disregarded as negligible.
Then in the same month they caused resorts like
Now the Met Office is involved with a spat over the fact that they are keeping secret the sources and processing of the data that give rise to their apocalyptic pronouncements. This is because they get them from the CRU, which is denying the public access to data they paid for, apparently on the grounds that people might be tempted to criticise them. This is not only a gross perversion of science, it is also a negation of the Freedom of Information that we are all supposed to be enjoying.
O brave new world that has such people in it!
Footnote: Your bending author, sixty years late, has now begun to understand a cryptic comment occasionally made by his old granny, who was born in the nineteenth century – “If someone is hiding something, they have something to hide.”
Link to this piece
Number of the month – 27
At last real scientists are beginning to show some guts and are standing up to the bullies. 27 is the number of letters published on July 27th in Chemical and Engineering News in response to a typically outrageous editorial attacking doubters of the anthropogenic global warming hypothesis. Almost all of them are hostile to that standard ad hominem attack on those who practise the sceptical approach that has long been the characteristic and glory of the scientific method.
Perhaps British chemists will now find the courage to do something about their own “Chief Executive” Dr Richard Pike, whose pronouncements are just as politically biased. Why do they need a CE anyway? Time was when learned societies just had a secretary and were run by their members.
Link to this piece