Excerpt from ‘Filters Against Folly’ by Garrett Hardin
Criticism is not so much directed at technology itself as they are at the reliability of the humans who install and operate the technology. That Humans are fallible has been known since the beginning of time, but modern technology adds new urgency to the recognition.
Technology is used by human beings whose reliability can be used as a fraction. Fraction, because convention dictates that perfect reliability be indicated by the numeral 1, while experience shows that all human reliabilities are equal to or greater than 0 but less than 1. Human reliability can never be raised to the divine number 1.
Sometimes technologies with the greatest promise for good bring with them equally great potential for harm when improperly used. Nuclear power and poisonous pesticides are examples.
Sellers of chemicals and machinery who want policy to be set solely on the basis of the reliability of their products miss the point. The reliability that matters is not the simple reliability of one component of a system, but the final reliability of the total control system. The total reliability is the product of two factors:
Technology reliability (TR) x human reliability (HR) = System reliability (SR)
To replace the abstract entities with particular fractions, thus passing from the literate format to the numerate. Arbitrarily, let us assume that TR equals 0.9, while HR equals 0.6; this means that the probability that technology will not fail us, over a period of time is 90%, while the probability that human beings involved will not let us down, over the same period of time, is 60%. Since both TR and HR are necessarily involved in the system, the probability that the total system will behave is the sum of the two partial probabilities.
0.9 x 0.6 = 0.54
The reliability of the total system is less than that of the less reliable component. Suppose we manage to increase the value of TR to 0.99? With such 1% failure rate (per unit time) system reliability rises to 0.594 – which is still less than HR alone.
The result may be seen as a special instance of the ancient wisdom that a chain is only as strong as the weakest link. Even without achieving a divine level of reliability, modern technology has been greatly perfected that the weakest link in most technology/human systems is the human element.
Let us suppose, that with equal effort and expense we could increase either component reliability by 5%. If, then, we want to conserve effort and expense, which improvement should we strive for?
(A) 0.95 X 0.6 = 0.57 (B) 0.90 X 0.65 = 0.585
Obviously, we should follow path (B), that is, devote efforts to improving human reliability (since technical reliability is already so high). The conclusion is strengthened when we note the lack of realism in the assumption that equal improvements in TR and HR will follow from equal effort. In most systems, the greatest theoretical potential for improving total system reliability lies in improving the reliability of humans.
There is another way to improve system reliability: move some of the fallible elements from the human side of the system to the technical side, where we know better how to make improvements.
Example: An electrical system can be destroyed by a lightning bolt. We could station a man at a switch to break circuit whenever there was a power surge – but the man might have gone to the bathroom at that time. We bypass this by installing an automatic circuit breaker. Over the long run this device is more reliable than human beings.
That’s the way to go: redesign technology/human systems by moving components from the human side to the technology side. In fact, that’s the way we have been going for a long time. But complete reliability still eludes us, for several reasons.
The replacement of fallible human beings by much more technical devices is not a one to one replacement. The human is marvelously flexible and adaptable: he or she can manage several things. No complex, skillful human actions can be replaced by a single machine. So it?s many to one. A single human being is replaced by a complex assemblage of machine elements.
Suppose we want to replace human element in manufacturing process by a chain of machine elements. Suppose human element has reliability of 0.6 and each link in the technological chain has a reliability of 0.99 (1% failure rate). A total of 51 links must be there for the total technical reliability to fall to the human level.
There is yet another door through unreliability can sneak in.
Example – In no area of human endeavor have there been greater efforts to achieve 100% reliability than in the multi-billion dollar space program. When the launch of Gemini VI had to be aborted, the failure was traced to a dust cover that should have been removed before the launch. Investigations showed that a workman had signed a sheet that certified that he had removed the dust cover. Moreover two inspectors had further certified that the operation had been performed correctly. Would another 10 inspectors have increased the reliability? It is doubtful.
Behind every complex machine there stands an invisible army of human beings. All inspection procedures suffer from infinite regress. Inspections are usually so easy to make that stupidity is seldom the reason for failed inspections. But it is boring to make the same inspections day after day. Certification is expensive. Laziness (on the part of inspectors) and greed (on part of contractors) account for more failures than stupidity.
From now on, the more reliable our physical devices become, the more the measure of human reliability becomes the final measure of the reliability of the technology/human systems on which human welfare depends. By pushing technology to its furthest limits, man has become more dependent on himself.
In reaching this conclusion we have used the resources of mathematics of the very simplest sort. Numeracy of the simplest sort, but out of it comes the essential humanistic conclusion that the only thing we can really count on in this uncertain world is human unreliability itself. Some of our most expensive technical experts have trouble adjusting to this humanistic conclusion.
Example – In mid-1970 a $2 mn study of nuclear reactor safety, known as the ‘Rasmussen study?, concluded that the probability of a significant reactor accident was less than the probability of a person?s being hit by a meteorite. Unfortunately, in casting their analytical net the experts had failed to capture the human element. Then in 1980 there occurred the accident at the Three Mile Island reactor in Pennsylvania. The damage to human health was trivial, but the cost of repairing the plant ran into billions of dollars. After investigation the reactor failure, a presidential commission concluded that “….except for human failures, the major accident would have been a minor incident.”
No amount of naked numeracy can make up for failings to understand that reliability is a function of a total system, and not the sole consequence of the reliabilities that are easiest to quantify. A different approach is needed to understand the systems. This is the approach that engineers call “systems analysis” and biologists call “ecology”.