The Global Missile Defense Race: Strong Test Records and Poor Operational Performance

On 18 January 1991, news crews covering Operation Desert Storm from Saudi Arabia were on hand to witness what they described as the first ever combat use of a missile defense system to shoot down an incoming missile. “There’s a streak of light,” Charles Jaco, of CNN, said describing the intercept, “The [Patriot] missile goes North, cants slightly to the East, goes North again, disappears into the clouds, and then there’s an illuminating flash…”¹ On the ground, ABC News correspondent Forrest Sawyer remarked that, “the Patriot did its job perfectly.” ² The event is still celebrated. Twenty-five years later, the Army Air Defense Artillery School invited one of the soldiers involved to participate in a commemoration of the first Patriot missile to shoot down an Iraqi Scud missile. “That single event,” the school’s commandant explained, “and the engagements that followed, ushered in a new era for air defense artillery.” ³ There is just one important missing detail – there was no Iraqi Scud missile fired that night. The Department of Defense would later admit that Iraq did not fire the first Scud in this area until January 20th, two days after the event. The Patriots fired at a false alarm.

To be fair, Desert Storm did usher in a new era. In the years since the Gulf War, missile defense systems designed to intercept ballistic missiles have become a common-place piece of military equipment for many countries. At least twenty-six countries either possess, or are in the process of purchasing, missile defense systems. Israel, Saudi Arabia, the United Arab Emirates, and the United States have all used missile defense systems in combat. ⁴ The commonplace nature of missile defense systems has led to a widespread assumption that these systems are as effective as advertised. Surely these systems must work, the argument goes, because if they did not, why would all of these countries want to buy them?

And, moreover, the widespread belief that defenses are working against short- and medium-range missiles has helped create the impression that the defenses will also work in much more demanding scenarios – such as an attack using intercontinental range ballistic missiles. When asked whether he believes the United States could shoot down a North Korean missile fired at the United States, General John Hyten, the Vice-Chairman of the Joint Chiefs of Staff, responded by saying “I have 100 percent confidence. Literally – I don’t say 100 percent very often. I have 100 percent confidence in those capabilities against North Korea.” ⁵

The data we have compiled, however, does not support such confidence. Our research shows a substantial gap between the effectiveness of missile defense systems in combat and the public statements from government and industry officials about their effectiveness. Moreover, our research suggests that claims about successful interceptors often do not stand up to scrutiny. In fact, we are unable to identify a single instance in combat where an interceptor unambiguously engaged in the successful intercept of a ballistic missile. And this is for the easiest class of targets – short- and medium-range missiles. Intercepting an ICBM, fired by a country like Russia, China or North Korea, would be far more difficult. That is not to say that there has never been a successful intercept – it is usually extremely difficult to assess whether an engagement resulted in a kill or not. Almost thirty years after the first intercept attempt in combat, we are still searching for an unambiguous success.

The CNS Global Missile Defense Database & Methodology

The James Martin Center for Nonproliferation Studies (CNS) created the first publicly available global dataset of all known missile defense engagements for the Nuclear Threat Initiative (NTI) in order to better understand tests, operational uses, and the overall state of missile defenses worldwide today. To select by a specific missile defense system, click the dropdown arrow under “system name” and select a system.

The dataset of missile defense engagements includes both tests and instances of combat use. In order to focus on modern missile defense interceptors, the following criteria were adopted:

• Most modern missile defense interceptors are hit-to-kill systems that use kinetic energy to destroy incoming nuclear warheads. Therefore, interceptors with hit-to-kill or explosive warheads are included, while tests of nuclear-armed interceptors or directed energy weapons are excluded.
• The dataset only includes engagements that involved another missile, either as a target or as a real threat. Test interceptors against “virtual” targets or points in space, as well as operational launches in error or against radar blips, are excluded.
• Only missile defense systems capable of intercepting missiles capable of carrying a 500 kg payload or travelling at least 300 km or more are included. ⁶ In cases where a system was tested at least once against a missile of this type, other tests and operational uses, such as against satellites or short-range missiles, were also included.
• An “engagement” is defined as an attempt to destroy a single missile – either in a test or in combat. In general, we recorded whether the intercept was announced as a success. Cases where multiple interceptors were fired against a single missile were counted as a single engagement. As long as one of the interceptors successfully intercepted the target missile, then the entire engagement was recorded as a success.
• For combat engagements, we have two separate metrics for evaluating success. First, we record what the state conducting the intercept claimed. The engagement is marked as “successful” as long as the state conducting the intercept claimed it was successful. Second, we record our attempt at verification (“assessed intercept outcomes”) using technical reports or other publicly available evidence. In cases where we cannot complete this verification, we mark the claimed intercept outcome as “unverified.”

Test Performance and Transparency

Although twenty-six countries have either used, bought, or are actively acquiring missile defense systems, there are only three players in the export market (and to-date only the United States and Russia are major players). The United States exports the Patriot and THAAD systems to its allies, Russia exports the S-400 (including, controversially, to NATO ally Turkey), and there is a joint France-Italy-UK missile defense system called PAAMS in active development that may be exported eventually.

Among the nine countries testing or deploying missile defense systems (whether domestic or imported), we identified 365 separate engagements dating from 1960 to 2019. This dataset includes 221 test and 144 operational engagements. It is likely there are more tests and operational uses of these systems than we were able to identify. The reason for this is a considerable lack of transparency with the testing and combat use of these systems.

The United States is by far the most transparent of countries when it comes to missile defense tests, yet even reports from the United States provide only superficial details about most tests. This is similarly the case with systems developed independently by Israel, as well as those developed in partnership with the United States — such as the AEGIS SM-3 developed by the United States and Japan, and the Arrow, developed by the United States with Israel.

In general, the data show that most U.S. missile defense systems have good rates of success in test engagements with an overall 72 percent successful intercept rate. The United States has conducted 121 test engagements since 1963, and with the exception of the Ground-based Midcourse Defense (GMD) System, most U.S. systems appear to show fair performance under test conditions and have improved over time. For example, the Terminal High Altitude Area Defense (THAAD) system suffered severe trouble in the early days of its program. The first six THAAD test engagements, which occurred between 1995 and 1999, all failed. The program was then reorganized, and reportedly carried out 16 consecutive successful test engagements.

Other countries are considerably less transparent about their test data. India has carried out at least 14 intercept tests aimed at either rockets or satellites. In five of these intercepts, it was unclear from available public information what the target was, beyond some sort of stand-in for a hostile missile. Most surprising of all is India’s assertion that only one of these 14 tests, the country’s first attempt at intercepting a satellite in March, was unsuccessful. By comparison, many of the early U.S. system tests were publicly reported as failures (such as the aforementioned first six THAAD tests). The single observed failure with India is also interesting in that India attempted to conceal it as a mere flight test instead of an intercept test. ⁷ That we do not observe a significant number of early failures with India’s systems suggests there may be quite a few earlier developmental tests which are not publicly acknowledged or known.

Perhaps most questionable of all are Russian missile defense claims. Although it is not alone in so doing, Russia has made spectacular claims about the success of the S-400, yet there are few public reports about individual tests of the system. Our research was only able to identify 32 tests of the S-400. The timing of these tests is interesting, as all 32 tests reportedly occurred across only six exercises, four of which involved intercepting six targets or more. This type of engagement activity looks more like an operational exercise as opposed to developmental interceptor testing. During these tests, Russian forces fired an unknown number of interceptors, intercepting an unknown number of targets of unknown capability. Furthermore, Russia reported to state media that 100 percent of the attempted S-400 intercepts were successful. To date we have not been able to identify any reports of failed intercept tests involving the S-400. Like our hypothesis involving India, this suggests Russia is concealing most of its developmental tests or other failed intercepts. The lack of Russian testing transparency, at least with respect to the public (we don’t know what information Russia might share with prospective buyers), has not stopped other countries from expressing interest in or moving forward with purchasing the S-400 system. China purchased the S-400 system and claims to have conducted a successful intercept test. China’s purchase is particularly notable (within this context) because China has developed and tested its own missile defense interceptors. The S-400 purchase suggests China may intend to rely, at least in the short-term, on non-indigenous technology.

It is not surprising that overall global missile defense test results are strong. Flight tests are expensive, and there is significant political pressure on programs to make sure they succeed. As a result, test conditions are usually intentionally set up to be near-optimal. The defender often knows the details of the target missile in advance, as well as any countermeasures, and the planned launch time. A test can be scrubbed due to inclement weather or technical problems without being recorded as a failure, and officials frequently exaggerate the success of tests. We can see examples of this within the U.S. testing record. For example, the Missile Defense Agency reported the “successful intercept” of a target on 1 September 2006—and that success is recorded in the dataset. However, the Department of Defense later stated that the kill vehicle achieved only a “glancing blow” against the target and would not have destroyed the reentry vehicle. “I scored the FTG-02 flight test a hit,” the Pentagon’s Director of Operational Test and Evaluation later explained, “but not a kill.” ⁸ Favorably interpreting testing results like this is almost certainly not limited to the United States; it is only because of the relatively exceptional transparency around U.S. missile defense systems that we are able to observe and interrogate what is probably a global practice.

The unreliability of global test statistics makes it difficult to fully gauge the protection that current-generation missile defense systems actually provide. While that may be part of the point, the high number of countries looking to purchase missile defense systems suggests most of the policymakers doing so may be overestimating the protection the systems provide or failing to understand the narrow operational conditions under which they are likely to be effective.

Operational “Combat Use” Challenges

The real test of a missile defense system’s efficacy comes only in combat. Despite their deployment globally, missile defenses against the class of missiles described in this paper have only been used in combat in the Middle East. Furthermore, except for two isolated Israeli engagements in 2017 and 2018, missile defense intercepts have occurred in three major military operations, as shown in the below table.

Table 1: Combat Use of Missile Defense Systems by Conflict

It is extremely difficult to determine in the fog of war whether an engagement has been successful. Most missile defense systems will self-report whether an engagement has been successful or not, but this is based on the limited information available from the interceptor itself. The system knows whether the interceptor reached where it was trying to go and, if it is designed to explode, whether it did so. It cannot tell if the incoming missile was destroyed, or if there was even a missile at all. If a missile breaks up in flight or misses its target, it is often impossible to tell whether this was because of a problem with the missile itself, or as the result of an intercept.

The experience of the Patriot during the 1991 Gulf War is illustrative in this regard. Early on, the U.S. Army claimed a near perfect success rate. Later, the Army revised downward its estimate, stating that 90 percent of the missiles aimed at Saudi Arabia and 60 percent of the missiles fired at Israel that were engaged had been intercepted successfully. ⁹ When Congress challenged this claim, the Army further revised it downward, stating 52 percent of the missiles the Patriot system engaged were intercepted successfully. ¹⁰ Overall, however, the Army had “high confidence” in only 25 percent of the engagements. A September 1992 report by the Government Accountability Office (GAO) concluded that this figure was still too high and that only about “9 percent of the Patriot’s Operation Desert Storm engagements are supported by the strongest evidence that an engagement resulted in a warhead kill.” ¹¹ A subsequent analysis conducted by the Congressional Research Service (CRS) assessed the Army’s methodology in evaluating successful intercepts and concluded there was only one such engagement where the Army could “claim a successful engagement based on its own data and methodology.” Interestingly the report notes, “[F]or other reasons, the Army chose to assess that case as a no kill.” ¹² This suggests significant problems with how the Patriot’s intercept attempts were initially evaluated.

We identified 44 total Patriot engagements during the first Gulf War: 27 by U.S. forces and 17 by Israeli forces. In initial statements following the engagements the U.S. Army claimed to have successfully intercepted missiles in 26 cases. Given the limited public information available, it is not possible to verify these cases. Evidence such as ground damage and injuries, initially claimed to have resulted from falling debris from a successful intercept, were later shown to have involved probable missed intercepts. ¹³

The initial 60 percent success rate Israel claimed with Patriot (later downwardly revised multiple times to 50 and then 40 percent) ¹⁴ is also suspect. Statements from Israeli officials following the conflict even further cast doubt on these figures, with a 1991 Israeli Air Force report stating “there is no evidence of even a single successful intercept” and only “circumstantial evidence for one possible intercept.” ¹⁵ Given this poor and conflicting data, we could not verify any of the intercepts initially claimed as successful by Israel.

The U.S. Army claims that, during the 2003 invasion of Iraq, there were nine engagements against short-range Iraqi missiles such as the Al Samoud. “Eight of these engagements were observed by enough other sensors to conservatively declare them successes,” according to the Army, and “the ninth engagement is judged to be a probable success.” ¹⁶ These nine engagements during the 2003 invasion of Iraq, against much less sophisticated missiles than the scuds used in 1991 appear to be the only good candidates for potentially successful intercepts that we were able to identify in our research. ¹⁷ Even then, there is only one engagement for which we have some independent confirmation. A CNN crew was present during this engagement in March 2003, but CNN did not report on it at that time. “Had anyone reported this – had we reported it, or had it gotten out,” CNN national security analyst Ken Robinson later explained, “it would have enabled [Iraq] to know that they had the exact grid coordinate they needed” to target the headquarters of coalition ground forces. ¹⁸ As a result of the decision not to report the event, there is no independent information that would allow observers to say conclusively that the interceptors were successful in destroying incoming missiles.

Saudi Arabia continues to use Patriot to this day, and its forces are equipped with a mix of mostly PAC-2 and PAC-3 systems. Saudi Arabia has attempted to intercept missiles launched at targets in Saudi Arabia by Houthi forces in Yemen. Saudi Arabia and its allies have claimed at least 77 successful engagements out of the 89 total engagements that we were able to identify. In two cases, where missiles were targeted against Riyadh, we were able to use social media information to reconstruct the timeline and geometry of the intercepts. In both cases, by modeling where the missile body and warhead fell, it became clear that neither missile was successfully intercepted. ¹⁹ In one case, it is not clear to us that there was even a Saudi attempt to intercept the missile. The evidence appears to suggest that Saudi authorities are asserting that missiles have been successfully intercepted as a matter of routine, likely both to reassure the domestic population and to deter adversaries. Like the 1991 Gulf War intercept attempts, much of the information surrounding Saudi Arabia’s combat intercepts is unavailable to outside observers. In the two cases where we were able to examine publicly available evidence on missile intercept attempts (depicted in the Tableau interactive “Analyzing Missile Defense Failures”), the evidence suggests the systems are not effective. ²⁰

Conclusion

Why are missile defense systems so common despite the lack of evidence to support claims of their combat effectiveness? Some political leaders may genuinely believe the claims of effectiveness, even when those claims are not supported by the engagement record. But more importantly, military and political leaders may feel it is important to express confidence – in order to reassure an anxious public, protect military troops, and to deter adversaries.

False confidence in these missile defense systems could be particularly dangerous, especially when talking about strategic defenses. Combat uses to-date have involved short- and medium-range missiles, which, compared to ICBMs, are a significantly easier target for missile defense systems. If policymakers exaggerate the effectiveness of the Patriot, that is one issue. But what if the president exaggerates the effectiveness of defense against nuclear-armed missiles capable of targeting the United States? We have already seen senior U.S. officials offer exaggerated claims of the effectiveness of missile defenses with regard to North Korea. Placing unwarranted confidence in these systems might make a military option seem far safer that it really is – with disastrous consequences.

And then there is the added risk of arms races. Adversaries may well believe exaggerated claims about the effectiveness of missile defense systems, although this may not always deter them. In some cases, concerns about missile defense systems may spur adversaries to develop missiles or adopt tactics that can defeat missile defenses. Iran used cheap, low-flying drones in its September 2019 attack on Saudi oil facilities, while North Korea has developed a new generation of solid-propellant ballistic missiles that fly on trajectories that are difficult to intercept. Russia and China, too, are developing a range of new capabilities to defeat U.S. missile defense systems. There is growing evidence that Russia’s array of new nuclear weapons, such as a nuclear-powered cruise missile and an autonomous underwater vehicle, were developed in response to the U.S. withdrawal from the 1972 ABM treaty. Even if missile defenses don’t provide the security that advocates promise, they may still prompt adversaries to invest in dangerous and destabilizing capabilities.

Even if the first Patriot intercept against an Iraqi missile was not the success that missile defense advocates describe, the Commandant of the Air Defense Artillery School was right when he said that this event ushered in a new era. That new era, however, may turn out to be far less stable and much more dangerous than anyone expected.

Media inquiries about the database or accompanying graphics can be directed to Jessica Varnum at [email protected] or Michael Duitsman at [email protected]. Graphics created by CNS’s Shea Cotton, Jeffrey Lewis, David Steiger, and Jessica Varnum. Graphics are free to use/embed by media with the credit: “Created by the James Martin Center for Nonproliferation Studies for the Nuclear Threat Initiative.”