Report

The Global Missile Defense Race: Strong Test Records and Poor Operational Performance

Part of The CNS Missile and SLV Launch Databases

The Global Missile Defense Race: Strong Test Records and Poor Operational Performance

Save to My Resources

Want to dive deeper?

Visit the Education Center

Shea Cotton

Senior Research Associate, The James Martin Center for Nonproliferation Studies

Jeffrey Lewis

Director of the East Asia Nonproliferation Program, The James Martin Center for Nonproliferation Studies

On 18 January 1991, news crews covering Operation Desert Storm from Saudi Arabia were on hand to witness what they described as the first ever combat use of a missile defense system to shoot down an incoming missile. “There’s a streak of light,” Charles Jaco, of CNN, said describing the intercept, “The [Patriot] missile goes North, cants slightly to the East, goes North again, disappears into the clouds, and then there’s an illuminating flash…” [1] On the ground, ABC News correspondent Forrest Sawyer remarked that, “the Patriot did its job perfectly.” [2] The event is still celebrated. Twenty-five years later, the Army Air Defense Artillery School invited one of the soldiers involved to participate in a commemoration of the first Patriot missile to shoot down an Iraqi Scud missile. “That single event,” the school’s commandant explained, “and the engagements that followed, ushered in a new era for air defense artillery.” [3] There is just one important missing detail – there was no Iraqi Scud missile fired that night. The Department of Defense would later admit that Iraq did not fire the first Scud in this area until January 20th, two days after the event. The Patriots fired at a false alarm.

To be fair, Desert Storm did usher in a new era. In the years since the Gulf War, missile defense systems designed to intercept ballistic missiles have become a common-place piece of military equipment for many countries. At least twenty-six countries either possess, or are in the process of purchasing, missile defense systems. Israel, Saudi Arabia, the United Arab Emirates, and the United States have all used missile defense systems in combat. [4] The commonplace nature of missile defense systems has led to a widespread assumption that these systems are as effective as advertised. Surely these systems must work, the argument goes, because if they did not, why would all of these countries want to buy them?

And, moreover, the widespread belief that defenses are working against short- and medium-range missiles has helped create the impression that the defenses will also work in much more demanding scenarios – such as an attack using intercontinental range ballistic missiles. When asked whether he believes the United States could shoot down a North Korean missile fired at the United States, General John Hyten, the Vice-Chairman of the Joint Chiefs of Staff, responded by saying “I have 100 percent confidence. Literally – I don’t say 100 percent very often. I have 100 percent confidence in those capabilities against North Korea.” [5]

The data we have compiled, however, does not support such confidence. Our research shows a substantial gap between the effectiveness of missile defense systems in combat and the public statements from government and industry officials about their effectiveness. Moreover, our research suggests that claims about successful interceptors often do not stand up to scrutiny. In fact, we are unable to identify a single instance in combat where an interceptor unambiguously engaged in the successful intercept of a ballistic missile. And this is for the easiest class of targets – short- and medium-range missiles. Intercepting an ICBM, fired by a country like Russia, China or North Korea, would be far more difficult. That is not to say that there has never been a successful intercept – it is usually extremely difficult to assess whether an engagement resulted in a kill or not. Almost thirty years after the first intercept attempt in combat, we are still searching for an unambiguous success.

The CNS Global Missile Defense Database & Methodology

The James Martin Center for Nonproliferation Studies (CNS) created the first publicly available global dataset of all known missile defense engagements for the Nuclear Threat Initiative (NTI) in order to better understand tests, operational uses, and the overall state of missile defenses worldwide today. To select by a specific missile defense system, click the dropdown arrow under “system name” and select a system.

The dataset of missile defense engagements includes both tests and instances of combat use. In order to focus on modern missile defense interceptors, the following criteria were adopted:

  • Most modern missile defense interceptors are hit-to-kill systems that use kinetic energy to destroy incoming nuclear warheads. Therefore, interceptors with hit-to-kill or explosive warheads are included, while tests of nuclear-armed interceptors or directed energy weapons are excluded.
  • The dataset only includes engagements that involved another missile, either as a target or as a real threat. Test interceptors against “virtual” targets or points in space, as well as operational launches in error or against radar blips, are excluded.
  • Only missile defense systems capable of intercepting missiles capable of carrying a 500 kg payload or travelling at least 300 km or more are included. [6] In cases where a system was tested at least once against a missile of this type, other tests and operational uses, such as against satellites or short-range missiles, were also included.
  • An “engagement” is defined as an attempt to destroy a single missile – either in a test or in combat. In general, we recorded whether the intercept was announced as a success. Cases where multiple interceptors were fired against a single missile were counted as a single engagement. As long as one of the interceptors successfully intercepted the target missile, then the entire engagement was recorded as a success.
  • For combat engagements, we have two separate metrics for evaluating success. First, we record what the state conducting the intercept claimed. The engagement is marked as “successful” as long as the state conducting the intercept claimed it was successful. Second, we record our attempt at verification (“assessed intercept outcomes”) using technical reports or other publicly available evidence. In cases where we cannot complete this verification, we mark the claimed intercept outcome as “unverified.”

Test Performance and Transparency

Although twenty-six countries have either used, bought, or are actively acquiring missile defense systems, there are only three players in the export market (and to-date only the United States and Russia are major players). The United States exports the Patriot and THAAD systems to its allies, Russia exports the S-400 (including, controversially, to NATO ally Turkey), and there is a joint France-Italy-UK missile defense system called PAAMS in active development that may be exported eventually.

Among the nine countries testing or deploying missile defense systems (whether domestic or imported), we identified 365 separate engagements dating from 1960 to 2019. This dataset includes 221 test and 144 operational engagements. It is likely there are more tests and operational uses of these systems than we were able to identify. The reason for this is a considerable lack of transparency with the testing and combat use of these systems.

The United States is by far the most transparent of countries when it comes to missile defense tests, yet even reports from the United States provide only superficial details about most tests. This is similarly the case with systems developed independently by Israel, as well as those developed in partnership with the United States — such as the AEGIS SM-3 developed by the United States and Japan, and the Arrow, developed by the United States with Israel.

In general, the data show that most U.S. missile defense systems have good rates of success in test engagements with an overall 72 percent successful intercept rate. The United States has conducted 121 test engagements since 1963, and with the exception of the Ground-based Midcourse Defense (GMD) System, most U.S. systems appear to show fair performance under test conditions and have improved over time. For example, the Terminal High Altitude Area Defense (THAAD) system suffered severe trouble in the early days of its program. The first six THAAD test engagements, which occurred between 1995 and 1999, all failed. The program was then reorganized, and reportedly carried out 16 consecutive successful test engagements.

Other countries are considerably less transparent about their test data. India has carried out at least 14 intercept tests aimed at either rockets or satellites. In five of these intercepts, it was unclear from available public information what the target was, beyond some sort of stand-in for a hostile missile. Most surprising of all is India’s assertion that only one of these 14 tests, the country’s first attempt at intercepting a satellite in March, was unsuccessful. By comparison, many of the early U.S. system tests were publicly reported as failures (such as the aforementioned first six THAAD tests). The single observed failure with India is also interesting in that India attempted to conceal it as a mere flight test instead of an intercept test. [7] That we do not observe a significant number of early failures with India’s systems suggests there may be quite a few earlier developmental tests which are not publicly acknowledged or known.

Perhaps most questionable of all are Russian missile defense claims. Although it is not alone in so doing, Russia has made spectacular claims about the success of the S-400, yet there are few public reports about individual tests of the system. Our research was only able to identify 32 tests of the S-400. The timing of these tests is interesting, as all 32 tests reportedly occurred across only six exercises, four of which involved intercepting six targets or more. This type of engagement activity looks more like an operational exercise as opposed to developmental interceptor testing. During these tests, Russian forces fired an unknown number of interceptors, intercepting an unknown number of targets of unknown capability. Furthermore, Russia reported to state media that 100 percent of the attempted S-400 intercepts were successful. To date we have not been able to identify any reports of failed intercept tests involving the S-400. Like our hypothesis involving India, this suggests Russia is concealing most of its developmental tests or other failed intercepts. The lack of Russian testing transparency, at least with respect to the public (we don’t know what information Russia might share with prospective buyers), has not stopped other countries from expressing interest in or moving forward with purchasing the S-400 system. China purchased the S-400 system and claims to have conducted a successful intercept test. China’s purchase is particularly notable (within this context) because China has developed and tested its own missile defense interceptors. The S-400 purchase suggests China may intend to rely, at least in the short-term, on non-indigenous technology.

It is not surprising that overall global missile defense test results are strong. Flight tests are expensive, and there is significant political pressure on programs to make sure they succeed. As a result, test conditions are usually intentionally set up to be near-optimal. The defender often knows the details of the target missile in advance, as well as any countermeasures, and the planned launch time. A test can be scrubbed due to inclement weather or technical problems without being recorded as a failure, and officials frequently exaggerate the success of tests. We can see examples of this within the U.S. testing record. For example, the Missile Defense Agency reported the “successful intercept” of a target on 1 September 2006—and that success is recorded in the dataset. However, the Department of Defense later stated that the kill vehicle achieved only a “glancing blow” against the target and would not have destroyed the reentry vehicle. “I scored the FTG-02 flight test a hit,” the Pentagon’s Director of Operational Test and Evaluation later explained, “but not a kill.” [8] Favorably interpreting testing results like this is almost certainly not limited to the United States; it is only because of the relatively exceptional transparency around U.S. missile defense systems that we are able to observe and interrogate what is probably a global practice.

The unreliability of global test statistics makes it difficult to fully gauge the protection that current-generation missile defense systems actually provide. While that may be part of the point, the high number of countries looking to purchase missile defense systems suggests most of the policymakers doing so may be overestimating the protection the systems provide or failing to understand the narrow operational conditions under which they are likely to be effective.

Operational “Combat Use” Challenges

The real test of a missile defense system’s efficacy comes only in combat. Despite their deployment globally, missile defenses against the class of missiles described in this paper have only been used in combat in the Middle East. Furthermore, except for two isolated Israeli engagements in 2017 and 2018, missile defense intercepts have occurred in three major military operations, as shown in the below table.

Table 1: Combat Use of Missile Defense Systems by Conflict
Table 1: Combat Use of Missile Defense Systems by Conflict
Of the 144 operational or “combat use” engagements recorded in this database, we had considerable difficulty identifying even a single, clear instance of a successful engagement.

It is extremely difficult to determine in the fog of war whether an engagement has been successful. Most missile defense systems will self-report whether an engagement has been successful or not, but this is based on the limited information available from the interceptor itself. The system knows whether the interceptor reached where it was trying to go and, if it is designed to explode, whether it did so. It cannot tell if the incoming missile was destroyed, or if there was even a missile at all. If a missile breaks up in flight or misses its target, it is often impossible to tell whether this was because of a problem with the missile itself, or as the result of an intercept.

The experience of the Patriot during the 1991 Gulf War is illustrative in this regard. Early on, the U.S. Army claimed a near perfect success rate. Later, the Army revised downward its estimate, stating that 90 percent of the missiles aimed at Saudi Arabia and 60 percent of the missiles fired at Israel that were engaged had been intercepted successfully. [9] When Congress challenged this claim, the Army further revised it downward, stating 52 percent of the missiles the Patriot system engaged were intercepted successfully. [10] Overall, however, the Army had “high confidence” in only 25 percent of the engagements. A September 1992 report by the Government Accountability Office (GAO) concluded that this figure was still too high and that only about “9 percent of the Patriot’s Operation Desert Storm engagements are supported by the strongest evidence that an engagement resulted in a warhead kill.” [11] A subsequent analysis conducted by the Congressional Research Service (CRS) assessed the Army’s methodology in evaluating successful intercepts and concluded there was only one such engagement where the Army could “claim a successful engagement based on its own data and methodology.” Interestingly the report notes, “[F]or other reasons, the Army chose to assess that case as a no kill.” [12] This suggests significant problems with how the Patriot’s intercept attempts were initially evaluated.

We identified 44 total Patriot engagements during the first Gulf War: 27 by U.S. forces and 17 by Israeli forces. In initial statements following the engagements the U.S. Army claimed to have successfully intercepted missiles in 26 cases. Given the limited public information available, it is not possible to verify these cases. Evidence such as ground damage and injuries, initially claimed to have resulted from falling debris from a successful intercept, were later shown to have involved probable missed intercepts. [13]

The initial 60 percent success rate Israel claimed with Patriot (later downwardly revised multiple times to 50 and then 40 percent) [14] is also suspect. Statements from Israeli officials following the conflict even further cast doubt on these figures, with a 1991 Israeli Air Force report stating “there is no evidence of even a single successful intercept” and only “circumstantial evidence for one possible intercept.” [15] Given this poor and conflicting data, we could not verify any of the intercepts initially claimed as successful by Israel.

The U.S. Army claims that, during the 2003 invasion of Iraq, there were nine engagements against short-range Iraqi missiles such as the Al Samoud. “Eight of these engagements were observed by enough other sensors to conservatively declare them successes,” according to the Army, and “the ninth engagement is judged to be a probable success.” [16] These nine engagements during the 2003 invasion of Iraq, against much less sophisticated missiles than the scuds used in 1991 appear to be the only good candidates for potentially successful intercepts that we were able to identify in our research. [17] Even then, there is only one engagement for which we have some independent confirmation. A CNN crew was present during this engagement in March 2003, but CNN did not report on it at that time. “Had anyone reported this – had we reported it, or had it gotten out,” CNN national security analyst Ken Robinson later explained, “it would have enabled [Iraq] to know that they had the exact grid coordinate they needed” to target the headquarters of coalition ground forces. [18] As a result of the decision not to report the event, there is no independent information that would allow observers to say conclusively that the interceptors were successful in destroying incoming missiles.

Saudi Arabia continues to use Patriot to this day, and its forces are equipped with a mix of mostly PAC-2 and PAC-3 systems. Saudi Arabia has attempted to intercept missiles launched at targets in Saudi Arabia by Houthi forces in Yemen. Saudi Arabia and its allies have claimed at least 77 successful engagements out of the 89 total engagements that we were able to identify. In two cases, where missiles were targeted against Riyadh, we were able to use social media information to reconstruct the timeline and geometry of the intercepts. In both cases, by modeling where the missile body and warhead fell, it became clear that neither missile was successfully intercepted. [19] In one case, it is not clear to us that there was even a Saudi attempt to intercept the missile. The evidence appears to suggest that Saudi authorities are asserting that missiles have been successfully intercepted as a matter of routine, likely both to reassure the domestic population and to deter adversaries. Like the 1991 Gulf War intercept attempts, much of the information surrounding Saudi Arabia’s combat intercepts is unavailable to outside observers. In the two cases where we were able to examine publicly available evidence on missile intercept attempts (depicted in the Tableau interactive “Analyzing Missile Defense Failures”), the evidence suggests the systems are not effective. [20]

Conclusion

Why are missile defense systems so common despite the lack of evidence to support claims of their combat effectiveness? Some political leaders may genuinely believe the claims of effectiveness, even when those claims are not supported by the engagement record. But more importantly, military and political leaders may feel it is important to express confidence – in order to reassure an anxious public, protect military troops, and to deter adversaries.

False confidence in these missile defense systems could be particularly dangerous, especially when talking about strategic defenses. Combat uses to-date have involved short- and medium-range missiles, which, compared to ICBMs, are a significantly easier target for missile defense systems. If policymakers exaggerate the effectiveness of the Patriot, that is one issue. But what if the president exaggerates the effectiveness of defense against nuclear-armed missiles capable of targeting the United States? We have already seen senior U.S. officials offer exaggerated claims of the effectiveness of missile defenses with regard to North Korea. Placing unwarranted confidence in these systems might make a military option seem far safer that it really is – with disastrous consequences.

And then there is the added risk of arms races. Adversaries may well believe exaggerated claims about the effectiveness of missile defense systems, although this may not always deter them. In some cases, concerns about missile defense systems may spur adversaries to develop missiles or adopt tactics that can defeat missile defenses. Iran used cheap, low-flying drones in its September 2019 attack on Saudi oil facilities, while North Korea has developed a new generation of solid-propellant ballistic missiles that fly on trajectories that are difficult to intercept. Russia and China, too, are developing a range of new capabilities to defeat U.S. missile defense systems. There is growing evidence that Russia’s array of new nuclear weapons, such as a nuclear-powered cruise missile and an autonomous underwater vehicle, were developed in response to the U.S. withdrawal from the 1972 ABM treaty. Even if missile defenses don’t provide the security that advocates promise, they may still prompt adversaries to invest in dangerous and destabilizing capabilities.

Even if the first Patriot intercept against an Iraqi missile was not the success that missile defense advocates describe, the Commandant of the Air Defense Artillery School was right when he said that this event ushered in a new era. That new era, however, may turn out to be far less stable and much more dangerous than anyone expected.

Media inquiries about the database or accompanying graphics can be directed to Sam Meyer at [email protected] or Michael Duitsman at [email protected]. Graphics created by CNS’s Shea Cotton, Jeffrey Lewis, David Steiger and Jessica Varnum. Graphics are free to use/embed by media with the credit: “Created by the James Martin Center for Nonproliferation Studies for the Nuclear Threat Initiative.”

Sources:
[1] Charles Jaco on Larry King Live, CNN, 18 January 1991, https://advance.lexis.com.
[2] Sherwood, Ben, “The Blip Seen ‘Round the World,” The Washington Post, 20 September 1992, www.washingtonpost.com.
[3] Crawley, Jeff, “Fort Sill remembers first Patriot missile intercept,” 22 January 2016. Accessed on 5 December 2019 at www.army.mil.
[4] Although twenty-six countries have either used, bought, or are actively acquiring missile defense systems, there are only 3 players in the export market (and to-date only the U.S. and Russia are the only major players). The United States exports the Patriot and THAAD systems to its allies, Russia does the same with the S-400 (including, controversially, to U.S. ally Turkey), and there is a joint France-Italy-UK missile defense system called PAAMS in active development. Egypt, The Netherlands, Germany, Kuwait, Jordan, Taiwan, Greece, Spain, South Korea, and Qatar have acquired U.S. Patriot systems. Romania, Poland, and Sweden are in the process of acquiring the Patriot. Currently, Belarus and Turkey have acquired the Russian S-400 system, and several other states have expressed interest in it. France is developing PAAMS (using the Aster-30 missile) alongside Italy and the UK (where it is called Sea Viper), though the system has only conducted a single intercept test against a long-range missile. Other states are developing their own systems that have not yet been tested, and several more claim to have “upgraded” older air defense systems to provide missile defense capabilities. These states are not listed in our database as of writing because we were not able to identify an intercept attempt associated with their systems.
[5] Yonhap, “U.S. military leader expresses full confidence in ability to defend against new NK missiles,” The Korea Herald, 18 January 2020, www.koreaherald.com.
[6] This includes both “Category I” and “Item 19” missiles under the Missile Technology Control Regime guidelines. For more information, see: Missile Technology Control Regime (MTCR) Equipment, Software And Technology Annex, MTCR/TEM/2017/Annex, 19 October 19 2017.
[7] Ankit Panda, “Exclusive: India Conducted a Failed Anti-Satellite Test in February 2019,” The Diplomat, 30 March 2019, https://thediplomat.com.
[8] Lewis, George, “What does ‘successful intercept’ mean? Maybe not what you think,” Mostly Missile Defense, 19 April 2012 https://mostlymissiledefense.com.
[9] Conyers, John, “The Patriot Myth: Caveat Emptor,” Arms Control Today, Vol. 22, No. 9. November 1992.
[10] Conyers, John, “The Patriot Myth: Caveat Emptor,” Arms Control Today, Vol. 22, No. 9. November 1992.
[11] “Operation Desert Storm: Data Does Not Exist to Conclusively Say How Well Patriot Performed,” GAO, September 1992. Accessed 15 November 2019 at www.gao.gov.
[12] Hildreth, Steven, “Evaluation of U.S. Army Assessment of Patriot Antitactical Missile Effectiveness in the War Against Iraq,” Congressional Research Services, prepared for the House Government Operations Subcommittee on Legislation and National Security, 7 April 1992. Accessed 15 November 2019 at https://apps.dtic.mil.
[13] For example, on 3 February 1991, a Scud aimed at Riyadh appears to have impacted a residential area injuring 29 people. CENTCOM later claimed the scud was successfully intercepted and the injuries and damage were the result of “debris.” See page 60 of “Performance of the Patriot missile in the Gulf War: hearing Before the Legislation and national Security subcommittee Of the Committee on Government operations House of representatives.” 7 April 1992.
[14] Conyers, John, “The Patriot Myth: Caveat Emptor,” Arms Control Today, Vol. 22, No. 9. November 1992; Gellman, Barton, “New Study Cuts Patriot Missile Success Rate to 9 Percent,” The Washington Post, 30 September 1992. Accessed 16 June 2020 at www.washingtonpost.com.
[15] Weiner, Tim, “Patriot Missile’s Success a Myth, Israeli Aides Say,” The New York Times, 21 November 1993. Accessed 16 June 2020 at www.nytimes.com.
[16] Department of Defense, Report of the Defense Science Board Task Force on Patriot System Performance, Washington, DC: Department of Defense, 2005. https://dsb.cto.mil.
[17] The rockets used, on their own, would not meet the 300 km threshold for inclusion in this database. It is only because the Patriot systems in use during the 2003 invasion were tested against systems that do meet that criteria that they are included.
[18] Schuster, Henry, “Iraqi missile targeted coalition HQ during war,” CNN, 29 May 2003, Accessed 5 December 2019 at www.cnn.com.
[19] Lewis, Jeffrey, “Patriot Missiles Are Made in America and Fail Everywhere,” Foreign Policy, 28 March 2018. Accessed 15 November 2019 at https://foreignpolicy.com.
[20] The Missile Defense Project at CSIS recently released a report examining the missile war in Yemen (“Report: The Missile War in Yemen”, Ian Williams and Shaan Shaikh, 9 June 2020). Numbers on missile defense intercepts (both successful and unsuccessful) were slightly different than our own, as were some of their conclusions. Our report attempted to verify, where possible, reports of successful intercepts and marked them as “unverified,” as we could not determine an outcome based solely on a report from the Saudi government or a coalition partner.

Explore the Collection

Your are currently on

Stay Informed

Sign up for our newsletter to get the latest on nuclear and biological threats.

Sign Up


Tutorial on Missiles and Other WMD Delivery Systems

Teaching Tool

Tutorial on Missiles and Other WMD Delivery Systems

Delivery systems―such as ballistic and cruise missiles, combat aircraft, and drones―determine how, when, and against whom a country can use conventional, nuclear, chemical, or biological weapons.



Glossary

Scud
Scud is the designation for a series of short-range ballistic missiles developed by the Soviet Union in the 1950s and transferred to many other countries. Most theater ballistic missiles developed and deployed in countries of proliferation concern, for example Iran and North Korea, are based on the Scud design.
Ballistic missile
A delivery vehicle powered by a liquid or solid fueled rocket that primarily travels in a ballistic (free-fall) trajectory.  The flight of a ballistic missile includes three phases: 1) boost phase, where the rocket generates thrust to launch the missile into flight; 2) midcourse phase, where the missile coasts in an arc under the influence of gravity; and 3) terminal phase, in which the missile descends towards its target.  Ballistic missiles can be characterized by three key parameters - range, payload, and Circular Error Probable (CEP), or targeting precision.  Ballistic missiles are primarily intended for use against ground targets.
Intercontinental ballistic missile (ICBM)
Intercontinental ballistic missile (ICBM): A ballistic missile with a range greater than 5,500 km. See entry for ballistic missile.
North Atlantic Treaty Organization (NATO)
The North Atlantic Treaty Organization is a military alliance that was formed in 1949 to help deter the Soviet Union from attacking Europe. The Alliance is based on the North Atlantic Treaty, which was signed in Washington on 4 April 1949. The treaty originally created an alliance of 10 European and two North American independent states, but today NATO has 28 members who have committed to maintaining and developing their defense capabilities, to consulting on issues of mutual security concern, and to the principle of collective self-defense. NATO is also engaged in out-of-area security operations, most notably in Afghanistan, where Alliance forces operate alongside other non-NATO countries as part of the International Security Assistance Force (ISAF). For additional information, see NATO.
Deployment
The positioning of military forces – conventional and/or nuclear – in conjunction with military planning.
Theater High Altitude Air Defense (THAAD)
THAAD: The U.S. Army's air defense program designed to provide extended defense, and to engage an incoming missile at ranges of up to several hundred kilometers. THAAD deploys a hit-to-kill interceptor equipped with an infrared seeker. The interception is intended to occur outside the earth's atmosphere, or high in the atmosphere.
Reentry Vehicle (RV)
A nuclear warhead on a ballistic missile specially designed to reenter the earth's atmosphere in the terminal portion of the missile's trajectory.

Close

My Resources