GAO Slams Testing Methods for New Port Detectors

By Jon Fox
Global Security Newswire

WASHINGTON — Testing conducted earlier this year on a next-generation radiation detector for U.S. ports was based on “biased” methodology and was not a rigorous examination of the technology’s capabilities, the Government Accountability Office said yesterday (see GSN, May 16).

What the Domestic Nuclear Detection Office describes as a calibration conducted prior to the testing of the machines, government auditors say may have favorably skewed the results of February and March testing in Nevada.

According to the GAO report, vendors of the new radiation detectors — called Advanced Spectroscopic Portal monitors — had been shown six of the seven nuclear source materials prior to the testing and all of the masking and shielding materials used in the evaluations.

In addition, nine of 16 configurations of nuclear, shielding and masking material used during earlier “dry runs” and “dress rehearsals” were the same combinations used during formal testing.

The Government Accountability Office contends this allowed the vendors to adjust the machines to get better results than they otherwise would, and “it is highly unlikely that such favorable circumstances would present themselves under real world conditions.”

The head of the Domestic Nuclear Detection Office, a division within the Homeland Security Department managing testing and deployment of the new detectors, said, however, that that is a “misrepresentation.”  The detectors he said could not have been completely recalibrated in the time available between the dry runs and formal testing.

The firms would have been unable to alter the complex computer programs that analyze the raw detector data in any significant way, Vayl Oxford said following a hearing of the House Energy and Commerce oversight committee.  The dry runs were more like a calibration necessary to set the machines, he said.

Oxford likened it to giving a working dog a scent.  If the dog does not know what it is looking for it will not be able to find the target.  The tests, he said, were not biased.

“You take a brand new set of potential explosive dogs, you get them used to their handlers, you never expose them to explosives and you say ‘Go find explosives,’” he said following the tense hearing regarding the testing.   “You’ve got to expose the systems that are doing the detection and identification to the threat materials of concern.”

The GAO report also contends that the tests were not designed to test the limitations of the devices’ detection capabilities, “a critical oversight in DNDO’s original test plan.”

The office, auditors said, did not use a “sufficient amount” of material that would mask or hide nuclear material that officials would likely encounter at U.S. ports.

Oxford conceded that there was a concern that the new detectors could return a false negative result in very rare instances and that additional testing would be conducted before deploying the new technology in a primary detection role.  At U.S. ports radiation detection is a tiered procedure, with secondary inspection taking place with additional machines if an alert is sounded.

“The cases that have been brought to our attention are very minute numbers, one in a million,” Oxford said, noting that in the past 18 months there have been only 24 instances at ports involving a masking agent that would have resulted in a false negative with the new detectors.

Oxford did not describe the specific scenario that could confuse the detectors but said in such particular case where the conditions for a potential false negative are present Customs and Border Protection officials could resolve the case with additional physical inspections.

“We don’t have to have these machines become perfect.  That’s why it’s hard for us to commit that we will test these to the outer limits because nobody will tell me what that is,” he said.  “We could be here five years from now continuing to do tests because someone will define a new outer limit.”

During the hearing, subcommittee Chairman Bart Stupak (D-Mich.), repeatedly called for “outer limit testing” of the new detectors.  He also called for “blind testing” in which the vendors are completely unaware of what type of material might be in the containers.

“We do it for the FDA, we do it for food safety, we do it for everywhere else; why shouldn’t we do it for radiation detection,” Stupak said.

The Government Accountability Office and the Domestic Nuclear Detection Office have been sparring for months over the next-generation radiation detectors. Federal auditors first assailed the office’s cost-benefit analysis as unsupportive of the planned $1.2 billion expenditure over five years for the new technology.

The criticism of the testing appears to be the latest salvo in what seems to have become an increasingly acrimonious disagreement between the agencies over the new detection systems.

GAO officials described difficulties eliciting information and test results from the Homeland Security division and during the hearing yesterday likened a fruitless back and forth between Oxford and Stupak over future testing plans to their own frustrations.

“This gives you some idea of what we’ve been trying to deal with,” Gene Aloise, director of the GAO Natural Resources and Environment Division.

The Government Accountability Office recommended that more testing and studies be conducted on the new monitors.  DNDO officials indicated yesterday that over the next six to 12 months there would be a series of computer models to resolve questions about the machines but that testing would not be completed before a recommendation is made to the homeland security secretary to certify the detectors as effective.

Congressional budget appropriators moved to hold fiscal 2007 funding for the new machines pending an official determination from the secretary that they are worth the high cost.  The systems cost as much as $500,000, compared to the $180,000 each current detector costs (see GSN, Oct. 3, 2006).

Lawmakers wrote language into both House and Senate versions of the fiscal 2007 Homeland Security appropriations bill requiring the secretary to find that new monitors represent a “significant increase in operational effectiveness” before funds are released.

Oxford said that that certification has nothing to do with deployment strategy and that the detectors would see a limited deployment with about 130 machines going to secondary screening locations as questions about placing ASPs in primary screening locations are resolved.

The original date for certification was set for late June, but that has been pushed back to accommodate more testing and the current projection for a decision is sometime in November.

Oxford explicitly declined to say he would wait until all test results were in before giving the secretary his thoughts on certification.  “I will not make that commitment today,” he said.

The new machines are not intended to represent an increase in detection sensitivity but rather allow port officials to both detect material and identify what isotope is being picked up by the monitors.  The current detectors, adapted from technology to stop radioactive scrap metal from going into processing plants, alert to the presence of radiation without determining what type is present.

That leads to thousands of false alarms that must be resolved with secondary hand-held detectors.  Materials such as kitty litter, granite and bananas can all set off the current detectors with trace amounts of natural radiation.  With the new detectors the roughly 500 containers that officials have to pay serious attention to resolve false alarms at the port of Los Angeles/Long Beach each day could be reduced to 20 to 25, Oxford said.

While Custom and Border Protection officials conceded that secondary inspections consume manpower and increase the officers’ workloads, they told GAO auditors that the current combination of deployed detectors and secondary scanning to resolve false alarms “provides the best possible radiological and nuclear screening coverage available with current technology,” the GAO reports states.

Back to top
About Newswire  |  Contact National Journal  |  Re-Use Guidelines © Copyright 2008 by National Journal Group, Inc. The material in this section is produced independently for NTI by National Journal Group, Inc. Any reproduction or retransmission, in whole or in part, is a violation of federal law and is strictly prohibited without the consent of the National Journal Group, Inc. All rights reserved.