Your Tax Dollars At Work: How Commerce Dept. Spent $2.7 Million Cleaning Out Two Malware-Infected Computersby Tim Cushing
Jul. 10, 2013
Progress: "Artist" Who Breastfed Dog, Fertilized Her Own Egg With Dog Cell Wins Prestigious Prize
Somali Terrorist Behind Canada Attack Entered U.S. From Mexico, Obama DHS Let Him Go
Anti-Establishment, Trump-Like Billionaire Wins Czech Election
U. Of Penn Teaching Aide: I "Always" Call On Black Female Students First, White Men Last
Trump: I Will Allow 'Long Blocked And Classified JFK Files' To Be Released
The cyber-Pearl Harbor is upon us and the only way to defeat it is to sink our own ships at the first sign of invasion. This is the sort of thing that happens when the legislators and advisors with the loudest voices value paranoia over rational strategy. The Department of Commerce, aided by a tragicomic string of errors, managed to almost stamp out its malware problem.
The Commerce Department's Economic Development Administration spent almost half of its IT budget last year to remediate a cyber attack that barely happened.Also included in the mass destruction were cameras and TVs. It wasn't just cyber-paranoia that led to this hardware cull. There was plenty of miscommunication too, along with the usual doses of bureaucratic clumsiness. The Inspector General's report breaks down the chain of missteps, which all began with a response team member grabbing the wrong network info.
In an effort to identify infected components, DOC CIRTís (Dept. of Commerce Computer Incident Response Team) incident handler requested network logging information. However, the incident handler unknowingly requested the wrong network logging information... Instead of providing EDA a list of potentially infected components, the incident handler mistakenly provided EDA a list of 146 components within its network boundary. Accordingly, EDA believed it faced a substantial malware infection.Yes. Much like "Reply" and "Reply All" will both get the job done, only one is the correct choice when firing off a devastating critique of your soon-to-be-former coworkers. The same goes for network logs. One shows you the correct info. The other "indicates" that more than half the EDA's computers are suffering from a malware infection.
DOC CIRT did try to get this fixed, pointing out the error to the handling team and re-running the analysis using the correct network log. Turns out, the original estimate was slightly off.
The HCHB network staff member then performed the appropriate analysis identifying only two components exhibiting the malicious behavior in US-CERTís alert.This new data in hand, a notification was sent out ostensibly to clear things up, but this too was mishandled so badly someone unfamiliar with bureaucratic ineptitude might be inclined to suspect sabotage.
DOC CIRTís second incident notification did not clearly explain that the first incident notification was inaccurate. As a result, EDA continued to believe a widespread malware infection was affecting its systems.For five weeks, things went from bad to worse to comically tragic to tragically comic to full-scale computercide. Looking at its list (2 components), DOC CIRT asked the EDA to attempt containment by reimaging the infected items. Looking at its list (146 components), the EDA responded that reimaging half its devices would be "unfeasible." Taking a look at the EDA's list (from the first, mistaken network log analysis), DOC CIRT assumed the EDA had received additional analysis indicating the malware had spread, and changed its recommendations accordingly.
Finally, both departments were on the same (but entirely wrong) page and scaled up the response accordingly. A copy went to the DHS, stating that "over 50%" of the EDA's devices were infected. The DHS then accepted this without seeking independent confirmation. The NSA cranked out its own concerned report, quoting heavily from the DHS report (which was still in draft form), both of which were based on DOC CIRT's first erroneous report. This went undetected for over a year, until the OIG informed the involved agencies of its findings in December 2012.
The end result? The EDA and DOC CIRT worked together, attempting to head off a "severe" malware threat before it spread to other connected government computers. Despite gathering more information from outside consultants that indicated the malware was neither "persistent" nor a threat to migrate, the two agencies began destroying devices in May of 2012, finally stopping three months later when the "break stuff" budget had been exhausted.
Fortunately for the agencies, taxpayers and the surviving equipment (valued at over $3 million), the OIG's findings were brought to the agencies' attention before the fiscal year began and a new "break stuff" budget approved. All in all, the EDA spent over $2.7 million fighting a malware "infection" confined to two computers.
There's nothing in this report that makes the EDA look good. A chart on page 8 shows the EDA has persistently ignored the OIG's recommendations on agency computer security, with some assessments going back as far as 2006. It's no surprise it managed to (along with the Dept. of Commerce's response team) transform a 2-computer infection into a nearly $3 million catastrophe.
OIG 13 027 A (PDF)