On Feb. 16, less than two weeks after a mysterious attacker made headlines around the world by hacking a water treatment plant in Oldsmar, Florida, and nearly generating a mass poisoning, the city's mayor declared victory.
"This is a success story," Mayor Eric Seidel told the City Council in Oldsmar, a suburb of 15,000 people, after acknowledging "some deficiencies." As he put it, "our protocols, monitoring protocols, worked. Our staff executed them to perfection. And as the city manager said, there were other backups. ... We were breached, there's no question. And we'll make sure that doesn't happen again. But it's a success story." Two council members congratulated the mayor, noting his turn at the press conference where the hack was disclosed. "Even on TV, you were fantastic," said one.
"Success" is not the word that cybersecurity experts use to describe the Oldsmar episode. They view the breach as a case study in digital ineptitude, a frightening near-miss and an example of how the managers of water systems continue to downplay or ignore years of increasingly dire warnings.
The experts say the sorts of rudimentary vulnerabilities revealed in the breach — including the lack of an internet firewall and the use of shared passwords and outdated software — are common among America's 151,000 public water systems.
"Frankly, they got very lucky," said retired Adm. Mark Montgomery, executive director of the federal Cyberspace Solarium Commission, which Congress established in 2018 to upgrade the nation's defenses against major cyberattacks. Montgomery likened the Oldsmar outcome to a pilot landing a plane after an engine caught fire during a flight. "They shouldn't celebrate like Tom Brady winning the Super Bowl," he said. "They didn't win a game. They averted a disaster through a lot of good fortune."
The motive and identity of the hackers, foreign or domestic, remain unknown. But Montgomery and other experts say a more sophisticated hacker than the one in Oldsmar, who attempted to boost the quantity of lye in the drinking water to dangerous levels, could have wreaked havoc. They're skeptical of the city's assurances that "redundant" electronic monitors at the plant protected citizens from any possible harm. "If the attackers could break into the lye controls," Montgomery said, "don't you think they could break into the alarm system and alter the checkpoints? It's a mistake to think a hacker could not introduce contaminated water into our water systems." Oldsmar officials, citing the ongoing investigation, declined ProPublica's requests for an interview or to address emailed questions about the city's cybersecurity practices.
The consequences of a major water system breach could be calamitous: thousands sickened from poisoned drinking water; panic over interrupted supplies; widespread flooding; burst pipes and streams of overflowing sewage. (This is not merely theoretical. In 2000, a former municipal wastewater contractor in Australia, rejected for a city job, remotely manipulated computer control systems to release 264,000 gallons of raw sewage, which poured into public parks, turned creek water black, spilled onto the grounds of a Hyatt Regency Hotel and generated a stench that investigators called "unbearable." The man was sentenced to two years in prison.)
In congressional testimony on March 10, Eric Goldstein, cybersecurity chief for the federal Cybersecurity and Infrastructure Security Agency, described the Oldsmar incident as illustrating "the gravest risk that CISA sees from a national standpoint." He said it should be "a clarion call for this country for the risk that we face from cyberintrusions into these critical systems."
Grave warnings have sounded for years. As far back as 2011, a Department of Homeland Security alert advised that hackers could gain access to American water systems using "readily available and generally free" internet search tools. Such admonitions have abounded in recent years. Booz Allen Hamilton's 2019 "Cyber Threat Outlook" called America's water utilities "a perfect target" for cyberattacks; a 2020 Journal of Environmental Engineering review found "an increase in the frequency, diversity, and complexity of cyberthreats to the water sector"; and the Cyberspace Solarium Commission's March 2020 report warned that America's water systems "remain largely ill-prepared to defend their networks from cyber-enabled disruption."
Despite the warnings, and some high-profile breaches dating back a decade, the federal government has largely left cyberdefense to the water utilities. For years, it relied on voluntary industry measures, dismissing any need for new regulation. Then, in 2018, Congress included a provision addressing cybersecurity in a 129-page water bill that covered everything from river levee repairs to grants for school water fountains.
The requirements were less than demanding. Every U.S. water system serving more than 3,300 customers was obliged to conduct a self-assessment of the risks and resilience of its physical and electronic systems and prepare an emergency-response plan. Different-sized utilities got different deadlines; for the smallest covered by the law, such as Oldsmar, the self-assessment must be done by June 30, 2021, more than two and a half years after the law was signed. (Oldsmar had completed its cybersecurity review by early November but hadn't yet incorporated its recommendations in the city's emergency response plan before the February hack, according to a statement provided by the city manager.) Tens of thousands of U.S. water systems with fewer than 3,300 customers were exempted entirely from the law's requirements.
Those utilities required to perform a self-assessment were not obliged to submit a report to any government agencies. The utilities merely had to attest to the Environmental Protection Agency that they had conducted the assessment. The 2018 legislation also provided $30 million for grants to help water districts deal with "risk and resilience" problems, including cyberattacks. But Congress never appropriated that money.
The water provisions fall far short of federal requirements (including penalties for violating those rules) and funding aimed at protecting electricity infrastructure, according to Montgomery. "An assessment's a good thing," he said. "But this is well short of what we require from energy companies. We have developed a tool for self-identification of problems. But if you're really bad at cybersecurity, I'm not sure your self-identification is going to solve the problem."
He also pointed to low staffing at the EPA's Water Security Division. "The water security office is a handful of people, probably three," Montgomery said. "It historically has not done much, if any, cybersecurity work. This is the product of 20 years of low prioritization." The agency's most recent report to Congress on "Drinking Water Infrastructure Needs," submitted in 2018, identified $472.6 billion in long-term priorities, but it didn't mention the word "cybersecurity" once in its 75 pages.
An EPA official, speaking on the condition of anonymity, agreed that the agency had only "a small team" devoted to water cybersecurity but said Oldsmar "and other recent incidents have highlighted the importance of the priority and the investments we need to make."
The origins of the problem are clear. The vast majority of the nation's water systems are small and publicly owned, with limited resources and aging infrastructure. As they turned to digital systems and monitors to boost efficiency while saving money and staff, they failed to install the safeguards and carry out employee training needed to secure the resulting vulnerabilities. "Every one of them had one guiding principle over the last 50 years: increased automation to lower the size of the workforce to keep costs down," Montgomery said. "Along with that, there should have been an investment in the cybersecurity of the infrastructure. But that did not happen."
Traditionally focused on physical risks, such as natural hazards, burst pipes and on-site intruders, most water systems also have little or no in-house IT staff. The pandemic, which encouraged remote management, has only made the problem worse. In testimony last month to the House Homeland Security Committee, former CISA Director Chris Krebs called Oldsmar's vulnerability "probably the rule rather than the exception. ... These are municipal facilities that do not have sufficient resources to have robust security programs. That's just the way it goes."
The industrial control systems that water districts use to manage valves, pipes and other infrastructure are notoriously open to attack. A 2018 study by IBM and a private security company found 17 major vulnerabilities in equipment widely deployed in "smart cities," a term that refers to municipalities that manage a wide array of their systems — anything from water treatment plants to parking meters and streetlamps — via the Internet. Among the security problems: Every product the group examined was still using the default passwords (such as "admin") they came with in the box, allowing "even the most novice hacker to easily gain access to these devices." A 2018 study by the firm Positive Technologies reported that it was able to penetrate nearly three-fourths of industrial organizations it investigated, revealing gaps offering hackers "plenty of opportunity to access critical equipment." The most common vulnerabilities: remote-access networks, obvious passwords and software so old that the manufacturer had stopped making fixes to protect against intruders. The report found that vulnerabilities known for years often "remain untouched, because organizations are afraid to make any changes that might cause downtime."
These industrial control systems are considered such obvious targets that hacking contests use them as quarry. At the DEFCON computer security conference, an "ICS Village" let curious programmers try to break into devices set up inside a Las Vegas hotel room — demos not connected to real-life systems — in an effort to expose weaknesses. At the event in 2018, one water pipe control system, likely used for a commercial building, had its computer screen defaced with graffiti-type messages.