ST. PETERSBURG, Fla. – A hacker’s botched attempt to poison the water supply of a small Florida city is raising alarms about just how vulnerable the nation’s water systems may be to attacks by more sophisticated intruders. Treatment plants are typically cash-strapped and lack the cybersecurity depth of the power grid and nuclear plants.
A local sheriff's startling announcement Monday that the water supply of Oldsmar, population 15,000, was briefly in jeopardy last week exhibited uncharacteristic transparency. Suspicious incidents are rarely reported and usually are chalked up to mechanical or procedural errors, experts say. No federal reporting requirement exists, and state and local rules vary widely.
Recommended Videos
"In the industry, we were all expecting this to happen. We have known for a long time that municipal water utilities are extremely underfunded and under-resourced, and that makes them a soft target for cyberattacks,” said Lesley Carhart, principal incident responder at Dragos Security, which specializes in industrial control systems.
“I deal with a lot of municipal water utilities for small, medium and large-sized cities. And in a lot of cases, all of them have a very small IT staff. Some of them have no dedicated security staff at all,” she said.
The nation's 151,000 public water systems lack the financial fortification of the corporate owners of nuclear power plants and electrical utilities. They are a heterogenous patchwork, less uniform in technology and security measures than in other rich countries.
As the computer networks of vital infrastructure become easier to reach via the internet — and with remote access multiplying dizzily during the COVID-19 pandemic — security measures often get sacrificed. That appeared to be the case at Oldsmar.
Cybersecurity experts said the attack at the plant 15 miles northwest of Tampa seemed ham-handed, it was so blatant. Whoever breached Oldsmar’s plant on Friday using a remote access program shared by plant workers briefly increased the amount of lye — sodium hydroxide — by a factor of 100, according to Pinellas County Sheriff Bob Gualtieri. Lye is used to lower acidity, but in high concentrations it is highly caustic and can burn. It's found in drain cleaning products.
How the hacker got in remains unclear, Gualtieri said. But some details have emerged.
An advisory that Massachusetts posted for its public water suppliers said the intruder entered through a remote-access program called TeamViewer. It was loaded on all computers used by plant personnel, all of which were connected to the plant’s control system, the advisory said, adding that all users shared the same password — ignoring cybersecurity best practices. Further, those computers “appeared to be connected directly to the Internet without any type of firewall protection installed.”
The Massachusetts advisory said the FBI and other agencies had issued a situational report on the incident. An FBI spokesperson declined to comment on the report.
Oldsmar officials declined to questions about cybersecurity measures at the plant.
The intruder's timing and visibility seemed almost comical to cybersecurity experts. A supervisor monitoring a plant console about 1:30 p.m. saw a cursor move across the screen and change settings, Gualtieri said, and was able to immediately reverse it. The intruder was in and out in five minutes.
The public was never in peril, though the intruder took “the sodium hydroxide up to dangerous levels,” the sheriff said. Also, plant safeguards would have detected the chemical alteration in the 24 hours to 36 hours it would have taken to affect the water supply, he said.
Gualtieri said Tuesday that water goes to holding tanks before reaching customers, and “it would have been caught by a secondary chemical check.” He did not know if the hacker was domestic or foreign, and said no one related to a plant employee was suspected. He said the FBI and Secret Service were assisting in the investigation.
Jake Williams, CEO of the cybersecurity firm Rendition Infosec, said engineers have been creating safeguards "since before remote control via cyber was a thing,” making it highly unlikely the breach could have led to “a cascade of failures” tainting Oldsmar's water.
There's been an uptick in hacking attempts of water treatment plants in the past year, the cybersecurity firm FireEye said, but most were by novices, many stumbling on systems while using a kind of search engine for industrial control systems called Shodan. At a congressional hearing Wednesday, former Cybersecurity and Infrastructure Security Agency director Christopher Krebs said he thought it “very likely” the Oldsmar hacker was a disgruntled employee.
The serious threat is from nation-state hackers such as the Russian agents blamed for the monthslong SolarWinds campaign that has plagued U.S. agencies and the private sector for at least eight months and was discovered in December. While U.S. officials have called SolarWinds a grave threat, they also call it cyberespionage, rather than an attempt to do damage.
Laying boobytraps that could be triggered in an armed conflict is another matter. Russian hackers are known to have infiltrated U.S. industrial control systems, including the power grid, and Iranian agents are blamed for the breach of a suburban New York dam in 2013. But there is no indication any “logic bombs” have been activated, as Russia did in Ukraine when military hackers briefly brought down parts of the electrical grid in the winters of 2015 and 2016.
A 2020 paper in the Journal of Environmental Engineering found that water utilities have been hacked by a variety of intruders, including amateurs just poking around, disgruntled former employees, cybercriminals looking to profit and state-sponsored hackers. Although such incidents have been relatively few, that does not mean the risk is low and that most water systems are secure.
After Friday's incident, Oldsmar officials disabled the remote-access system and warned other city leaders in the region — which was hosting the Super Bowl — to check their systems.
Chris Sistrunk, a technical manager at FireEye's Mandiant division, said cybersecurity issues are relatively new for U.S. water utilities, whose biggest problems are pipes freezing and busting in winter or getting clogged with disposable wipes. The Oldsmar hack highlights the need for more training and basic security protocols, but not drastic measures such as sweeping new regulations.
“We have to do something, we can’t do nothing. But we can’t overreact,” he said.
___
Bajak reported from Boston and Suderman from Richmond, Virginia. AP Technology Writer Matt O’Brien contributed from Providence, Rhode Island.