Back
Key Takeaways from the Florida Water Facility Hack
Now that the dust has settled on the recent Oldsmar, Florida water treatment facility breach, let’s take a deeper look at some of the lessons we can learn from the incident.
For those unfamiliar with the breach, on February 5, hackers accessed a Florida water facility that treats water for around 15,000 people near the Tampa area. The hackers were able to increase the amount of sodium hydroxide, or lye, distributed into the water supply, which is dangerous to consume at high levels. Luckily, there was an attendant that noticed the suspicious behavior and reported it, mitigating the breach without consequence.
They gained access to the computer system through TeamViewer, a popular remote access software application commonly used for remote desktop control, screen sharing, online meetings, and file transfers. Third party IT support is a common use case for TeamViewer and, according to its website, it has been installed on over 2.5 billion devices today. There has not been confirmation on how the attacker got ahold of the remote access system credentials, but we can speculate that an employee of the water facility fell victim to a social engineering attack, such as phishing.
Given the breach itself was not sophisticated and its impact was minimal, many in the cyber security community are surprised that this is making national headlines. But it is the potential of what could have happened that is causing a panic – and rightfully so.
Investigative journalist Brian Krebs interviewed a number of industrial control systems security experts, and discovered that there are approximately 54,000 distinct drinking water systems in the U.S. Of which, nearly all of them rely on some type of remote access to monitor and/or administer these facilities. Additionally, many of these facilities are unattended, underfunded, and do not have 24/7 monitoring of their IT operations. In other words, this type of breach is likely to happen again and, if we don’t take the necessary security considerations into account, the consequences could be devastating.
The industrial control systems and utilities notoriously prioritize operational efficiencies over security. This is a wakeup call for the industry to start looking at their systems from a security and safety perspective. To get started, here are the key lessons I learned from the incident.
Lessons Learned from the Florida Water Facility Breach
Many of the reports written about the breach are centered around remote access. That is not surprising as the security concerns of remote access and host-based security have escalated amid COVID-19. Host-based security represents a large attack surface that is rapidly evolving as employees continue to work disparately.
Think back to March 2020. Organizations needed to get people online fast and began enabling Remote Desktop Protocol (RDP) which is known to be vulnerable. Cyber security firm Kapersky found that the number of brute force attacks targeting RDP rose sharply after the onset of the coronavirus pandemic. Further, internet indexing service Shodan reported a 41 percent increase in RDP endpoints available on the internet as the virus began to spread. When determining the type of remote access to give systems the decision should be based on the level of security desired and which type of remote access is deemed appropriate.
That being said, in my opinion there is more to learn from this incident beyond the remote access system vulnerabilities.
It is critical to analyze your security program holistically
These systems are complex and require a design-level review to understand what could go wrong rather than completing ad hoc security assessments that look at the technology separately.
For example, say you performed an assessment of your desktop images and are notified that you have TeamViewer installed as a potential risk. This is something that is likely to get written off as a valid use case because it is how the IT team accesses the computer to troubleshoot operational issues remotely. Unless you assess all the systems involved in the environment and how they work together, it can be difficult to understand the risk your organization faces.
This is where threat modeling and design reviews prove vital. According to software security expert Gary McGraw, 50 percent of application security risks come in the form of software design flaws that cannot be identified by automated means. Threat modeling and design reviews leverage human security experts to evaluate your program in its entirety and provide you with an understanding of the current level of security in your software and its infrastructure components. Threat modeling in particular analyzes attack scenarios, identifies system vulnerabilities, and compares your current security activities with industry best practices. And with a design review, you gain clarity on where security controls exist and make strategic decisions on absent or ineffective controls.
Defense in depth is non-negotiable
The software the facility uses to increase the amount of sodium hydroxide should have never been able to reach dangerous levels in the first place. When software is developed, it should be built with security and safety in mind. For example, the maximum threshold should be an amount of sodium hydroxide that is safe, not one that is potentially life-threatening.
What if it was a disgruntled employee that decided to change the amount of sodium hydroxide? Or if the technology attendant had been bribed? The outcome of the situation would have looked much different.
It’s a best practice in security to create as much segregation in your operational technology (OT), or technology that physically moves things, and information technology (IT), the technology that stores and manages data, to avoid incidents that could result in physical harm. To achieve this, defense in depth is essential.
Defense in depth is a cyber security approach that layers defensive controls to ensure that, if one control fails, there will be another to prevent a breach. Authentication and access management are protections at the front line of a defense in depth strategy and a critical security pillar for industrial control systems and utilities. For systems or tasks that can have a detrimental impact if breached, add multiple layers of authentication so that not one computer or one individual can carry out the task. Additionally, adopting the concept of Least Privilege, or only allowing employees access to the minimum number of resources needed to accomplish their tasks, would be a good practice to implement industry wide.
We are not prepared for disaster scenarios
We are reliant on the use of outdated systems that are not prepared for certain disaster scenarios. For an industrial control system to experience downtime, it does not require an adversary to compromise a system. Look at what happened with the Texas winter storm. No one expected the weather to get that bad, but we could have better prepped our systems for it.
That is the challenge with utilities and industrial control systems. If you are not preparing for adversaries in tandem with natural disasters and other unforeseen circumstances, you could have major issues to deal with in the long run.
Another key factor to consider is time. When something goes wrong, coming up with the easiest, least expensive, and most feasible solution isn’t possible because of time constraints. And with water, heat, electricity, energy, or gas companies the pressure of time is mounting because they are critical part of our lives. Say your furnace in your home breaks when it is below freezing out. You typically have two options: have someone come out and evaluate the situation, wait weeks for the part, and fix the existing furnace or buy a new one and have it installed in days. To avoid frozen pipes and infrastructure issues, most would choose the fastest option. In a recent study, those who did not test their disaster recovery plan cited time and resources as the biggest barriers.
At utility facilities, there remains a lack of awareness around cyber security. Regular tabletop exercises that simulate a crisis scenario are necessary when working with systems this complex.
The three key learnings discussed in this blog should work in concert with one another. Use the findings from your holistic security assessment and dust off your disaster recovery and incident response plans to remediate your biggest security and safety gaps – and, in turn, strengthen your defense in depth.
