Bill’s Security Blog

When most people talk about SCADA, they are generally including a whole lot of stuff that is not SCADA.

In general, true SCADA systems are and must be connected in some way. This is generally because they are located over a large geographical area. DCS systems are sometimes called SCADA, but they are not. DCS systems are what you find in small foot-print facilities, like a prison, a power plant, a bottling company.

These systems are generally similar in that they monitor and control a physical process. In the case of SCADA, these are dispersed over a large geographical area. In DCS, they are centrally located. In almost ALL cases, there is never a need to receive data from “outside” the SCADA or DCS system. The data is generated inside the system, and is often pumped out for analysis and monitoring. (It’s when we connected them that we screwed the poodle.)

This is important when it comes to securing these systems. When your digital assets are located over a large geographical area, you need some type of connectivity (4G, wired Internet, RF, whatever) that ties these together. This increases the attack surface.

However, if all your digital systems are in one small geographical area (e.g., a industrial plant), it’s easy to “snip the wire.”

Now, we can have a religious debate about air gaps - but that’s meaningless. When people say, “there’s no such thing as an air gap” they assume a whole bunch of conditions that may or may not be true. For the record, I will smack the next person that says, “there’s no such thing as an air gap.” The correct expression is “creating an air gap is hard.”

When securing industrial facilities that use digital equipment of various vintage and capability, creating an electronic air-gap is one step in implementing a robust “air gap”.

I try to think about the “air gap” as implementing controls to address “bad juju” that can get into or out of your digital equipment along 5 key vectors:

1. Direct network access (all types)
2. Wireless network access (all types)
3. Portable media and equipment (used to maintain digital assets, or to move data to-or-from digital assets)
4. Direct physical access (if you can touch it, you can pwn it…)
5. Supply chain (including vendor patching processes, and procurement-related issues)

Certainly, these can be collapsed into two vectors - logical and physical access - however, adding some granularity helps with discussion!

To illustrate - if the digital asset has no wired or wireless networking, physical access is controlled to authorized people with appropriate training, portable media and equipment that is connected to the system for maintenance and patching is rigorously protected, and we ensure any software or hardware coming in through the supply chain has reasonable degrees of security - then we have a well protected system. (Note, this gives you a good start on the prevention piece, but does not solve the monitoring/detection or incident response piece).

To establish an “air gap” we must address threats that can materialize along any of the 5 key vectors. If we do this successfully, then I will say, “you have a pretty good air gap.”

These vectors are very hard to address in a corporate environment, where the critical asset is DATA, and not hardware. The data is too easy to move, so it is nearly impossible to address all vectors to the data. Securing SCADA systems is also hard for similar reasons. But in terms of complexity, to get equivalent levels of protection for Data, SCADA, or DCS, I think the distribution of work would look as follows:
DATA - 85 %
SCADA - 10 %
DCS - 5 %
IE, if you spent 5% of your security budget on your DCS system security, you’d get equivalent protection as the 85% you spent to protect DATA.

The challenge is - we’ve spent decades working on data security (centuries if you consider crypto a part of data security), but only short years talking about SCADA and DCS security. So, as all humans do, we have to screw this up 1,000 times before we spend 10 minutes trying to figure out how to do it right once…

IMHO, end-user ’smart meter’ device security is the smallest issue to be resolved with moving toward a Smart Grid.

The real issue with Smart Grid is having thousands of electric devices connected to the network that have intermittent production capacities.

Currently, from what I understand, the electrical grid is somewhat sensitive to large changes in the amount of electricity being put onto or being pulled down from the grid.

As energy consumption goes up, power producers put more energy on the line.  As consumption goes down, those sources are throttled back.

But what do you do when you plug 1500 windmills into the grid?

What happens when the wind starts blowing, then suddenly stops.

Here’s a hypothetical.  Suppose a 1500 turbine wind farm, producing 15 MW of power and placing it on the grid.

If consumption = supply, we are all good.

But what happens if the wind, literally, stops blowing for 30 minutes.

Is everyone going to scramble to shut off their air conditioners and unplug their fridge?

With respect to Availability aspects of the CIA triad, we have a big issue here.

Compound that concern with the fact that electrons traveling across a grid monitoring system travel at the same speed as the electrons traveling from producers to consumers, and you get an ugly producer/cosumer problem.

An effective smart grid must mitigate these intermittent sources of power by ensuring that access to the grid happens in a controlled manner.

That involves rapid ability to disconnect an intermittent source, or to store it’s electricity for later consumption.

The devices that perform that function are, in my opinion, the biggest cyber risk.

Though it’s not necessarily security related, the other issue that needs to be addressed with the introduction of large-scale intermittent power sources on the grid, is the need to match all intermittent sources with 100% non-intermittent sources.

For example.  In our example above, in the case where  you have 15MW wind being put on the grid, you must have at least 15MW stand-by power producing capacity spinning and ready to dump energy onto the grid in the event that the wind dies down.

This issue is tough, and involves, essentially, rebuilding vast amounts of the grid to attempt to decentralize the alternate energy sources as broadly as possible.

So, for example, wind turbines in Maine might be providing power to consumers in Arizona during the night, but Solar Production in Arizona might be powering Air Conditioners in DC during the mid-day.  At the moment, our grid just ain’t built that way.

I was at a FERC Commissioner’s meeting last week, and I assure you, they weren’t talking about the issues with end-point monitors.  How you secure the devices protecting the grid as a whole was on everyone’s lips.

Bill Gross

Good article on the security of SCADA and control networks.

Some good background and some insights into how people might go about hacking these networks.  Social engineering almost always seems to work…

I know a lot of IT Security people don’t know a ton about SCADA or control networks, but they sure have an easy time hacking them.  I see a growing convergence between Control System people and IT people - hopefully they’ll talk about security.

Cyber watch

This is common theme for me.

Perhaps I should state my feelings clearly: “If you depend on wireless in a control system environment, you are friggin’ crazy!”

A few weeks ago, I wrote about a SCADA tool from Conlab called U.C.ME.

This tool touts two-way SMS management of your infrastructure! Sweet, eh?

Now, I wrote Conlab but haven’t heard anything meaningful back. Err… Well, I did. I got a quippy email… “You said you emailed us, but you never did.”

Well, I have tried again, and to no avail.

Looking at the product’s material, they seem to be referencing a company ControlSee.  I’ll contact them.  Perhaps Conlab is just a reseller?  Who knows.

I wrote Conlab, who I thought was the product creator, to ask them what kind of authentication they are doing, and how they are preventing simple things like replay attacks.

Crickets.

Google alerts just tossed me this:
“Imagine you could speak to your SCADA system… Send a text message to your SCADA system.”

More talk of controlling your infrastructure through remote access.

Zero talk of configuration and control of access.

Conlab, ControlSee, whoever, for the love of god and all that is holy, if you have any information on this, please send it to me. This kind of stuff keeps me up at night.

Bill

I have been spending some time lately researching the growing trend of using wireless and other remote connectivity services in SCADA networks and other control system applications.

Being an Infosec guy, this is hugely concerning.

Subversion of wireless and other remote technologies is very easy relative to a wired network at a facility.

SImple examples of the types of issues we can expect:

• Replay attacks
• DOS attacks
• Unauthorized disclosure
• Additional attack vectors

Some control systems (running a candy making facility, for example) - may not associate very high risk values to these vectors.

But what if you are a water plant, sewage treatment facility, auto manufacturer.

What kind of damage can be done if:

• An attacker can sent arbitrary alarm messages
• An attacker can send arbitrary alarm reset messages
• An attacker can flood the wireless device - preventing alarms from sounding

You get the idea.

On PACE, I just read a post: New software for wireless automation control, regarding Conlab’s U.C.ME product:

U.C.ME-OPC interfaces with any SCADA/HMI, OPC or DDE server to provide efficient, rapid and secure two-way communication with SCADA platforms, remote devices and users, via text messaging (SMS) or telephony, says the company.

Now, from what it looks like, the U.C.ME system does not alter settings on any devices - it’s simply a monitoring tool, but what if your command and control decisions are based on the information coming out of U.C.ME?

I have contacted Conlab for more information and will report back.

For now, it’s best to take some precautions:

1. Know that every message sent by your systems over a wireless network will be read by an adversary.
2. Assume that all data received from this system, or by this system may be forged.
3. Know that not hearing from the system doesn’t mean anything is going wrong, or that hearing from the system means there is a problem.

Essentially: If you use wireless technologies of any tye in the control system space, don’t trust the data at all… Not one bit.

Bill

From Electric Light & Power:
FERC proposes to close nuke cyber security gap

In an April 8, 2008 public joint meeting of FERC and the NRC, NRC staff indicated that the agency has proposed regulations to address cyber security at nuclear generating plants, but raised a concern about a potential gap in regulatory coverage because the requirements only will be directly associated with reactor safety security and emergency response, and will not extend to power continuity systems.

I’ve pondered the ramifications of this for a while.

One company, two regulatory bodies.

In the end, I believe that safety and security will be the casualty.

Nuclear power generating facilities in the US are regulated by the United States Nuclear Regulatory Commission.

Over the last year, NRC has gotten on the cyber security bandwagon. They have released a draft regulatory guide that, in theory, will increase the cyber security posture of nuclear power facilities.

In the absence of broad guidance, the industry has been operating under a policy developed by the industry, and approved by the NRC as an acceptable approach for ensuring cyber security.

Both these documents (much to my distress) are withheld from public disclosure.

The NRC’s job is to oversee the nuclear fleet (and other non-military nuclear related industry) and protect the public from potential harmful effects of radiation. After all, nuclear power plants in the US are using a radioactive isotope of Uranium (U235) as a heat source.

Since safety is the NRC’s primary mission, continuity of power concerns may be secondary.

The Federal Energy Regulatory Commission, on the other hand, is tasked with ensuring the safety, security, and continuity of the power grid.

In non-nuclear facilities, there is not an additional regulatory body concerned with the fuel source. The entire scope of operations falls under the regulatory oversight of the FERC.

The gap in regulatory oversight of nuclear generating facilities lies in the area between those set of assets that can be considered “safety related” and those that are considered non-safety related, but are important to continuity of power.

And things get really ugly when certain devices play both roles.

And that’s where this ball of wax gets really out of hand.

NRC is developing one set of standards for securing safety systems, and FERC (via NERC) has developed standards to ensure continuity of power.

Anyone who’s made it past basic algebra knows you can’t maximize two variables in an equation. You can maximize safety, or you can maximize your ability to ensure continuity of power, but not both.

Having grown up in Washington, I think I can smell which way the wind is blowing here.

The result will be that many systems will be under the regulation of two different entities, both with competing interests.

In the end, the operator will have to make a choice, and no matter which he chooses, he fails at achieving the requirements of either body.

In the situation, the best the operator can do is to shoot for the minimum set of configurations that meets all the requirements that match both sets of regulation.

I have to tell you, given that we are talking about a nuclear power reactor, I’d rather the operator not skimp on safety to ensure continuity of power.

What should be done?

It’s my opinion, having read the NRC’s draft regulatory guide, and spent some time looking over the NERC CIP’s that what we really need to do is take a step back.

I believe that continuity of power is a byproduct of the secure operation of any power facility.

I believe as well that safety is a byproduct of the same secure operation.

That said, I believe that what is best will be the development of a set of cyber security policies and procedures that are facility agnostic, and that ensure that digital systems within power generating facilities are secured.

Let the chips fall where they may. Implementing such a plan my require major modifications to the regulatory framework, but if it is what needs to be done, it needs to be done.

I can assure you that crackers do not care if FERC or NRC is running the show.

Bill

Quoting pieces from PC Magazine’s “Electrical Grid Vulnerable to Hackers, House Told”

“The harm could extend not only to the economy and the health and welfare of our citizens, but event to the ability of our military forces to defend us, since many military installations rely on the bulk power system for their electricity,” said Joseph Kelliher, chairman of the Federal Energy Regulatory Commission (FERC).

The Energy Policy Act of 2005 gave FERC the authority to approve reliability standards regarding the nation’s bulk power system. But FERC can only approve standards; it cannot actually craft them. That job falls to the North American Electric Reliability Corporation (NERC), which worked with the industry to develop standards it presented to FERC in August 2006. FERC gave those standards its final approval in January 2008.

That type of timeline is acceptable for most issues relating to the nation’s power system, but when it comes to possible cyber attacks, the government needs to be able to act within hours or days, not three years, Kelliher said.

FERC is limited by the paper shuffling and red tape created by the 2005 legislation. If FERC identifies a problem, it can order NERC to develop a solution within 60 days, but Kelliher said he is not sure NERC “could meet this schedule in practice.”

FERC “does not have sufficient authority to guard against national security threats to reliability of the electric system,” Kelliher said.

He said FERC should also be able to compel utilities to make changes, and the commission’s power should perhaps extend beyond the bulk power system, which at this point does not include Alaska, Hawaii, or local distribution facilities, which include major cities like New York and Washington, D.C.

Ok.

So after NERC’s lambasting in front of congress a few month’s ago, FERC probably feels like it needs to flex it’s muscles a little. You know, show us that it’s relevant in the space.

To me this smells of “bureaucrat seeks to increase his power using fear-mongering and scare tactics.”

The problem with granting FERC these additional powers will be obvious to anyone working in a regulated environment. Currently NERC issues standards to which utilities must comply. FERC is now seeking the ability to issue additional regulatory requirements.

You can’t serve two masters.

FERC is seeking these responsibilities under the guise that they need to respond to “urgent and sudden” threats.

But in reality, if they gain this power, it’s my contention that the situation will be worse, not better. An alternate approach should be considered.

So, what constitutes the kind of threat that would trigger a FERC requirement? Would it be the kind of threat that triggered the TSA to stop allowing passengers to carry any liquids on air-planes?

One core problem I see is that rapid, non-vetted changes in a system increases overall risk.

If this wasn’t the case, then why are our bulk power generation facilities setting up sophisticated configuration management (CM) procedures to ensure that their systems don’t move into an inconsistent state without sufficient vetting?

Additional regulation and mandate will not get us to a true, lasting security.

The regulatory environment is just plain broke in just about every instance that I see. The answer is not more regulation and oversight, the answer is using a different approach.

Achieving confidence in the security posture of the electrical grid requires a few simple things, that as of yet, are not done:

• Define what “a secure grid” looks like. A realistic view can help drive where we want to go. Security for security’s sake is a waste of time, effort, and money. Tell the industry where it needs to go.
• Build smart, responsible regulation that pushes the industry in a measurable way toward that goal.
• Develop tools and resources to facilitate this migration. For example, develop generic contract language that companies can use when making software and hardware purchases that shifts the burden of secure product development onto the appropriate party.
• Develop financial incentives to motivate the change. In the corporate security arena, you see a rapid movement toward data security in the wake of very costly data disclosures. Power operators don’t make money when the lights are out, but you can also incentivize good performance through corporate tax credits for up-time and availability.
• Foster an environment of open communication where all parties can come together to enhance their practices, policies, and procedures.

The writing is on the wall if the bulk power generation industry does not see the light and start moving aggressively toward showing a significant commitment to security.

Other industries should take note.

Bill

On June 11, Core Security Technologies released a remotely exploitable CitecSCADA ODBC vulnerability. (US-CERT VU#476345)

Exploit code is now in the wild.

A patch is available upon request from the vendor.

Bill

A good ‘overview’ article on Energy Central, “Securing the Grid”

This is a little scary, though:

Utilities, meanwhile, are trying to adapt to this changing world. The evidence suggests that while no company has ignored earlier advisories, those entities have varied understandings of what it required of them. In fact, all of the utilities interviewed by government regulators have requested more information as it relates to how they could be attacked and what mitigation efforts are available to stop such breaches.

Couple that paragraph with the fact that most control system people think their situation is unique, and that traditional IT security doesn’t have anything to offer them, and it’s a recipe for disaster.

Bill

According to Wall Street Journal’s MarketWatch, Waterfall Solutions will be presenting at the 2008 PCSF.

In a presentation on Wednesday, August 27th, Mr. Turniansky will give a detailed presentation on the topic of “Unidiresctional Connectivity - A Novel Robust Method for Absolute Protection of Process Control Systems”.

Some are too willing to throw out the notion of unidirectional communication in a control system because it requires some special configuration.

My take is that if you want Real security, you must consider the overwhelming value of a one-way communication link, particularly when critical control systems come into place.

In control networks, there are some devices, like sensors and other detectors, that simply generate traffic.  They do not need to receive anything.

Another possibility is to prevent two way traffic between systems at higher and lower levels of “criticality.”

A no-brainer example would be to employ one way communication between the contol network and the corporate network.  This would allow business units to monitor the control network, but not have data (and attacks) flow from the business side back.

This would have prevented the shutdown of Unit 2 of the Hatch nuclear power plant earlier this year.

From the Washington Post’s “Cyber Incident Blamed for Nuclear Power Plant Shutdown”

A nuclear power plant in Georgia was recently forced into an emergency shutdown for 48 hours after a software update was installed on a single computer.

The incident occurred on March 7 at Unit 2 of the Hatch nuclear power plant near Baxley, Georgia. The trouble started after an engineer from Southern Company, which manages the technology operations for the plant, installed a software update on a computer operating on the plant’s business network.

Though I don’t know all the details, it’s clear that there was some computer on the business network that must have, through some route, been able to feed back to the control network.

Why introduce the potential for a network based attack on one of these devices, simply employ one-way communication.

Method’s I’ve heard about in the past include physically modifying the network cabling so that the cable is physically incapable of sending traffic in both directions.

In any event, I wont’ be at PCSF, but hope the presentation is available.

Bill