Bill’s Security Blog

IMHO, end-user ’smart meter’ device security is the smallest issue to be resolved with moving toward a Smart Grid.

The real issue with Smart Grid is having thousands of electric devices connected to the network that have intermittent production capacities.

Currently, from what I understand, the electrical grid is somewhat sensitive to large changes in the amount of electricity being put onto or being pulled down from the grid.

As energy consumption goes up, power producers put more energy on the line.  As consumption goes down, those sources are throttled back.

But what do you do when you plug 1500 windmills into the grid?

What happens when the wind starts blowing, then suddenly stops.

Here’s a hypothetical.  Suppose a 1500 turbine wind farm, producing 15 MW of power and placing it on the grid.

If consumption = supply, we are all good.

But what happens if the wind, literally, stops blowing for 30 minutes.

Is everyone going to scramble to shut off their air conditioners and unplug their fridge?

With respect to Availability aspects of the CIA triad, we have a big issue here.

Compound that concern with the fact that electrons traveling across a grid monitoring system travel at the same speed as the electrons traveling from producers to consumers, and you get an ugly producer/cosumer problem.

An effective smart grid must mitigate these intermittent sources of power by ensuring that access to the grid happens in a controlled manner.

That involves rapid ability to disconnect an intermittent source, or to store it’s electricity for later consumption.

The devices that perform that function are, in my opinion, the biggest cyber risk.

Though it’s not necessarily security related, the other issue that needs to be addressed with the introduction of large-scale intermittent power sources on the grid, is the need to match all intermittent sources with 100% non-intermittent sources.

For example.  In our example above, in the case where  you have 15MW wind being put on the grid, you must have at least 15MW stand-by power producing capacity spinning and ready to dump energy onto the grid in the event that the wind dies down.

This issue is tough, and involves, essentially, rebuilding vast amounts of the grid to attempt to decentralize the alternate energy sources as broadly as possible.

So, for example, wind turbines in Maine might be providing power to consumers in Arizona during the night, but Solar Production in Arizona might be powering Air Conditioners in DC during the mid-day.  At the moment, our grid just ain’t built that way.

I was at a FERC Commissioner’s meeting last week, and I assure you, they weren’t talking about the issues with end-point monitors.  How you secure the devices protecting the grid as a whole was on everyone’s lips.

Bill Gross

Good article on the security of SCADA and control networks.

Some good background and some insights into how people might go about hacking these networks.  Social engineering almost always seems to work…

I know a lot of IT Security people don’t know a ton about SCADA or control networks, but they sure have an easy time hacking them.  I see a growing convergence between Control System people and IT people - hopefully they’ll talk about security.

Cyber watch

This is common theme for me.

Perhaps I should state my feelings clearly: “If you depend on wireless in a control system environment, you are friggin’ crazy!”

A few weeks ago, I wrote about a SCADA tool from Conlab called U.C.ME.

This tool touts two-way SMS management of your infrastructure! Sweet, eh?

Now, I wrote Conlab but haven’t heard anything meaningful back. Err… Well, I did. I got a quippy email… “You said you emailed us, but you never did.”

Well, I have tried again, and to no avail.

Looking at the product’s material, they seem to be referencing a company ControlSee.  I’ll contact them.  Perhaps Conlab is just a reseller?  Who knows.

I wrote Conlab, who I thought was the product creator, to ask them what kind of authentication they are doing, and how they are preventing simple things like replay attacks.

Crickets.

Google alerts just tossed me this:
“Imagine you could speak to your SCADA system… Send a text message to your SCADA system.”

More talk of controlling your infrastructure through remote access.

Zero talk of configuration and control of access.

Conlab, ControlSee, whoever, for the love of god and all that is holy, if you have any information on this, please send it to me. This kind of stuff keeps me up at night.

Bill

I have been spending some time lately researching the growing trend of using wireless and other remote connectivity services in SCADA networks and other control system applications.

Being an Infosec guy, this is hugely concerning.

Subversion of wireless and other remote technologies is very easy relative to a wired network at a facility.

SImple examples of the types of issues we can expect:

• Replay attacks
• DOS attacks
• Unauthorized disclosure
• Additional attack vectors

Some control systems (running a candy making facility, for example) - may not associate very high risk values to these vectors.

But what if you are a water plant, sewage treatment facility, auto manufacturer.

What kind of damage can be done if:

• An attacker can sent arbitrary alarm messages
• An attacker can send arbitrary alarm reset messages
• An attacker can flood the wireless device - preventing alarms from sounding

You get the idea.

On PACE, I just read a post: New software for wireless automation control, regarding Conlab’s U.C.ME product:

U.C.ME-OPC interfaces with any SCADA/HMI, OPC or DDE server to provide efficient, rapid and secure two-way communication with SCADA platforms, remote devices and users, via text messaging (SMS) or telephony, says the company.

Now, from what it looks like, the U.C.ME system does not alter settings on any devices - it’s simply a monitoring tool, but what if your command and control decisions are based on the information coming out of U.C.ME?

I have contacted Conlab for more information and will report back.

For now, it’s best to take some precautions:

1. Know that every message sent by your systems over a wireless network will be read by an adversary.
2. Assume that all data received from this system, or by this system may be forged.
3. Know that not hearing from the system doesn’t mean anything is going wrong, or that hearing from the system means there is a problem.

Essentially: If you use wireless technologies of any tye in the control system space, don’t trust the data at all… Not one bit.

Bill

From Electric Light & Power:
FERC proposes to close nuke cyber security gap

In an April 8, 2008 public joint meeting of FERC and the NRC, NRC staff indicated that the agency has proposed regulations to address cyber security at nuclear generating plants, but raised a concern about a potential gap in regulatory coverage because the requirements only will be directly associated with reactor safety security and emergency response, and will not extend to power continuity systems.

I’ve pondered the ramifications of this for a while.

One company, two regulatory bodies.

In the end, I believe that safety and security will be the casualty.

Nuclear power generating facilities in the US are regulated by the United States Nuclear Regulatory Commission.

Over the last year, NRC has gotten on the cyber security bandwagon. They have released a draft regulatory guide that, in theory, will increase the cyber security posture of nuclear power facilities.

In the absence of broad guidance, the industry has been operating under a policy developed by the industry, and approved by the NRC as an acceptable approach for ensuring cyber security.

Both these documents (much to my distress) are withheld from public disclosure.

The NRC’s job is to oversee the nuclear fleet (and other non-military nuclear related industry) and protect the public from potential harmful effects of radiation. After all, nuclear power plants in the US are using a radioactive isotope of Uranium (U235) as a heat source.

Since safety is the NRC’s primary mission, continuity of power concerns may be secondary.

The Federal Energy Regulatory Commission, on the other hand, is tasked with ensuring the safety, security, and continuity of the power grid.

In non-nuclear facilities, there is not an additional regulatory body concerned with the fuel source. The entire scope of operations falls under the regulatory oversight of the FERC.

The gap in regulatory oversight of nuclear generating facilities lies in the area between those set of assets that can be considered “safety related” and those that are considered non-safety related, but are important to continuity of power.

And things get really ugly when certain devices play both roles.

And that’s where this ball of wax gets really out of hand.

NRC is developing one set of standards for securing safety systems, and FERC (via NERC) has developed standards to ensure continuity of power.

Anyone who’s made it past basic algebra knows you can’t maximize two variables in an equation. You can maximize safety, or you can maximize your ability to ensure continuity of power, but not both.

Having grown up in Washington, I think I can smell which way the wind is blowing here.

The result will be that many systems will be under the regulation of two different entities, both with competing interests.

In the end, the operator will have to make a choice, and no matter which he chooses, he fails at achieving the requirements of either body.

In the situation, the best the operator can do is to shoot for the minimum set of configurations that meets all the requirements that match both sets of regulation.

I have to tell you, given that we are talking about a nuclear power reactor, I’d rather the operator not skimp on safety to ensure continuity of power.

What should be done?

It’s my opinion, having read the NRC’s draft regulatory guide, and spent some time looking over the NERC CIP’s that what we really need to do is take a step back.

I believe that continuity of power is a byproduct of the secure operation of any power facility.

I believe as well that safety is a byproduct of the same secure operation.

That said, I believe that what is best will be the development of a set of cyber security policies and procedures that are facility agnostic, and that ensure that digital systems within power generating facilities are secured.

Let the chips fall where they may. Implementing such a plan my require major modifications to the regulatory framework, but if it is what needs to be done, it needs to be done.

I can assure you that crackers do not care if FERC or NRC is running the show.

Bill

Quoting pieces from PC Magazine’s “Electrical Grid Vulnerable to Hackers, House Told”

“The harm could extend not only to the economy and the health and welfare of our citizens, but event to the ability of our military forces to defend us, since many military installations rely on the bulk power system for their electricity,” said Joseph Kelliher, chairman of the Federal Energy Regulatory Commission (FERC).

The Energy Policy Act of 2005 gave FERC the authority to approve reliability standards regarding the nation’s bulk power system. But FERC can only approve standards; it cannot actually craft them. That job falls to the North American Electric Reliability Corporation (NERC), which worked with the industry to develop standards it presented to FERC in August 2006. FERC gave those standards its final approval in January 2008.

That type of timeline is acceptable for most issues relating to the nation’s power system, but when it comes to possible cyber attacks, the government needs to be able to act within hours or days, not three years, Kelliher said.

FERC is limited by the paper shuffling and red tape created by the 2005 legislation. If FERC identifies a problem, it can order NERC to develop a solution within 60 days, but Kelliher said he is not sure NERC “could meet this schedule in practice.”

FERC “does not have sufficient authority to guard against national security threats to reliability of the electric system,” Kelliher said.

He said FERC should also be able to compel utilities to make changes, and the commission’s power should perhaps extend beyond the bulk power system, which at this point does not include Alaska, Hawaii, or local distribution facilities, which include major cities like New York and Washington, D.C.

Ok.

So after NERC’s lambasting in front of congress a few month’s ago, FERC probably feels like it needs to flex it’s muscles a little. You know, show us that it’s relevant in the space.

To me this smells of “bureaucrat seeks to increase his power using fear-mongering and scare tactics.”

The problem with granting FERC these additional powers will be obvious to anyone working in a regulated environment. Currently NERC issues standards to which utilities must comply. FERC is now seeking the ability to issue additional regulatory requirements.

You can’t serve two masters.

FERC is seeking these responsibilities under the guise that they need to respond to “urgent and sudden” threats.

But in reality, if they gain this power, it’s my contention that the situation will be worse, not better. An alternate approach should be considered.

So, what constitutes the kind of threat that would trigger a FERC requirement? Would it be the kind of threat that triggered the TSA to stop allowing passengers to carry any liquids on air-planes?

One core problem I see is that rapid, non-vetted changes in a system increases overall risk.

If this wasn’t the case, then why are our bulk power generation facilities setting up sophisticated configuration management (CM) procedures to ensure that their systems don’t move into an inconsistent state without sufficient vetting?

Additional regulation and mandate will not get us to a true, lasting security.

The regulatory environment is just plain broke in just about every instance that I see. The answer is not more regulation and oversight, the answer is using a different approach.

Achieving confidence in the security posture of the electrical grid requires a few simple things, that as of yet, are not done:

• Define what “a secure grid” looks like. A realistic view can help drive where we want to go. Security for security’s sake is a waste of time, effort, and money. Tell the industry where it needs to go.
• Build smart, responsible regulation that pushes the industry in a measurable way toward that goal.
• Develop tools and resources to facilitate this migration. For example, develop generic contract language that companies can use when making software and hardware purchases that shifts the burden of secure product development onto the appropriate party.
• Develop financial incentives to motivate the change. In the corporate security arena, you see a rapid movement toward data security in the wake of very costly data disclosures. Power operators don’t make money when the lights are out, but you can also incentivize good performance through corporate tax credits for up-time and availability.
• Foster an environment of open communication where all parties can come together to enhance their practices, policies, and procedures.

The writing is on the wall if the bulk power generation industry does not see the light and start moving aggressively toward showing a significant commitment to security.

Other industries should take note.

Bill

On June 11, Core Security Technologies released a remotely exploitable CitecSCADA ODBC vulnerability. (US-CERT VU#476345)

Exploit code is now in the wild.

A patch is available upon request from the vendor.

Bill

A good ‘overview’ article on Energy Central, “Securing the Grid”

This is a little scary, though:

Utilities, meanwhile, are trying to adapt to this changing world. The evidence suggests that while no company has ignored earlier advisories, those entities have varied understandings of what it required of them. In fact, all of the utilities interviewed by government regulators have requested more information as it relates to how they could be attacked and what mitigation efforts are available to stop such breaches.

Couple that paragraph with the fact that most control system people think their situation is unique, and that traditional IT security doesn’t have anything to offer them, and it’s a recipe for disaster.

Bill

According to Wall Street Journal’s MarketWatch, Waterfall Solutions will be presenting at the 2008 PCSF.

In a presentation on Wednesday, August 27th, Mr. Turniansky will give a detailed presentation on the topic of “Unidiresctional Connectivity - A Novel Robust Method for Absolute Protection of Process Control Systems”.

Some are too willing to throw out the notion of unidirectional communication in a control system because it requires some special configuration.

My take is that if you want Real security, you must consider the overwhelming value of a one-way communication link, particularly when critical control systems come into place.

In control networks, there are some devices, like sensors and other detectors, that simply generate traffic.  They do not need to receive anything.

Another possibility is to prevent two way traffic between systems at higher and lower levels of “criticality.”

A no-brainer example would be to employ one way communication between the contol network and the corporate network.  This would allow business units to monitor the control network, but not have data (and attacks) flow from the business side back.

This would have prevented the shutdown of Unit 2 of the Hatch nuclear power plant earlier this year.

From the Washington Post’s “Cyber Incident Blamed for Nuclear Power Plant Shutdown”

A nuclear power plant in Georgia was recently forced into an emergency shutdown for 48 hours after a software update was installed on a single computer.

The incident occurred on March 7 at Unit 2 of the Hatch nuclear power plant near Baxley, Georgia. The trouble started after an engineer from Southern Company, which manages the technology operations for the plant, installed a software update on a computer operating on the plant’s business network.

Though I don’t know all the details, it’s clear that there was some computer on the business network that must have, through some route, been able to feed back to the control network.

Why introduce the potential for a network based attack on one of these devices, simply employ one-way communication.

Method’s I’ve heard about in the past include physically modifying the network cabling so that the cable is physically incapable of sending traffic in both directions.

In any event, I wont’ be at PCSF, but hope the presentation is available.

Bill

This afternoon I tuned-in to a presentation by Tripwire regarding the upcoming release of a NERC CIP policy compliance module that’s due out September 16 for their Tripwire Enterprise product.

I had used Tripwire back in it’s open source days.  Back then it was all host-based integrity checking.  And that was a Long time ago.  I kind of long for those days.  The product was simple and reliable.  Host based integrity checking, IMHO is still a cornerstone of good security, and I have yet to find a suitable small-footprint replacement.

I must admit that I fell out of touch with the product after it went closed-source and they started building a business around it.

Well, the little script that was has turned into a rather mature, end-to-end, device agnostic policy auditing and compliance solution.  Tripwire can audit firewall configurations, router configs, hosts, you name it.

Their Enterprise product is modular, allowing you to install pre-built policy checks for tons of stuff (PCI, CIS, FISMA, COBIT, SOX, ISO 27001, FDCC), or build custom checks.

The purpose of this particular presentation was to learn about a new policy compliance module geared toward evaluating compliance with NERC CIPs.

The North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection (CIP) standards are given, essentially, force of law by the Federal Energy Regulatory Commission (FERC).

I don’t want my reader’s eyes to glaze over, as frequently happens when discussing the myriad of regulatory bodies in the energy space, so I’ll break it down for you:

$1,000,000 per day, per infraction for failure to be in compliance with NERC CIPs.

Let me see if I can explain that in plain English:

$1,000,000 per day, per infraction for failure to be in compliance with NERC CIPs.

The Tripwire policy module focuses on compliance for the technical CIP’s (CIPs 002 to 009, with a focus on 003, 003-6, 005, 007).

Tripwire has a matrix mapping their compliance and auditing checks against the specific NERC CIP requirements, and provides a holistic approach to auditing and assessment.

One nice feature is that where non-compliance is detected, remediation recommendations are presented that can then be attached to a change order so that technicians can implement the recommendations.

The Tripwire people also discussed their products ability to maintain the required auditing and compliance documentation for the minimum required one year.

Concerns

It wouldn’t be one of my posts if I didn’t express some concerns :)

I have two:

1) Does the Tripwire product require an installation on the devices to be audited.

Though I’m not a control system guy, I have heard that some of these devices are extremely ‘fragile’.  Certainly no one wants to install a security module that decreases the availability or integrity of a control system device.

I also get nervous because, in some cases, making any modification to a control device can trigger a rather nightmarish change control process that has serious cost and can have serious regulatory implications.

2) Does the Tripwire system generate a great deal of network load while checks are run.

Again, the worry here is that, from what I’ve read, some control networks are extremely sensitive to latency and load.

Spikes in network traffic or device load can have negative consequences.

Special consideration must be given when considering implementing an automated, scheduled auditing system.

Competitors?

Tripwire is a household name in the IT security space, but are there other solutions out there?

Certainly.

Whereas Tripwire is coming from the IT security space, Digital Bond is a control system security research and consulting outfit that, well, specializes in control system security.

Our friends over at Digital Bond have been developing the Bandolier product.

At the moment, the focus of the product seems to be heavily weighted on the assessment side.  Nevertheless, there is documentation on how to use Bandolier to test for compliance with the NERC CIP’s.

I expect that as the Bandolier product matures, automated CIP compliance reports may be generated from the product.

Conclusions

I first got interested in control system/criticil infrastructure protection when I began hearing reports of what I perceived to be complete failures of security surrounding SCADA and other control systems.

The more I research, though, I see that there is a lot of work being done in this relatively small space.

And I read more and more about vendors coming to the table.

One neat trend I’m seeing… Vendors who build control system hardware are coming to security outfits with their wares asking for help on how to make them more secure.

Now that’s good stuff right there.

Bill

Updated 20080627:

More vendors joining the automated NERC CIP compliance front:
Nexant, Promia to Offer Compliant Cyber Security to Energy Firms