Bill’s Security Blog

IMHO, end-user ’smart meter’ device security is the smallest issue to be resolved with moving toward a Smart Grid.

The real issue with Smart Grid is having thousands of electric devices connected to the network that have intermittent production capacities.

Currently, from what I understand, the electrical grid is somewhat sensitive to large changes in the amount of electricity being put onto or being pulled down from the grid.

As energy consumption goes up, power producers put more energy on the line.  As consumption goes down, those sources are throttled back.

But what do you do when you plug 1500 windmills into the grid?

What happens when the wind starts blowing, then suddenly stops.

Here’s a hypothetical.  Suppose a 1500 turbine wind farm, producing 15 MW of power and placing it on the grid.

If consumption = supply, we are all good.

But what happens if the wind, literally, stops blowing for 30 minutes.

Is everyone going to scramble to shut off their air conditioners and unplug their fridge?

With respect to Availability aspects of the CIA triad, we have a big issue here.

Compound that concern with the fact that electrons traveling across a grid monitoring system travel at the same speed as the electrons traveling from producers to consumers, and you get an ugly producer/cosumer problem.

An effective smart grid must mitigate these intermittent sources of power by ensuring that access to the grid happens in a controlled manner.

That involves rapid ability to disconnect an intermittent source, or to store it’s electricity for later consumption.

The devices that perform that function are, in my opinion, the biggest cyber risk.

Though it’s not necessarily security related, the other issue that needs to be addressed with the introduction of large-scale intermittent power sources on the grid, is the need to match all intermittent sources with 100% non-intermittent sources.

For example.  In our example above, in the case where  you have 15MW wind being put on the grid, you must have at least 15MW stand-by power producing capacity spinning and ready to dump energy onto the grid in the event that the wind dies down.

This issue is tough, and involves, essentially, rebuilding vast amounts of the grid to attempt to decentralize the alternate energy sources as broadly as possible.

So, for example, wind turbines in Maine might be providing power to consumers in Arizona during the night, but Solar Production in Arizona might be powering Air Conditioners in DC during the mid-day.  At the moment, our grid just ain’t built that way.

I was at a FERC Commissioner’s meeting last week, and I assure you, they weren’t talking about the issues with end-point monitors.  How you secure the devices protecting the grid as a whole was on everyone’s lips.

Bill Gross

Good article on the security of SCADA and control networks.

Some good background and some insights into how people might go about hacking these networks.  Social engineering almost always seems to work…

I know a lot of IT Security people don’t know a ton about SCADA or control networks, but they sure have an easy time hacking them.  I see a growing convergence between Control System people and IT people - hopefully they’ll talk about security.

Cyber watch

This is common theme for me.

Perhaps I should state my feelings clearly: “If you depend on wireless in a control system environment, you are friggin’ crazy!”

A few weeks ago, I wrote about a SCADA tool from Conlab called U.C.ME.

This tool touts two-way SMS management of your infrastructure! Sweet, eh?

Now, I wrote Conlab but haven’t heard anything meaningful back. Err… Well, I did. I got a quippy email… “You said you emailed us, but you never did.”

Well, I have tried again, and to no avail.

Looking at the product’s material, they seem to be referencing a company ControlSee.  I’ll contact them.  Perhaps Conlab is just a reseller?  Who knows.

I wrote Conlab, who I thought was the product creator, to ask them what kind of authentication they are doing, and how they are preventing simple things like replay attacks.

Crickets.

Google alerts just tossed me this:
“Imagine you could speak to your SCADA system… Send a text message to your SCADA system.”

More talk of controlling your infrastructure through remote access.

Zero talk of configuration and control of access.

Conlab, ControlSee, whoever, for the love of god and all that is holy, if you have any information on this, please send it to me. This kind of stuff keeps me up at night.

Bill

I have been spending some time lately researching the growing trend of using wireless and other remote connectivity services in SCADA networks and other control system applications.

Being an Infosec guy, this is hugely concerning.

Subversion of wireless and other remote technologies is very easy relative to a wired network at a facility.

SImple examples of the types of issues we can expect:

• Replay attacks
• DOS attacks
• Unauthorized disclosure
• Additional attack vectors

Some control systems (running a candy making facility, for example) - may not associate very high risk values to these vectors.

But what if you are a water plant, sewage treatment facility, auto manufacturer.

What kind of damage can be done if:

• An attacker can sent arbitrary alarm messages
• An attacker can send arbitrary alarm reset messages
• An attacker can flood the wireless device - preventing alarms from sounding

You get the idea.

On PACE, I just read a post: New software for wireless automation control, regarding Conlab’s U.C.ME product:

U.C.ME-OPC interfaces with any SCADA/HMI, OPC or DDE server to provide efficient, rapid and secure two-way communication with SCADA platforms, remote devices and users, via text messaging (SMS) or telephony, says the company.

Now, from what it looks like, the U.C.ME system does not alter settings on any devices - it’s simply a monitoring tool, but what if your command and control decisions are based on the information coming out of U.C.ME?

I have contacted Conlab for more information and will report back.

For now, it’s best to take some precautions:

1. Know that every message sent by your systems over a wireless network will be read by an adversary.
2. Assume that all data received from this system, or by this system may be forged.
3. Know that not hearing from the system doesn’t mean anything is going wrong, or that hearing from the system means there is a problem.

Essentially: If you use wireless technologies of any tye in the control system space, don’t trust the data at all… Not one bit.

Bill

From the Register:
Citect yanks ‘misleading’ SCADA bug advisory

Citect, a designer of software used by manufacturing plants and other industrial facilities, has removed an advisory that played down a vulnerability in one of its popular pieces of software.

Citect’s move followed last week’s release of proof-of-concept code that exploited a vulnerability in CitectSCADA, which is used to manage industrial control mechanisms known as SCADA (Supervisory Control And Data Acquisition) systems. The bug meant systems that relied on the software could potentially be exposed to tampering by disgruntled employees or terrorists.

Traditional IT software vendors have been learning this lesson for a long time.

Apple and Microsoft are still slow to the table.

But it looks like the tide is rising.

Vendors of SCADA system software are starting to feel the heat.

Many of these companies are used to writing systems that will be in place for decades. And from what I’ve read over the past few months, most of these companies use security modeling from decades ago.

I’m an advocate of two things:
1) A central set of cyber security requirements for all power related critical infrastructure.
2) US Government putting it’s money where it’s mouth should be.

The goal of the above two goals is to push security requirements into the front end of the procurement process. This pushes security needs back onto the vendors who are in the best place to ensure secure products.

Ask me about the secure composition problem later. For now, I’d settle for just one secure product…

PS - this article reminds me of the Helm’s Deep battle scene from The Two Towers, for those of you into LOTR (start at 8:40). You can say you are secure, but what say you when your defenses are obliterated?

Bill

On June 11, Core Security Technologies released a remotely exploitable CitecSCADA ODBC vulnerability. (US-CERT VU#476345)

Exploit code is now in the wild.

A patch is available upon request from the vendor.

Bill

According to Wall Street Journal’s MarketWatch, Waterfall Solutions will be presenting at the 2008 PCSF.

In a presentation on Wednesday, August 27th, Mr. Turniansky will give a detailed presentation on the topic of “Unidiresctional Connectivity - A Novel Robust Method for Absolute Protection of Process Control Systems”.

Some are too willing to throw out the notion of unidirectional communication in a control system because it requires some special configuration.

My take is that if you want Real security, you must consider the overwhelming value of a one-way communication link, particularly when critical control systems come into place.

In control networks, there are some devices, like sensors and other detectors, that simply generate traffic.  They do not need to receive anything.

Another possibility is to prevent two way traffic between systems at higher and lower levels of “criticality.”

A no-brainer example would be to employ one way communication between the contol network and the corporate network.  This would allow business units to monitor the control network, but not have data (and attacks) flow from the business side back.

This would have prevented the shutdown of Unit 2 of the Hatch nuclear power plant earlier this year.

From the Washington Post’s “Cyber Incident Blamed for Nuclear Power Plant Shutdown”

A nuclear power plant in Georgia was recently forced into an emergency shutdown for 48 hours after a software update was installed on a single computer.

The incident occurred on March 7 at Unit 2 of the Hatch nuclear power plant near Baxley, Georgia. The trouble started after an engineer from Southern Company, which manages the technology operations for the plant, installed a software update on a computer operating on the plant’s business network.

Though I don’t know all the details, it’s clear that there was some computer on the business network that must have, through some route, been able to feed back to the control network.

Why introduce the potential for a network based attack on one of these devices, simply employ one-way communication.

Method’s I’ve heard about in the past include physically modifying the network cabling so that the cable is physically incapable of sending traffic in both directions.

In any event, I wont’ be at PCSF, but hope the presentation is available.

Bill

This afternoon I tuned-in to a presentation by Tripwire regarding the upcoming release of a NERC CIP policy compliance module that’s due out September 16 for their Tripwire Enterprise product.

I had used Tripwire back in it’s open source days.  Back then it was all host-based integrity checking.  And that was a Long time ago.  I kind of long for those days.  The product was simple and reliable.  Host based integrity checking, IMHO is still a cornerstone of good security, and I have yet to find a suitable small-footprint replacement.

I must admit that I fell out of touch with the product after it went closed-source and they started building a business around it.

Well, the little script that was has turned into a rather mature, end-to-end, device agnostic policy auditing and compliance solution.  Tripwire can audit firewall configurations, router configs, hosts, you name it.

Their Enterprise product is modular, allowing you to install pre-built policy checks for tons of stuff (PCI, CIS, FISMA, COBIT, SOX, ISO 27001, FDCC), or build custom checks.

The purpose of this particular presentation was to learn about a new policy compliance module geared toward evaluating compliance with NERC CIPs.

The North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection (CIP) standards are given, essentially, force of law by the Federal Energy Regulatory Commission (FERC).

I don’t want my reader’s eyes to glaze over, as frequently happens when discussing the myriad of regulatory bodies in the energy space, so I’ll break it down for you:

$1,000,000 per day, per infraction for failure to be in compliance with NERC CIPs.

Let me see if I can explain that in plain English:

$1,000,000 per day, per infraction for failure to be in compliance with NERC CIPs.

The Tripwire policy module focuses on compliance for the technical CIP’s (CIPs 002 to 009, with a focus on 003, 003-6, 005, 007).

Tripwire has a matrix mapping their compliance and auditing checks against the specific NERC CIP requirements, and provides a holistic approach to auditing and assessment.

One nice feature is that where non-compliance is detected, remediation recommendations are presented that can then be attached to a change order so that technicians can implement the recommendations.

The Tripwire people also discussed their products ability to maintain the required auditing and compliance documentation for the minimum required one year.

Concerns

It wouldn’t be one of my posts if I didn’t express some concerns :)

I have two:

1) Does the Tripwire product require an installation on the devices to be audited.

Though I’m not a control system guy, I have heard that some of these devices are extremely ‘fragile’.  Certainly no one wants to install a security module that decreases the availability or integrity of a control system device.

I also get nervous because, in some cases, making any modification to a control device can trigger a rather nightmarish change control process that has serious cost and can have serious regulatory implications.

2) Does the Tripwire system generate a great deal of network load while checks are run.

Again, the worry here is that, from what I’ve read, some control networks are extremely sensitive to latency and load.

Spikes in network traffic or device load can have negative consequences.

Special consideration must be given when considering implementing an automated, scheduled auditing system.

Competitors?

Tripwire is a household name in the IT security space, but are there other solutions out there?

Certainly.

Whereas Tripwire is coming from the IT security space, Digital Bond is a control system security research and consulting outfit that, well, specializes in control system security.

Our friends over at Digital Bond have been developing the Bandolier product.

At the moment, the focus of the product seems to be heavily weighted on the assessment side.  Nevertheless, there is documentation on how to use Bandolier to test for compliance with the NERC CIP’s.

I expect that as the Bandolier product matures, automated CIP compliance reports may be generated from the product.

Conclusions

I first got interested in control system/criticil infrastructure protection when I began hearing reports of what I perceived to be complete failures of security surrounding SCADA and other control systems.

The more I research, though, I see that there is a lot of work being done in this relatively small space.

And I read more and more about vendors coming to the table.

One neat trend I’m seeing… Vendors who build control system hardware are coming to security outfits with their wares asking for help on how to make them more secure.

Now that’s good stuff right there.

Bill

Updated 20080627:

More vendors joining the automated NERC CIP compliance front:
Nexant, Promia to Offer Compliant Cyber Security to Energy Firms

Processingtalk.com posted an article describing new passive monitoring module for the Tofino security product.

Sounds pretty neat.

When it discovers a new device, it prompts the system administrator to either accept its deductions and insert the new device into the network inventory diagram, or flag the device as a potential intruder.
…
It also guides the user through creating appropriate firewall rules to allow or block messages, based on what it has learned about the network traffic.

Technical complexities such as IP addressing and TCP/UDP port numbers are managed behind the scenes, making the normally byzantine art of firewall configuration easy for the controls professional.

I guess there’s been a history of typical IT security tools wreaking havoc on control systems:

In 2005, Sandia National Laboratories released a report describing a number of serious events from use of these tools, including this example: “A ping sweep was being performed to identify all hosts that were attached to the network, for inventory purposes, and it caused a system controlling the creation of integrated circuits in the fabrication plant to hang.

The outcome was the destruction of USD50K worth of wafers”.

A concern I’d have about a product like this is the need to assume that all the systems on the network are trusted at the time you are configuring the rule set.

Another is that caution must be used when such a device is operating in the presence of safety systems.

This system has the capacity to block communication, and in a safety system, that could be hazardous.

But all things considered - much of the control system infrastructure seems to be “tough to secure” without unacceptably high cost.

Bolt-on security is rarely effective, but if the system offsets sufficient risk, they may provide the needed security.

It’s also nice to see a product that doesn’t require the user to have in-depth knowledge of protocols and firewall configuration.

If the control systems people know the devices on their networks, what they do, and which devices should be communicating to which other devices, the Tofino product may be a big help.

If that’s not the case, then the product may be of little value, and simply help provide a false sense of security.

It would be good if Tofino creator Byres Security offered some kind of auditing process to verify that users are implementing the system correctly.

Bill

ControlGlobal recently posted an article, “Advances Needed in Control System Cyber Security” discussing the Control System Security Program (CSSP).

Quoting:

CSSP has produced some significant risk reduction products, including a cyber security self-assessment tool, a detailed cyber security procurement language for control systems, a pocket guide to securing SCADA and control systems, and a set of recommended practices. CSSP has also set up a group within US-CERT to produce control-systems- related vulnerability notices, and CSSP teaches control systems security awareness and mitigation training classes. All of this information is available at http://www.US-CERT.gov/control_systems.

One of the most important initiatives CSSP has undertaken, Hoffman revealed, is the technology assessments they do under contract, and with nondisclosure agreements, for control systems vendors. “Basically, we get the hardware and the control systems engineers from the vendor, and we build a system and get it ready. Then our “Red Team”—that’s the attackers—get six weeks to invade and take control of the system.

“In four years of doing this,” Hoffman said, “we have never been stopped from gaining full operational control over the control systems.”

Now that’s what I’m talking about.

If industry can engage this kind of effort, I think they will be able to avoid ugly regulatory requirements that, IMHO, will generally be reactionary and low value.

From the looks of the CSSP, they are doing some really good things in control system security from a standards development and best-practices establishment perspective.

I’m an IT security guy, not a control systems security guy.

I hear a lot of complaining from the control system side that the IT guys don’t understand the complexities of control systems.

I’ll agree that there’s a lot I don’t know about control systems, but I also believe there’s a lot that the control system community doesn’t know about IT security.

But we can both learn from each other.

One aspect I think the IT security can bring is that, in general, IT security systems are constantly under attack, and we’ve had to embrace certain principles in order to achieve a degree of success.

Good IT security embodies the principles of:

• Secure by design
• Secure by default
• Defense in depth
• Constant monitoring and improvement the security posture

Though the same “technical security controls” I use in a corporate network may not work in the control system space, the fundamental principles can drive a good security practice.

Bill