The reality of OT segregation

One of the areas I find most fascinating about industrial control systems and related operational technology is the perception that OT networks are segregated and isolated from the wider IT network.

We’re often told that “IT and OT are totally separated” as there’s a genuine belief that OT is isolated for reasons of security. We have tested very few environments where this is actually the case.

Siloed approaches to IT and OT are still remarkably common: OT engineers have almost completely different challenges and use different sets of network protocols to IT engineers. As a result, a lack of common understanding has become embedded over the years, reinforcing those silos.

However, business needs and additional functionality from OT vendors have resulted in the segregation between OT and IT being broken down or otherwise exposed, accidentally or not.

What’s changing?

So, there is an increase in IT / OT convergence. Ideally that convergence should be driven by tangible business benefits, not just for the sake of it, or because of vendors pushing it as an up-sell. The benefits are an increased visibility of ICS and physical assets, and KPI driven gains in:

• Performance and productivity
• Agility in responding to market fluctuations
• Equipment maintenance – proactive not reactive

It’s interesting to see the value that different companies place on security. The difference between the cyber security budgets of those who’ve never had a  breach compared with those who have tells a clear story.

What challenges still exist?

OT Engineering teams are likely to be small with team members wearing many hats. Safety, availability, reliability, uptime and productivity will likely  always be the key business metrics.

They are unlikely to have specialist and dedicated cyber security knowledge in house though. The human factor is massive. Policies and operating procedures can be written and implemented, but if security is a barrier to the job in hand people will find a way to bypass it.

Even if the official company line is that OT and IT are segregated, there will be personnel onsite who knowingly or unknowingly reveal working practices and examples that begin to blur those perimeter boundaries.

Really isolated?

What do clients mean when they say their networks are segregated and isolated? That’s such an overused and vague phrase and can mean different things to different people, even in the same company / department.

It could mean a simple perimeter boundary defence, between corporate IT and the OT network, but what about internal lateral movement within the OT network?

It could mean that they’re using a well-known segregation reference model e.g. the Purdue model, NIST SP800-82, IEC 62443, ISA-95 / 99, SANS ICS410 reference model.

Consider the segregated zones that exist within the OT network and how they are segmented. There’s the Plant information Zone, Basic Process Control Zone (BPCS) DCS, SCADA, HMI’s etc. and Safety Instrumented Systems (SIS).

Other factors are whether the approach to OT security is reactive or proactive, or whether they rely on segregation to be their main / only form of cyber defence. Lastly, the monitoring / alerting in place.

Physical security

Control rooms and ICS data centres are likely to be well protected by access control, onsite security, CCTV etc. But what about remote and other onsite locations? They’re probably in a locked building or cabinet, possibly covered by CCTV, but it’s hard to know if someone has plugged something in.

To deal with that there are some key things to check and review:

• Are there any anti-tamper alarms / notifications for remote locations?
• Can an intruder tap into any spare network ports in those locations?
• What communication protocols are in use?
• Are they unencrypted – how confident is the asset owner that they cannot be intercepted and tampered with? Modbus, ftp, telnet are still common on legacy sites / devices.

Remote access and control

Access to remote outstations is a key benefit of telemetry. RF, cellular data and other remote connections remove the need to visit to gather data. One can therefore make more efficient decisions and reduce cost.

Over the years we have found numerous security issues in the technology that enables remote access. Examples include:

• Unencrypted and trivially intercepted serial radio comms
• Serial to GSM devices with no authentication controls at all
• M2M devices with critical security flaws that exposed encryption keys remotely
• Devices without local security controls, exposing them to physical compromise

There are many more.

Accidental and intentional breakdown of segregation

We’ve all done it. Changed something to troubleshoot an issue and get a system back up and running, then forgotten to reverse that change. It’s remarkable how often this happens.

All of that careful segregation work is undone through an oversight.

It’s not always accidental though. It’s common for engineers to bypass security controls to make their lives easier. Maybe to remove the need to take a trip down several staircases or similar. It’s also remarkable how many engineers are confident of their security skills yet make basic mistakes. We all make mistakes, that is why it is so important to audit and test afterwards to make sure nothing has been forgotten and no dodgy workarounds are left in place.

Vendor management

Remote management of PLCs and RTUs is appealing – the vendor takes on the responsibility for maintaining devices and has the ability to remotely troubleshoot issue without a costly visit from an engineer.

However, you’re then putting a lot of trust in the hands of your vendors. Do they have the same approach to security as you? How careful are they with remote access, key management, credential management and more? A very thorough review of your suppliers would be wise.

At the very least, a physical switch to enable remote access could be a good idea. Just make sure that it is switched off when the vendor isn’t explicitly supporting your device.

Undisclosed back doors

Some ICS vendors are still stuck in the past in terms of cyber security. The larger vendors are getting on top of it, but the long life of operational technology means that older less secure industrial controllers will be in place for years to come.

The phrase ‘back door’ is emotive, but many vendors will have routes to access a device if it is bricked or otherwise mis-configured and needs to be recovered. That route might be local through a serial connection, or it might be network-based. If that device is unexpectedly exposed, that back door might be a critical flaw.

One device we looked at a while back during a test for a utility contained a back door. The vendor argued strongly that it wasn’t a back door, but you can decide for yourself after looking at their code:

After testing and raising the issue this was swiftly fixed with a notification to all the relevant parties. This is just one important reason to be a part of groups like the NCSC Industrial Control System Community of Interest to ensure you get these updates.

Cloud platforms

Cloud management of OT is being offered by many OT vendors. Unfortunately, cloud security isn’t always as good as it should be. We’re seeing very similar security issues in OT cloud platforms as we see in IoT cloud platforms.

We’ve found critical authorisation flaws in a number of OT cloud management platforms. These allowed remote compromise of every OT device on these cloud platforms, resulting in the ability to take control of industrial processes.

We strongly recommend that you carry out independent audits of any OT vendors cloud platforms that you are considering implementing. It is very important that authentication and authorisation controls are thoroughly tested to ensure that they are sufficiently robust to protect your device security.

Conclusion

Likely points for consideration regarding blurring of segregation:

• Engineering workstations, laptops and USB sticks – cross contamination between IT and OT networks
• Jump boxes
• OT config files and firmware stored off OT network and on corporate network
• Remote access for OT engineers to remote (and not so remote) equipment – convenience for the engineers
• Vendor updates / patching processes
• Physical switches left in ‘convenient’ programming mode – not ‘secure’ run-only mode – to enable engineer programming access on an ad hoc basis
• Whitelists for allowed devices, communications and access not always maintained 100% – if they even exist
• Emergency bypass procedures – for bypass faulty equipment and rapid return to production. Will these bypass any segmentation / security measures?
• Physical controls – are these monitored and followed up? What happens if / when someone plugs a laptop or USB stick into the ICS network? Does anyone even know?

We highly recommend that you think about where the risks are and how they can be effectively mitigated to ensure security while not impacting operations.