ICS testing best results. Hint: Blend your approach

TL;DR

• Onsite ICS testing is risk averse
• Laboratory ICS device testing uncovers more
• A blended approach is key
• How that works
• Demonstrable benefits

Introduction

For safety’s sake onsite ICS testing adopts a risk averse approach, even if scheduled during downtime or a maintenance period. It’s vital that no systems or devices are altered in any way, or their safety and reliability may fall short of requirements. This cautious approach tends to only find already known vulnerabilities and simple issues like easily guessed passwords. It’s unwise to perform invasive or aggressive testing of components such as routers, RTUs, PLCs, HMIs, and gateways.

Compare that with testing ICS devices in a lab environment. We take an example device, get it onto a bench and really tear into it. The ability to perform risky and even potentially disruptive techniques often uncovers more deeply hidden issues. These could be command injection on web interfaces, manufacturer backdoor accounts, and insecure firmware update mechanisms.

Generally, it is not possible to test all of your ICS devices in a lab environment and nor is it necessary. Many devices will have a low impact if compromised and are well protected by other security countermeasures, whereas others are crucial to keeping your network secure.

For example, if every substation in your network uses the same industrial router to establish connectivity to the upstream supervisory and control systems, the security of these devices is vital. If only a few sites have a PLC that controls a non-safety critical system and is behind several layers of firewalls, it’s unlikely that lab testing is worthwhile.

A better approach

Combining these ICS testing approaches hits the sweet spot, where onsite ICS tests can provide guidance about which devices should be tested in the lab. This provides a far higher level of assurance than purely onsite ICS tests, whilst also being safe and cost-effective.

What does it look like?

• It starts with a design / network review and asset inventory analysis. This identifies common equipment, sites that are representative of the wider estate and any outliers.
• A sample of sites will then be tested using the conventional risk averse methodology. This points us to which devices are crucial for the ICS network’s security.
• Those devices then undergo lab testing to uncover vulnerabilities using a range of techniques. As their position in the wider network is already understood, the risk of these vulnerabilities can be put into context, rather than thinking about just the device.
• Lastly, the newly discovered vulnerabilities are used in further onsite testing to understand the true impact.

Network discovery and mapping

Some of the most important steps in testing your ICS network are asset discovery and mapping the network. It is only possible to discover high impact, high likelihood attack paths once you have these fundamentals.

Asset inventories and design documentation are often incomplete or out-of-date, particularly with the increase in third-party implemented systems in ICS. Whilst endpoint visibility tools and firewalls have helped better understand networks, they still do not have full oversight.

An undocumented cellular router – but with no antennas or SIM card presents little risk:

We’ve often found that previous ICS tests have been carried out fully remotely or sat at a single desk from a single network perspective. This rarely provides a good level of assurance that your network is secure.

Every vessel had a remote access gateway that was unknown to the crew:

Using physical site surveys, documentation reviews and interviews makes sure that any gaps in documentation, understanding or asset inventories are found. This provides a solid foundation for onsite testing and providing context to any vulnerabilities discovered.

These sources of information are incredibly useful:

• A full site survey including examination of the insides of control cabinets and server rooms.
• Documentation review – including the amended ones we find inside control cabinets.
• Interviews with site engineers and technicians to leverage their understanding of how the system is architected and secured.
• Wi-Fi surveys to find any undocumented access points.

Also, these techniques carry no risk of disrupting your systems, unlike network scanning.

Real world example

Previous onsite testing of a series of ICS sites had used a conventional risk-averse, network-only approach from the single provided network port in the site office. Whilst this did find issues, entire network segments – particularly those implemented by third-parties and only accessible via gateways – were not being covered. From the network perspective, these segments may look like a single IP address – but in reality there is an entire control network behind a gateway. Without understanding the network architecture and what sits on each network segment, it is difficult to properly test and find issues.

We proposed that their more complete network discovery and mapping was used, including full site surveys and working more closely with onsite contacts.

After this was carried out, multiple third-party networks were discovered onsite. Whilst several of these were not connected to the central ICS network, some systems were. Crucially, several networks had remote-access cellular gateways.

Whilst these third-party networks were secured from the perspective of the central ICS network, the opposite was not the case. Ultimately, it was discovered that several third-parties could have the ability to remotely connect, without appropriate oversight and controls. Other weaknesses in the implementation of the network meant that there was potential for these remote vectors to have massive impact on the ICS network. Significant changes were made in both process and the implementation of network.

Guidance was created to allow the client’s own onsite staff to survey all their sites for similar gateways. This allowed them to discover how widespread the use of these devices were, without needing a PTP consultant to visit each site.

This identified one specific cellular gateway model was in use at nearly all sites. Lab-based testing was carried out. This uncovered that each gateway communicated with a central messaging platform. The permissions assigned to each gateway found that they could send messages and commands to each other, including reading back the configuration and running commands. Anyone with one of the gateways, even one purchased from the manufacturer, could obtain a set of credentials that could be used to compromise other gateways. The vulnerabilities were reported to the third-party systems integrator and the vendor of the gateway.

Conclusion

By adopting a combined on-site and lab approach for ICS security you can identify the best candidate devices for testing, and do so in a safe and efficient way.