IoT security testing methodologies

Infrastructure testing, penetration testing, and web application testing have been around for years, and most providers and customers understand how they are carried out.

IoT itself – and IoT security testing –  are somewhat different though, so it’s worth explaining how they differ, what types of testing can be done, what the advantages and disadvantages are, and how you can make the most out of working with us to test your products.

IoT is not as simple as many make out.

Many people think of IoT as three components. A device, a server, and a user.

Nearly all of the published research on IoT vulnerabilities focus on the device. A lot of the training available focuses on attacking the device.

But that is only a tiny part of the picture. If this is all you focus on, you will leave the rest of your system exposed to risk.

When it comes down to it, a real-world IoT system is far more complex. There are few other systems where you are responsible for so much:

• The devices themselves
• The operating system that runs on the devices
• The software on the devices
• The mobile application
• The servers themselves
• The build on the servers
• …

To compound this, the devices could be placed in physically exposed locations and on potentially hostile networks that you have no control over. They are installed by people with no networking knowledge. And you have essentially placed a big part of your system directly in the hands of the attacker.

This is very, very different to normal infrastructure IT.

Black/White/Grey Box Testing

There are three overall methodologies that can be used to test IoT systems, each with their own advantages.

Black box testing

The testers will approach the system as a real-world attacker. The only knowledge they have is what is publicly available. The only difference to an actual attack is that the testers are authorised to attack servers and services.

Often, a large part of this testing will focus on recovering firmware or rooting the device. With this, you get information about how the system operates, including APIs. This can be crucial in finding serious systemic issues.

Black-box testing is often time-boxed rather than task-driven. Although testers will be following procedures, the testing will flow in an organic manner, following paths most likely to yield vulnerabilities.

Advantages

• Lowest customer effort – a scope is agreed, production devices are sent to us, and testing begins.
• Possible with nearly all systems – legal, technical or even internal politics sometimes prevent external parties performing white box testing.
• Simulates the real world – this is how any attacker would approach the system
• For us, reverse engineering is great fun!

Disadvantages

• Does not maximise use of time – spending time on tasks like obtaining restricted access data sheets, determining part numbers from obfuscated parts, and reverse engineering multi-layer boards do not improve your security.
• Potentially low coverage – as time is spent reverse engineering and exploring the system, less time is spent on security testing itself.
• Often ignores defence-in-depth – if the firmware of a device cannot be recovered during the test, but that firmware is full of hidden vulnerabilities, you will not have the multiple layers of protection that are required to maintain a secure system.

White box testing

The testers have access to design documentation, specifications, data sheets, schematics, architectural diagrams, firmware, and even potentially source code.

Using this knowledge, they attack the system.

White box testing can be task driven, as the open access to documentation allows the tester to develop a plan for testing before testing starts.

Advantages

• Maximises use of time – the testers can move directly to security testing.
• Full coverage – everything that is documented can be tested appropriately.
• Defence-in-depth – many layers of the system can be examined, from aspects such as compile time security protections, through to intrusion detection on hosts.

Disadvantages

• Highest customer effort – providing the documentation and other information can be time consuming.
• Not real-world – this can result in remediation efforts being spread too thin, focusing on deeply hidden vulnerabilities that are unlikely to be exploited.
• Hard to get buy-in – many vendors are still unwilling to allow this level of access to their documentation, systems, intellectual property or code.
• Third-parties – if the system being tested involves multiple third-parties, obtaining white-box level of access across all of these can be extremely challenging.

Grey-box testing

This is the middle ground between black and white box testing. Some information, but not all, is provided.

This is the most common type of testing we carry out.

It carries some of the advantages and disadvantages of both black and white box testing.

Our preferred method

At this point in time, our preferred method is:

1. A period of black box testing is performed.
2. If, at the end of this period, we do not have access to the device/firmware, we will ask for it (“break glass access”).
3. Grey-box testing continues.

We have found that this delivers the best results, providing confidence that the device will withstand attack from real-world attackers, allows defence-in-depth practices to be followed, and avoids wasting time with reverse engineering.

Common testing and security misconceptions

Over the many tests we have carried out, we have noticed some common misconceptions around security testing of IoT products.

Will you totally own it? Look what Miller and Valasek did to that Jeep!

There are several differences here:

1. Miller and Valasek had months to take the Jeep hack from vulnerability to exploit. Most security tests last a fraction of this time.
2. Their goal was to take control of the vehicle remotely, ignoring many other viable risks.
3. A conventional security test would have discovered several of the vulnerabilities they exploited, and fixing these would have stopped their attack working.

Whilst we will aim to develop proof-of-concept exploits against the system, it is often not the best use of testing time.

Will you find all the issues?

Security testing will never find all the issues. For example, thousands of web application tests were performed without detecting Heartbleed.

Testing aims to find the most serious vulnerabilities in the time given. The longer spent, the more issues that will be found. White box testing will nearly always find more issues than black box testing.

So if you couldn’t get the firmware, you wouldn’t have found the vulnerabilities?

You should accept that at some point, your firmware will end up in the hands of an attacker. This could be due to failings in the hardware, a version control system being exposed to the Internet mistakenly, or a member of staff losing a USB stick with it on.

Sometimes recovering firmware from a well secured microcontroller can take months of effort, but once an attacker has taken this step, the security barrier is down and cannot be put back.

Placing too much weight on stopping access to the firmware means that your other layers of defence cannot be checked properly during testing.

Should we fix these medium and low severity issues?

Yes.

Many high severity issues are the result of multiple medium and low severity issues chained together.

Fixing the medium and low severity issues can terminate these chains of vulnerabilities, resulting in a far more secure system. This is part of the concept of defence-in-depth.

Why do you need access to documentation, firmware or source code?

Nearly everything can be reverse engineered, given time.

For example:

• Finding access to JTAG can take a long time on a multi-layer board with a BGA processor. We know we can do this; so can other attackers. This is not a strong security mechanism, it is simply time consuming.
• Disassembling and decompiling binaries would likely find the same vulnerabilities as source code inspection, it just takes an order of magnitude longer.
• Brute-forcing a password hash recovered from firmware could take weeks, but once obtained, allows access to every device.

Dropping these barriers gives better results.

Can you provide an hour-by-hour plan?

Certain industries have always been driven by checklists and compliance based testing. Generally, these types of tests can be planned for weeks in advance.

Security testing will always have an element of the unexpected. This means that even white box testing can deviate from plan, focusing attention on the more probably vulnerabilities.

This makes it difficult to provide highly detailed plans ahead of time.