Blog: OT, ICS, IIoT, SCADA

Ewon Flexy IoT Router. A Deep dive

Stuart Kennedy 18 Jun 2019

First off I would like to thank the techs at PTP for their insights and help during this process. I know what I know, and I don’t know what I don’t know, so I asked for help sometimes.

I’ve learned a lot from this project e.g. how XOR works, and how to use IDA to analyse ARM binaries better, so I thought I’d share. I’ve listed my fails and wins along the way too, in case you think it was too easy ?

There’s also a shorter TL;DR version available here.

Disclaimer

This was not a paid test or assessment of this device. Responsible disclosure was done at multiple points during this project.

The research started when an actual client assessment came in with one of these devices attached to the product. I got one of those moments that set my Spidey senses tingling, but the device itself was not the scope of the testing.

So, I asked Pen Test Partners to buy me one. They did and I was told to “go and play”!

This whole “Deep dive” is from a black box point of view, assuming nothing about the device or any authentication given.

There’s a table of contents more for my own structuring  than anything else, and you get the added bonus of being able to skip the bits you’re not interested in.

So, sit back, grab a coffee (or beer) and I hope you enjoy this read.

Contents

Ewon IoT 200 Flexy Router

The reach of these devices is incredible, these are the stats that Ewon give:

Impressive.

Here’s what Shodan has to say on the matter:

Ah not bad. Remember, Shodan has plenty of reach, its not 100% perfect, but it will do.

Additionally, Ewon devices looks like they might be manufactured by HMS Industrial Networks.

HMS Networks AB is an international company in the field of industrial communications with the motto “Connecting Devices”. HMS is headquartered in Halmstad, Sweden and is listed on the Nasdaq Nordic stock exchange, employing over 500 people in ten countries and with reported sales of 101 million Euro in 2016.

Source: https://www.hms-networks.com/products-solutions

Right Let’s get into it!

Web Interface

The device turned up and I got mega excited and just straight plugged it in. That’s as far as I got before I had to break out the instruction manual. Also, I couldn’t be bothered to hunt down the IP address it was set to.

By default, the device is set to 10.0.0.53 with a default account of:

Username: adm
Password: adm

…I’ll come back to this later.

Note: I updated to the latest firmware at the time of writing this post. I didn’t want Ewon coming back and saying “but that’s not latest version.. blah blah blah”

WEB – Basic Authentication

I really don’t need to go into this, but yeah, the Ewon web application is using Basic Authentication for its users:

And yes, its communicating over HTTP, not HTTPS. Again, not going into that either as its well known, but this will be helpful later on for us ?

“BASIC AUTHENTICATION IS NOT SECURE AND SHOULD NOT BE USED IN APPLICATIONS”

WEB – XSS

Yes, you already know what’s coming, super elite hax and pure unadulterated pwnage because, remember kids, they are ISO 27K1 compliant….

So, we can steal some creds because the base64 cookie of the Basic authentication is so poor!

But wait, Ewon knew about these already (2015), in fact they knew about a lot more as well. Some issues look like they have been fixed(ish) in updates since this disclosure was released:

Ewon sa Industrial router – Multiple Vulnerabilities : Karn Ganeshen (@juushya)

Date: Thu, 24 Dec 2015 – https://seclists.org/fulldisclosure/2015/Dec/118

Karn published a lot more for the web interface, but I’ve gone in a different direction.

Obtaining the Master Decryption Key

As it turns out, once you get access to the device; either by phishing creds via XSS or if they are left as default (adm:adm) you can pretty much do anything. One thing I didn’t go over was separation of authorisation between the admin account and lower level user accounts.

When viewing the web application or gaining access to the files that store the sensitive data such as VPN Private Certificates and keys or even the user’s passwords, they seemed to be encrypted in a custom way I had not seen before. This intrigued me, so I dug a lot deeper.

The picture below is what an authenticated user would see after the values have been entered and stored in the configs:

Or you can just do a simple post request and grab the Encrypted VPN Private Cert (remember you can get creds ?)

Regardless of this, lets break the stored string down and get into the issue.

We will work with the password string as it’s a lot smaller for now, but the same applies to the bigger VPN certs and keys.

This is the output of the username and stored password for the admin user (my user):

[“ptp”,”#_1_EHyXHCXlKSnkcW2f7kthnIg=”]

Now the eagle eyed of you will instantly see that there is a prefix and a base64 contained value

“#_1_” = Prefix

“EHyXHCXlKSnkcW2f7kthnIg=” Base64 Encoded data.

Unfortunately, the decoded data isn’t of use:

This isn’t great, I want to know what the actual password is.

Yes, I DID already know the password from the XSS/Insecure HTTP comms, but if I can decrypt the data, can I decrypt the VPN Private Key as well?

I started by adding another user to the Ewon Router and setting a password to the minimum length that was accepted “aaa”.

The test user password gets encrypted to “#_1_BXWfyGY=”. We then remove the prefix “BXWfyGY=” and decode the base64 data. It gives us some random looking data “.u.Èf”.

There seems to be some additional characters added at the end of the decoded string, as remember, we only put in 3x“a” as the password. Strange behaviour!

Without fully understanding the encryption method here I decided to modify the password to “aba” to see how it changed. “#_1_BXafyJY=” was the outcome.

There’s clearly some similarity:

“BXWfyGY=” = aaa
“BXafyJY=” = aba

Notice how the first two characters are the same? Does that mean that its encrypted per character?

Let’s change the password again, to “aab”:

“BXWfyGY=”-> “.u.Èf” = “aaa”
“BXafyJY=” -> “.v.È.” = “aba”
“BXWciGc=” -> “.u..g” = “aab”

My encryption game is weak so I went to my colleagues. We all agreed that it was doing the encryption per character and then adding something at the end. One method that was mentioned was XOR.

Let’s fire up CyberChef and get it working for us!

We knew the length of the password “3” and we know what the plaintext variant was “aaa” so we let CyberChef do its thing:

BINGO! Although we still have the two ending chars at the end, we can for now forget those. Now let’s see if we can get it do the full 22 length password.

And yes, we can, this was very labour intensive on CyberChef (Calculating 9.578097130411805e+52 values) so we wrote a python script to do this and left it running:

…so does this key work on the original password for the PTP user too?

Yes, it does. It also does the Private Certificate data too:

We also tried this on numerous Ewon Flexy devices (that we owned), and it worked across the board.

What does this mean? Well, lets recap a little from this point before moving forward?

We can steal credentials from a valid user via XSS, or if we are on the same LAN (for whatever reason), we can also grab encrypted sensitive data and decrypt it (other user’s passwords, VPN keys and configs) etc… We can use the Private Key and password that was stored in the configs to join an existing VPN connection (yes, we configured it and tried, it works)!

We disclosed this to Ewon and got a response much like Karn did. Apparently we needed Authentication for this to be viable! (I would agree with this if there were not multiple XSS and it used HTTPS). Also, none of these devices should be connected to the internet.

Well guess what, A LOT ARE.

WEB – Default Creds

Let’s now go back to the default creds thing. These are well known within the Ewon world and are well publicised. Fine, for initial setup this is ideal BUT, when you enter the wizards, the username and password are pre-populated for you, and guess what they are… “adm:adm”!

Ewon should put some checks here to stop people accepting the defaults and clicking “Next>”

Now, if you look at the reach that Ewon say they have with the products they sell, and the Shodan.io Search from earlier, you make up your own mind on how many have changed from default creds!

I will have a guess at probably not many!

Additionally, to all this, these creds are also used for the FTP service running on the same host ?

As default the Web Portal and ftp is set on Port 80 and 21 respectively, also, there is a secondary port set for the web portal on 81. More on this later.

Hardware

Let’s now jump into a hardware review of the device.

As it’s an IoT device with a lot of usable functionality I was expecting it to be a lot worse that it initially was.

Stripping the Ewon Flexy down enough to get to the board was easy enough. It looks like a single controller for the peripherals/daughter boards and a OS board that is slotted in on the back:

There are slots available for a total of four daughter boards to be attached, and an SD card slot and 4 ETH ports:

On the back there is a slot for the OS board and a number of test pads available:

Things of interest are the NAND Flash on the OS board, the Test pads on the OS board (Top left 2×5 rows of round pads) and the three test pads on the main board on the back next to the cartridge slot (near TP12)

Hardware – JTAG

This is one of my fails.

When seeing a double row of five round test pads I instantly think “JTAG”, but lets be honest, it could be anything It does kind of make sense for an initial test and program of the board. You can see at the top left corner there is a 2-20 slot connector that I suspect can be used to quickly attach to a JTAG programmer and flash the device.

I soldered a wire to every one of the test pads and fired up a JTAGulator.

Please excuse the different coloured wires, it’s all I had to hand at the time

Turns out this is what JTAGulator said the pins were

Getting the ID for the device gave this output:

JTAG > d
TDI not needed to retrieve Device ID.
Enter TDO pin [1]:
Enter TCK pin [4]:
Enter TMS pin [2]:
All other channels set to output HIGH.
Device ID #1: 0000 0001001010000001 00000100001 1 (0x01281043)
-> Manufacturer ID: 0x021
-> Part Number: 0x1281
-> Version: 0x0

After some Googling it looks like a Lattice Semiconductor Device, not a ATMEL.
There’s more info here: http://bsdl.info/details.htm?sid=89730f33d939207188ca59b002289d77

I’m not sure what’s going on there really, but it did halt the rest of the JTAG enumeration and connection:

> halt
halt
Halt timed out, wake up GDB. timed out while waiting for target halted

Hardware – Serial

Next I moved on to another set of test pads that looked interesting. A row of three can either mean Serial or SWD to me, so first thing was to hook up the Saleae Logic Analyser to the three pins and see what was going on.

The three pins were accessible via the bottom of the device and even had a cut out on the plastic housing, this was ideal.

I soldered some wires to the pads and connected them to the Saleae, to see if I could get a read of something that might help me out:

Here’s a zoom-in for clarity:

BOOTLOADER!

I tried not to get too excited at this point, but I was actually doing this!

Now I needed to focus on getting it setup to be interactive. I broke out the USB >serial dongle, threw the wires on and used a generic terminal to connect to the device:

Linux-3.2.11; Hardly up to date. Also looks like we can stop the boot process, hopefully to a shell:

MMC: mci: 0
In: serial
Out: serial
Err: serial
Net: macb0
Hit any key to stop autoboot: 0
Net: enable eth switch
U-Boot> help
U-boot console is locked
U-Boot> ?
U-boot console is locked
U-Boot>

Now what? Let’s mess around and see if I get lucky:

U-Boot> unlock pw
Var pw not def
U-boot console is locked

Oh Hello! This looks a bit more promising.

Unfortunately that was far as I got for a while, we will come back to this part later.

Hardware – NAND Flash Dump

After my intial frustrations I de-soldered the NAND flash and threw it in the DATAMAN reader we have at the office. A few minutes later I had a successful Flash dump. For the sake of clarity I did this twice, in case of any errors with the first, but both were identical.

Now that we have the NAND-dump we can start the dive into getting the firmware extracted and set about analysing the file structure (hopefully).

First, I downloaded this, its awesome and a go-to tool for any NAND dumps.

https://bitbucket.org/jmichel/tools/src/default/

I did some Googling of the NAND FLASH chipset MT29F1G08ABADA. Even though the chip is supposed to handle 1G of storage, this one didn’t and the NAND dump from the reader came out at 138.4 MB.

I then looked up the chipset idcode.

https://www.usbdev.ru/databases/flashlist/flcbm93e98s98p98e/. This took quite a bit of searching to be honest and Dave (@tautology0) at PTP pointed me in the right direction for this!

Now that we’ve got that done, let’s see what Binwalk has to say:

There are a lot of interesting parts in that file. Lets first look at the UBI part. We will use DD to extract that part from the dump.bin file:

Nice! Let see if it’s really what it says it is:

Winning! Let’s keep going:

Excitement mode rising!

Now that we have the firmware successfully extracted it looks like we have both the root filesystem and the Ewon Application file system.

I’m going to go over the Root file system first as I feel that will give us a better understanding of how the Ewon system is built.

In all honesty, there was one file in “init.d” that was of interest and nothing else. All this file did was start the Ewon service. No additional users, no private keys for ssh, no password hashes. Boring.

Let’s get into the Ewon SquashFS.

The SquashFS file structure looks like its specifically for the Ewon environment and gets mounted at /opt/ewon/.

Most of the folders I extracted were empty but the bin/ and patch/ folder had valuable info. The patch/ folder:

These files in the patch/ folder were just for updates from the last firmware update that was run, a few newer binaries, a few newer libs.

Again, most of that folder was quite boring, but it does show that they regularly add linux-arm binaries and libraries in the updates for the device.

After a tip from my colleague Andrew (@cybergibbons) I now always scan folder structures for any executable scripts (bash scripts):

These two files, although a basic bash script, only gave build information really, nothing that helps me right now.

Moving on to the bin/ folder:

A-ha! Now it’s getting interesting. We have a binary, some config scripts and some bash scripts, oh and what looks to be a Java binary and jar files as well. Let’s start with the config files first I think, then get to the Ewon binary last.

The config files are quite simplistic, and as you can see, obviously for a set of different environments. At91sam9g (https://www.microchip.com/design-centers/32-bit-mpus/microprocessors/sam9), QEMU (probably for debug purposes) and RaspberryPi (again, debug and build purposes).

The configs are very similar with most of the values exactly the same, only the name and os dependant stuff has changed.

Here is a sample of the all_config.conf file:

Hmm, what does it say on the second line? “Common part to all product” and lists the Flexy and Cosy Ewon products. Does that mean they are the same product?

There are a lot of similarities, and some massive differences.

“designed to offer easy remote access, across the internet”

Also, Port 443 (HTTPS). How come the Cosy gets https but the Flexy doesn’t?!

I tried to run the Java binary and it errored out, I kind of expected this but it did give me info about a few more files:

Using JD-Gui (http://java-decompiler.github.io/) I decompiled the JAR file it was trying to reach:

I will probably dive into that some more later, but for now lets get stuck into the Ewon binary.

Reverse Engineering the Ewon Binary

As you can see from earlier screenshots there really wasn’t much that we could see in the NAND Flash dump. I was expecting the web interface to be setup using pre-built tools such as apache2, lighthttpd or nginx, and for the FTP using vsftpd or similar. Fortunately for us (hackers) when developers decide to roll their own variant of servers, they miss a lot of security features or protections that other servers have battled against.

When reverse engineering binaries it makes sense to  go straight for the strings.

I did a few basic grep’s of the strings in the binary, “password, pass and private”. We will go through interesting ones later as I don’t want to spoil the rest of the blog.

Throwing the binary into IDA Pro gave fantastic disassembly straight away. It wasn’t perfect, but it was more than usable:

One of the first things I wanted to investigate was the encryption on the stored values (VPN keys, passwords) in the config files (that we broke earlier) ?

After a while searching I came across this function cfgcrypt_DecryptCfgStr.

Here is an IDA layout graph of that function:

Those functions in a little more detail:

With the help from Dave the walking disassembler we were able to work out that the header “#_1_” is dictating what encryption method was used.

Also, in the actual Ewon comcfg.txt file there is a setting “CryptMode:1”. Makes sense to me.

At no point did we see any other variant of the header during this time, however we believe that this is the reason for multiple occurrences of the same key and IV hardcoded in the Ewon binary.

If the CryptMode is set to > 2, then the function will error out and set the value to 0.

I believe if its set to “#_0_” it will use “blowfish” and we have the Key and IV that are used, but in this case, Its set to “#_1_”.

So what does “#_1_” do?

There is a function called “cfgcrypt_CheckCryptmode” that checks the header of the encrypted string and returns a value depending on the header value set, remember, it’s always been seen as “#_1_”.

However, this function is never called in the app. Strange. But there is a similar code base in the “cfgcrypt_DecryptCfgStr” function.

This looks like a fair bit of code reuse across the board.

Essentially all the encrypt and decrypt functions come down to the “ctr_encrypt” function.

And the awesome, “ctr_decrypt” function:

I’m not going to show the specifics, but there is an XOR function contained in the “ctr_encrypt” code.

Firmware Reverse Engineering

One of the things that I was immediately stuck on from the very beginning was the firmware. I could easily download the firmware from the Ewon site:

But then looking at it in some detail it became apparent there was encryption in play:

It’s obvious the file has a header of sorts, due to the version number, release date etc being in clear text, then it moves on to garbage!

So back to the Ewon binary I went, looking for anything that referenced the firmware file name.

The first set of results came up with references to the SDcard:

So, it turns out that if the Firmware is present on the SD card it will try and update. That’s pretty standard to be honest, but also, from the user documentation, you can just upload the “ewonfwr.edf” file to the FTP (if you have authentication) .The Ewon will then process the firmware as necessary.

I hunted around for ages for something that would satisfy my interest in the firmware file, until I stumbled upon a function called “loem_UncryptFile” that looked like it was opening a file and calling decrypt functionality!

Essentially this is what is happening; the Ewon binary detects if a firmware file has been uploaded to the ftp server or put on the SD card, loads the manifest (checks firmware version, date etc.) then decrypts the firmware file and data.

The firmware encryption is based on Blowfish (this is referenced a lot in the binary!) and the binary has the Key hardcoded in ?!

AWESOME!!!

So, back to the firmware file.

After looking at the data, I decided that the encrypted part started at 0x0140h and carried on to the end of the file. I cut the file at these points and tried to decrypt with the given key in the binary:

I was able to get a decent decrypt of the data, but it seemed like it was only decrypting half of it. Really strange, but I was on the right path.

Going back to the binary it seemed like the provided IV was for the binary to create a backup of the firmware before upgrade, so where the hell was this IV that we needed?

I reached out again to the magic channel and explained where I was at. Dave, came back and said, if its not in the binary, it’s probably in the firmware. This apparently is quite common.

Let’s go back to the firmware and look at the header. We are looking for an 8 bit data stream that could be used. After removing the encrypted junk, this is what I was left with, and the only obvious 8bit data stream was the one highlighted at the bottom, followed by some null bytes then the firmware encrypted data:

This was then fed back into the python script to see what happens:

We now have full decryption!

Let’s have a look at the file system and especially that “update.sh” file:

Plenty of files for our Spidey senses! But to be honest, it’s just the Ewon /opt folder as it was from the NAND Flash Dump. So, nothing new there!

The update.sh on the other hand! (again, a lot has been redacted):

NICE! Let’s see what this is doing in the binary, if anything:

I see potential command execution!

I changed the update.sh, remade the firmware, and reuploaded with a higher version number:

Annoyingly this didn’t work either. Looks like there is a CRC on the file as we get a “riftp-Invalid checksum” error in the Ewon logs:

Let’s go back to the binary:

From what I can tell, there is a CRC16x2 check. Now I’m not 100% sure if it’s on the encrypted blob in the firmware, OR, if it would be the squashfs blob pre encryption.

Here is a comparison between two firmware versions. Obviously I made the bits that are common across each file the same to reduce the amount of differences:

The parts in red are the differences in the headers.

Looking a bit further into this I can see some similarities in the red bits between the files:

Let’s run some CRC checksums across the encrypted blog of the firmware file, see if we can see any relevant info in the header!

Ver 13_2s1 = 8.433 mb (Latest)

CRC16 (encrypted blob) = AF3C

CRC32 (encrypted blob) = F388096F

Ver 13_0s0 = 12,589 mb (Not Latest)

CRC16 (encrypted blob) = A65D

CRC32 (encrypted blob) = 259BC946

Wait a moment!

That is the CRC32 value of the encrypted blob! In the header file! At 0x00B0! Lets try changing that then:

Hmm NOPE. Something’s still not right.

So, it’s back to the Ewon binary. When searching through the binary I came across this function that outlined some of the header parts:

“riftp_EDFHeaderPtr” is the start of the firmware file so let’s say “0”.

On line 144, it compares the (rfiftp_EDFHeaderPtr + 168) if its not equal too v10 (a CRC16 x 2 calculation).

At 0xA8 in the firmware, there is the CRC value. Nice! I also mapped out a few other bits that I could from other bits of info in the binary:

One thing that confused me (and others) is the CRC calculation function. We know (or at least think) that it is calculating the CRC16x2 on the encrypted blob from the binary code, but this is how the calculation is done:

Dave and I decided to rewrite the CRC16x2 function to see where we could get to:

Although we are getting different results we were confident that we were very close at this point.

With this we also ran a brute force against the Firmware file, to see if any part matched the value stored in the header:

Unfortunately, this is where this ends.

We tried a ton of different CRC16 Checks, different poly’s, xor bits etc, but nothing came close. We just could not match the value at 0xA8.

A great resource for this was pycrc (https://pycrc.org/models.html) it helped me so much.

Disclosure timeline

2019-01-29 – First contact with Ewon to report vulnerabilities

2019-02-04 – Confirmation of vulernabilities and estimated timeline provided by Ewon

2019-02-11 – Response from PTP that the timeline was too long, Ewon response as well

2019-05-23 – Final chase before disclosure

2019-05-28 – Response from Ewon, test firmware provided

2019-06-04 – Disclosure made by Ewon

2019-06-18 – PTP disclosed, updated FW 13.3s0 now available https://websupport.ewon.biz/support/product/manual-firmware-update/manual-firmware-download

We reported this to HMS Networks, the manufacturer of the Ewon Flexy on 29th January 2019. The disclosure process wasn’t completely pain free, but the end result was that the vulnerability was acknowledged and fixed.

Initial responses to us from HMS were speedy, so a big ‘tick’ for responsiveness there.

We had some positive, constructive interactions with their security manager.

However, it took ~120 days for new firmware to be published to fix the flaw. Not so good.

Then things got a little weird when we stated that we were going to publish this blog. They really didn’t like that – a veiled threat was made by a senior person at HMS.

We pointed out that threats usually result in drawing more attention to vulnerabilities. The ‘Streisand Effect’?

Then things returned to normal and we had a further, really constructive discussion with their product marketing people.

They regained their ‘cool’ in our view when they published this, including credit for the finding:

https://websupport.ewon.biz/support/news/support/ewon-security-vulnerability

Mitigations and Recommendations

The really obvious stuff:

Don’t put your Ewon Flexy (or any ICS device) on the public internet. That’s asking for trouble!

Change the default credentials.

Segment your IT and OT networks with decent access controls, obvs.

Use the free of charge Talk2M security connectivity cloud with your Ewon devices.

From the vendor:

HMS have a secure remote access solution called Talk2M which should mitigate the issue also: https://www.ewon.biz/cloud-services/talk2m

We haven’t tested Talk2M, so can’t comment on its security, but any additional layers of defence are usually a good thing. We’ll be taking a look at the new firmware when I get some time.

New firmware is available which should fix the bug. It should be available here: https://websupport.ewon.biz/support/product/download-firmware/ewon-firmware-0.

From us:

Update your Ewon Flexy firmware and keep it up to date.

Do that with all of your industrial control systems. If you can’t update an ICS (e.g. it’s too old / out of support / too critical to touch) then consider building additional security controls around it.

Conclusion

I hate to leave the blog here and end it like this, but there WILL be a part two, I promise.

This has been one hell of a journey and I have personally learnt so much, my brain actually hurts! I now have a love for IDA, but more and more I’m moving to Ghidra (as its Free and just that little bit better for some stuff)

The Ewon Flexy is a great device and this is not a blog to slam them (too much). It was to see how far I could take my skills and where I could grow and have a focus on security related stuff. This post doesn’t have mega WTFBBQHAX but its my output, regardless of being great or not.

If by any chance I do finish creating my own working firmware, I’m sure there is a ton more to publish another time.

A lot of this has now been disclosed to Ewon.

Thanks for reading and good luck.