DEF CON 28: 747 Walkthrough from a Hacker’s Perspective

This post is a companion to the DEF CON 28 video available here https://www.youtube.com/watch?v=yq8wgJO-JXY

Airframe tour

Alex: Welcome to this virtual 747-400 walkthrough. One of the advantages of DEF CON Safe Mode this year is that we’re able to bring you things like this. Nothing beats being able to climb onboard and poke around a real airframe. This session will give you some insights into the avionics, and secret spaces, on board this very recently retired aircraft.

Ken and I work for Pen Test Partners in the UK. We’re very fortunate to have had access to this end-of -life airframes at a breaker’s yard. We’ve learned an awful lot and look forward to sharing some of these things with you, especially if you’ve never seen the inside of an avionics bay before.

Alex: Here we walking up the stairs to door B2, which is the second from the front on the starboard side. It gives us a nice view back over the wing, and the observant amongst you will notice that there are some engines missing. There should be four, but three have already been sold as these are expensive and highly sought after items.

Alex: We’re onboard now and going into one of the galley areas. This is where the ground services teams would load trolleys full of drinks , snacks, and meals. Sadly no dining today as the aircraft its pretty much empty of everything except the seats.

Alex: We’re now walking through to the back of the aircraft, past the stairs to the upper deck. The 747 is truly a monster at 71 metres long (230 feet) so there’s plenty of space on this aircraft for a four class cabin layout.

Alex: Down here right at the back is one of the “secret” crew rest areas. Through this door at the back are comfy seats and then a small spiral stair leading up to the crews bunk area.

Alex: Moving towards the forward end now and we’re in first class Towards the back of this section is the floor access hatch down into the electrical engineering or avionics bay. You won’t find this kind of access on smaller aircraft. You can see the open outside hatch at the bottom of ladder.

Alex: In the avionics bay now and his is our main set of avionics, all neatly racked up. Each black box is called a “Line Replaceable Unit” or LRU, and carries out an individual function. Here we have LRUs for cabin heating control, data management, and at the bottom, the three inertial reference units for navigation.

Alex: Looking at the same rack from the other side now we can see big bundles of cabling running from the back of the racks to other areas, primarily to the cockpit which is now two decks above us. Each individual LRU will have discrete cabling up to the switches or displays in the cockpit, and if needed, discrete cabling to any other LRUs it needs to talk to.

On this aircraft most of the cables carry data in the ARINC 429 protocol and this needs a pair of cables, to transmit and receive. There is no network in the traditional sense to connect to. You can’t just clip into a pair of wires at the back of the aircraft and gain access to all of these other LRUs.

Another little “secret” is that behind the LRU rack are doors leading to the forward cargo bay. The cargo and avionics bay are pressurised so in theory you could use this area in flight. It would be pretty cold and noisy so you really wouldn’t want to.

This aircraft now has large tanks of water in here now to balance it. This is because with the engines removed it would be too tail heavy and would fall backwards without the counterbalancing weight.

Alex: Let’s now take a closer look at these LRUs. This unit is for the para visual display, an indicator which shows the pilots if they’re aligned on a runway. They’re held on to a standard sized rack with a screw thread and rubber vibration mount. Some LRUs have more than one locking screw. If we pull these out we find a custom multi pin connector on the back of the LRU for the cable breakout. This then runs off to wherever it needs to go in the cockpit.

Alex: Here are two of the units involved with ACARS, which is the aircraft’s datalink system. The CMU acts like a router sending ACARS traffic between the various radios and output devices, like display units and the printers which we’ll see later.

Alex: Let’s follow those cables back upstairs via the unique set of steps up to the upper deck bubble. There’s a galley up here too which we’ll cover later.

Right at the very front we find the cockpit behind armoured doors.

Alex: Here we are in the cockpit. On the left is the captain’s seat and to the right is the first officer’s. There are two jump seats as well, and behind me is the flight crew rest area (just a bunk bed again). There’s also a separate toilet that they can use without going back into the passenger area. On long flights there will be three pilots (or more) which they rotate so that someone is getting some rest.

The 747-400 was the first 747 with a “glass cockpit”, in that the flight instruments are screens rather than what are termed “steam driven” dials and gauges. There would also have been a flight engineer too, but their role is now computerised for the most part.

Each pilot has their own control column or yoke, which on this aircraft is directly connected to the control surfaces by wires, pulleys and gears.

The centre console is mostly taken up with the multifunction control and display units, throttles, and controls for the radios.

There is a primary flight display right ahead, and a navigational display. Both are independent. In the centre are the EICAS displays (Engine Indicating and Crew Alerting System) which show the status of all the major systems like engines, fuel, electric, hydraulics. They also show a log-style set of messages in yellow or red depending on severity.

Alex: So let’s power her up. We have a single ground power unit available to us, but in working life there would be two. This means that we need to be a bit careful with our power use. By pressing the overhead button to tie in “ground power 1” most things come alive.

Alex: The primary flight display showing the artificial horizon and flight director bars, and the navigational display, light up. Here I’m sat in the first officer’s seat so the PFD is on the right and ND on the left.

Alex: My lower EICAS display shows the electrical busses (one for each engine) but as you can see from the upper display, we have three engines missing. The aircraft will interpret this as a fire as the detection loops aren’t connected.

I can also use my MCDU to access the ACARS system that we saw downstairs in the avionics bay. There are lots of things I can do through ACARS but that’s the topic for another of my DEF CON 28 talks.

Alex: Also in the cockpit are some other interesting bits of avionics. This navigation database loader still uses 3.5” floppy disks. This database has to be updated every 28 days so you can see how much of a chore this must be for an engineer to visit each month.

Alex: There is also a quick access recorder which is used for gathering lots of data about the aircraft’s status and health. At the end of nearly all flights an engineer will remove the PC card (although actually this one has a CF to PC card converter) and download the data. This helps with predictive maintenance or detecting if there’s been a heavy landing or tail strike for example.

More and more aircraft are starting to become e-enabled which means this data is streamed in near real time back to the airlines and engine makers over SATCOM. This is so they can have replacement parts ready and waiting if needed when the aircraft next lands.

Alex: This is the infight entertainment. There’s no Wi-Fi on this 747, but under the stairs down from the upper deck, the cabin services director has a small office and this is where the 747’s IFE is driven from.

There is a small touch screen PC which actually running Windows NT4. This can be used to change aspects of the system, such as add the day’s news broadcast video, reboot seat boxes, and turn individual seat lights on and off.

Alex: Above this IFE’s PC are the digital media servers which contain all the video and audio content, ready to stream to each seat.

Alex: There’s also the boarding music and safety announcement controller which plays off digital audio tape.

Alex: There’s even a little printer here!

Alex: Lastly, remember the upper galley? There’s a lift to take catering carts between the main and upper decks, which is pretty cool!

Discussion

Ken: For us one of the things that is a bit of a game changer is that there are so many airframes being retired right now. What difference does that make for us as researchers?

Alex: Aircraft are really expensive beasts. Airframe manufacturers won’t just let you do an exploratory pen test on an aircraft because you don’t really know what state you’re going to leave it in.

It is not like an office network where you can easily clean up and reset everything  afterwards. Doing that with avionics and whole airframes would be ludicrously expensive. Performing adversarial testing of an aircraft in an airworthy state is not something that we really get to do. The pandemic has changed the landscape though.

One of the changes is that airlines are bringing forward a lot of their scrapping programs, particularly these larger types like the 747 and  380s and they’re ending up in salvage yards. No one really wants whole 747s anymore, but the avionics can have a second lease of life. All of this means that we have a really good opportunity to have a poke around.

Ken: We’ve looked at some airframes in the past, probably the oldest one was a 10 year old A320, but even then it doesn’t represent what’s coming off production line right now does it? For last year’s DEF CON Aerospace Village we bought some LRUs, but even those were upwards of 20 years old so. Do you think this is a game changer, the fact that we’re seeing more recent kit being scrapped or decommissioned?

Alex: For sure. This 747-400 is 23 years old, and remember the design lifecycle of these things is 30 years. There’s a decade or more of design work that goes into an aircraft before it even goes into production, and then it has 20 plus years of service.

This means that what people are designing today will still be in service in 30 years’ time. As we know it’s really difficult to anticipate future security needs and requirements.

What we have access to today is obviously not state of the art. There are huge differences between the 747-400 that we looked at and more modern aircraft like the A380 that are flying at the moment- there are big differences in the way that their avionics are setup and how the networking is done on board.

Ken: This particular plane was still flying two and half months ago, I flew in it last year, so that makes it entirely representative of scrappage programs being accelerated. We went through the avionics bay and checked the install dates of many of the components. There’s a lot of kit in there that is only 5 to 10 years old so there’s plenty that isn’t 23 years out of date.

Alex: That’s right, plenty of it is reusable, the ground proximity warning system, for example. Those LRUs are interchangeable between many aircraft. I’ve seen the same LRU make and model on the 747 as I did an Airbus A320 and they sell for $20,000 – a tiny little box and it’s $20,000. Stuff does come off these things that is scrapped but for mega bucks a researcher could have a poke at them.

Ken: One of the things that surprised me about the avionics bay was the lack of a network as we know one today. There was discrete wiring for point-to-point cabling, which obviously adds a lot of weight, but it also makes it much more difficult to attack from any other point in on the aircraft.

Alex: Yes, that is the way that ARINC 429 networks were designed – a point-to-point bus a bit like serial. It has a twisted pair for transmit and a twisted pair for receive, and often interconnections between LRUs will be one way. A good example is the Flight Management System (FMS) computer that continuously computes where the aircraft is in 3D space. It takes lots of inputs from lots of systems e.g. GPS and navigation beacons and inertial reference units and it amalgamates all of these things to produce what it believes to be the true position of the aircraft. The FMS will then send that navigational positional data onto other systems.

The IFE is another good example. In order to draw the moving map display for passengers, showing where the plane is in the world and how long it is to get to your destination, that map system needs a feed from the flight management system. Interestingly that does mean there is a connection between the FMS computer and the IFE but it is a one-way connection.

There is a transmit pair from the flight management computer to the IFE system only, so it is physically impossible to for any data to go back the other way from the IFE. So although we’re talking about protocols that have no inherent security, there’s no encryption and no signing for example, the inherent physical security means that you can’t jump from the IFE and take control of the aircraft.

Ken: So obviously ARINC 429 carries a lot of heavy wiring as a result of that. I know that the 777 used an inductively coupled bus called ARINC 629, but that was really only used in the 777 itself. More recently we’ve seen a move towards AFDX or ARINC 664. How does that change the risk profile?

Alex: 664 is basically Ethernet with some extra quality of service layers on top to make sure the flight critical things can always talk to each other. There is basically a fiber network on more recent aircraft like the 777-X,  the 787 and A380 and everything plugs into that fiber network. Instead of the LRU boxes that we saw on the 747 there are now two single compute crates, one for redundancy, an A side and a B side computer.

Many things also just run in software, so there is a real time operating system like VXWorks that handles the flight critical stuff, and there tends to be a Linux side for less important tasks, and some are just run as applications on these computational nodes. In this scenario it does mean that it looks a bit more like a traditional network, but one that’s hardened and resilient.

If things aren’t set up correctly e.g. if you don’t get iptables right, then there is potential for traffic to move from one segmented zone to another. That is true on an aircraft, but obviously it requires physical access.

Ken: That’s a good point. When we talk about ARINC 429 physical access is key. It means that the good physical security enjoyed by an airside plane is much better much better than the average office network. Do you see that threat model changing as we start to connect airplanes e.g. things like engine monitoring?

Alex: There is a clear advantage to connecting aircraft to the Internet, not only for predictive maintenance monitoring also for efficiencies, again in maintenance. On the tour we saw that the navigational database on the 747 was updated by floppy disks. One of the issues caused by the pandemic is that there aren’t enough laptops with floppy disks on them to go around and update aircraft that are coming out of storage.

Wouldn’t it actually be easier if we could use a web app to centrally push out, navigation updates to all of our aircraft in one go? Then we’d know what state they’re all in, without having to physically visit them. It makes certain that we’re running the same version on all of our aircraft, and it reduces costs because we don’t have to send someone out with a laptop and spend two or three hours doing it.

Ken: In the past we’ve looked at navigational databases recovered from airplanes. What we noticed was that there wasn’t any code signing or cryptographic validity, particularly with those on floppies. That’s not such a problem when you’re required to physically visit the plane, and there is good physical security in place. But as we move towards the idea of updating remotely over the air do you think that’s a problem?

Alex: Where manufacturers have moved to over-the-air deployment model they have implemented code signing. This has brought its own problems though. Because the code validity certificate is only valid for a month. If you were trying to make updates to an Airbus you would have have to update the certificate revocation list, and the certificate chain, before you could do anything else. They have dealt with this.

With Boeing they have physical data interlocks on the aircraft. This means you can stage software updates for navigation for example, onto the aircraft. However they are held in an area until either a pilot or maintenance engineer physically operates a key switch to select where that software update should be deployed.

The aircraft has to have its weight-on-wheels switch closed for any of this to be possible. That means it must be on the ground. It can’t be flying, and someone has to physically operate a mechanical switch in order to update that particular part.

Ken: Another thing we have in common asides from being security researchers is that we’re both pilots. These security measures give me great comfort knowing there’s a very experienced human analysing all the data and making decisions.

As we both know there can be difficult days in the cockpit, where information overload while flying can almost be too much to deal with. It can actually start to cause a breakdown. It’s a bold pilot who can actually say “right, let’s stop and start from scratch. Where are we? What are we doing? We aviate, we navigate, and then we communicate”. Do you think there are opportunities within the systems we’ve looked at to maybe introduce overload and confusion that might start a cascade of negative events for a pilot?

Alex: Yes, for sure. There can be an over reliance on what computers tell you. You can see from the A320 simulator right behind you, and from the 747-400 we looked at, that everything is “glass cockpit”. You’re relying on flying the computers’ preprogramed route, and often that route is sent to you by the airline over a system called ACARS directly into the aircraft.

You do have to check it, and there is a big button labelled ACCEPT. There could be a  temptation to not crosscheck that information and merely press the button marked ACCEPT, loading that particular set of calculations, or that new route, or that new request from air traffic control.

Ken: Yes, I’ve made some dodgy decisions myself, including landing at the wrong airport, but that’s another discussion. There was one thing I did notice from looking at our particular cockpit display units and then while talking to a 747 pilot last week. They said that they get the flight plan through ACARS and then review it on a primary flight display- to see what’s going on. What was interesting about that? The fact that on our 747-400s CDU the button that was worn the most was the one you press to accept the ACARS.

There’s a lot of connectivity being added to planes, and a particular technology I’m quite interested in is Gatelink. It’s used to upload and download all sorts of things such as passenger manifest information and movies, so that someone physically doesn’t have to put a CD or floppy inside the IFE. What other things are we seeing being connected to planes on the ground?

Alex: Primarily it’s the health and maintenance data of the aircraft that’s the driver here. There’s a big drive from manufacturers to have cloud offerings so that you as an operator can see the status of your aircraft in real time. Some of it is really cool. For example they can see how long it takes an individual valve to open or close. If that particular valve is now taking a fraction of a second longer to function then maybe we need to arrange maintenance to come and change it before it requires more maintenance and down time.

We have to remember that all the time an aircraft is sat on the tarmac it’s losing money for the airline. To keep our fares down we want to maximize how often these things are flying, in a safe way, by making the most of this connectivity.

Ken: A couple of years ago a security researcher claimed to have accessed the thrust management computer from the IFE. Do you think that was that was a real issue, or do you think perhaps the media got a bit excited about it?

Alex: It was thoroughly investigated and knowing the airframe manufacturers they take reports of this nature pretty seriously. Most of them have avionics labs where they have everything in pieces on a bench. They also have a dedicated cyber security team that take reports like this and replay them in the lab. I know it was thoroughly investigated and then debunked. As we spoke about earlier this is just not possible with ARINC 429.

In the work we’ve done we’ve never found any evidence of the possibility of two-way communication between passenger domain systems like IFE and the control domain. DMZ and more than adequate segregation make sure of that.

Ken: Exactly, there’s no way to substantiate those claims from what we’ve seen. Certainly we haven’t found a way to compromise aircraft control from the IFE.

Alex: It is an area that is super interesting to research though. As with Maritime and Automotive these systems are taking legacy protocols (e.g. ARINC 429 in Aviation, CAN in Automotive) and are connecting them to the internet. They are fundamentally insecure-by-design with protocols like ARINC 429 predating the dating the internet being designed in 1978.

It also predates any notion of public key cryptography. So where we have taken legacy protocols and are connecting these devices to the Internet then we need to keep that in mind. We need to bake defence in when we were doing so.

Ken: So Alex, where are you going next with your research? What are your hunches? Where are your Spidey sense is telling you to go and look next?

Alex: IFE, because it’s semi accessible from the cabin. I think lots of people are interested in it. Also the middle ground, the Information Services on board the aircraft. These aren’t flight critical but if they could be turned off in flight they could create expensive inefficiencies.

Also things like wireless quick access recorders, ACARS, SATCOM, Gatelink, all of these are really interesting as they sit in this middle ground “niche” area in the ecosystem. If we look at the wireless quick access recorder for example or digital flight data acquisition unit, these have access to a lot of other things on the aircraft.

Ken: Sometimes we look in the wrong places though. Everyone gets excited about airplanes because they use an unusual technologies, and that’s great for us inquisitive researchers. But the most likely area for causing disruption is going to be the ground system. If a plane can’t be dispatched because the ground systems are down, or the maintenance systems are down, it simply doesn’t go anywhere.

I think that planes are really interesting and lots of fun and it’s great that we can help ensure their security, but really you need to watch out for those ground systems, right?

Alex: Yes definitely. It reminds me of flying out to DEF CON 27 almost a year ago. The airline we were traveling with to Vegas had a massive outage of its passenger information system. None of the check-in crew at the airports could scan your boarding pass and get you onto the aircraft. They were having to do it with pen and paper, all by hand for thousands of people. At one airport the queues were out of the terminal and around the car park and this was global.

That’s a huge amount of disruption and cost. They did a really stellar job of getting us onto the aircraft in time but you can see just how little things like this can have a real snowball effect.

Ken: That’s interesting, isn’t it? Whilst that was inefficient and slow, the fact there are humans in the loop is important. That’s particularly so on a plane, where you know there are two or more pilots on an aircraft. Even if the systems are doing weird things, there’s still someone there to take over and make sensible decisions.

Alex: Related to that, pilots pretty much always have iPads with them for performance and flight planning, they could also easily revert back to paper charts if they had to.

Of course they should always be double checking what is being sent to them as a route, or changes that air traffic is sending them over ACARS for example. But if everything doesn’t quite match up they can take control of the aircraft with the control sticks and yolks and fly it by hand.

They don’t have to rely on the bells and whistles that we can see behind you on the simulator. Actually, all they need is a couple or 3 gauges as they are trained to fly an aircraft in difficult circumstances.

Ken: Before the lockdown I used to fly pretty much every week. I feel very comfortable being a passenger knowing how aviation systems work, I’m confident in their security. How do you feel about it?

Alex: It’s important that you know we’re not here to scaremonger. Like any industry there are areas of improvement and I’m pretty sure they would be the first to admit that. But do any of these deficiencies mean that it’s unsafe for people to fly? No.

I’m a pilot, I want to carry on flying for work. I have friends and family flying so I don’t wish them any harm. No, I don’t think there are any risks.