Millions of Binaries Later: A Look into Linux Hardening in the Wild

Figure 4

We notice that for almost all distributions, 60% of the packages have at least 10 dependencies. Moreover, we notice that a few packages have large numbers of dependencies (> 100). The same applies to the reverse dependencies of a package: as expected, there are few packages which are used by many packages across the distribution, and therefore vulnerabilities in those select few are of high risk. As an example in the following table, we present the top 20 packages with the most reverse dependencies for SLES, Centos 7, Debian 9, and Ubuntu 18.04 (each cell denotes the package and the number of reverse dependencies).

SLES 12.4

CentOS 7

Debian 9

Ubuntu 18.04

bc (3768)

glibc-0:2.17-260.el7_6.3.x86_64 (7065)

libc6 (23246)

libc6 (21592)

glibc (3667)

glibc-0:2.17-260.el7.i686 (7036)

libstdc++6 (7703)

libstdc++6 (6575)

dconf (1336)

libgcc-0:4.8.5-36.el7.x86_64 (2433)

libgcc1 (7184)

libgcc1 (5683)

at (1118)

libstdc++-0:4.8.5-36.el7.x86_64 (2127)

zlib1g (3416)

libglib2.0-0 (3037)

ed (1053)

bash-0:4.2.46-31.el7.x86_64 (1664)

libglib2.0-0 (3309)

zlib1g (2485)

sed (913)

glib2-0:2.56.1-2.el7.x86_64 (1190)

libgmp10 (2079)

libgmp10 (1926)

file (894)

zlib-0:1.2.7-18.el7.x86_64 (1074)

libx11-6 (1962)

libx11-6 (1576)

rpm (807)

libX11-0:1.6.5-2.el7.x86_64 (638)

libqt5core5a (1751)

libqt5core5a (1486)

pkg-config (746)

cairo-0:1.15.12-3.el7.x86_64 (597)

libcairo2 (1614)

ruby (1315)

coreutils (731)

openssl-libs-1:1.0.2k-16.el7.x86_64 (597)

libgdk-pixbuf2.0-0 (1547)

nodejs (1298)

perl (626)

gdk-pixbuf2-0:2.36.12-3.el7.x86_64 (575)

libpango-1.0-0 (1483)

libqt5gui5 (1123)

gcc (608)

pango-0:1.42.4-1.el7.x86_64 (571)

ruby (1452)

dpkg (1045)

python (542)

atk-0:2.28.1-1.el7.x86_64 (495)

nodejs (1434)

libqt5widgets5 (984)

xli (389)

libxml2-0:2.9.1-6.el7_2.3.x86_64 (484)

libqt5gui5 (1362)

libghc-base-dev-4.9.1.0-d28d6 (898)

tk (375)

perl-4:5.16.3-294.el7_6.x86_64 (463)

libpangocairo-1.0-0 (1295)

libgdk-pixbuf2.0-0 (892)

gd (309)

python-0:2.7.5-76.el7.x86_64 (463)

libatk1.0-0 (1202)

python (874)

bash (290)

freetype-0:2.8-12.el7.x86_64 (417)

libqt5widgets5 (1180)

libgtk2.0-0 (869)

systemd (236)

systemd-0:219-62.el7.x86_64 (401)

libatomic1 (1109)

libgtk-3-0 (795)

info (229)

libgcc-0:4.8.5-36.el7.i686 (393)

libxml2 (1101)

libcairo2 (793)

Table 3

Interestingly, although all the OSes analyzed were built for x86_64 architecture, and the majority of the packages also have a specified architecture of x86_64 and x86, we discovered that the packages often contained binaries for other architectures, as seen in Figure 5.

Figure 5

In the following section we will dive deeper into the characteristics of the binaries analyzed.

Binary Hardening Statistics

At absolute minimum, we wanted to examine a basic set of hardening options for the binaries at hand. Several Linux distributions ship with scripts that perform such hardening checks. For instance, Debian/Ubuntu exposes a hardening-check script that provides such useful information. Here’s an example:

$ hardening-check $(which docker)
/usr/bin/docker:
 Position Independent Executable: yes
 Stack protected: yes
 Fortify Source functions: no, only unprotected functions found!
 Read-only relocations: yes
 Immediate binding: yes

We notice that the above script checks for the following 5 hardening features :

• Position Independent Executable (PIE): Indicates if the text section of the program can be relocated in memory, so as to achieve randomization if ASLR is enabled in the kernel.
• Stack Protected: Whether stack canaries are in place to guard against stack smashing attacks.
• Fortify Source: Whether unsafe functions (e.g., strcpy) are replaced with their safer counterparts and calls that are verifiable at runtime replace their non-runtime-checked counterparts  (e.g., __memcpy_chk vs memcpy).
• Read-only relocations (RELRO): Whether relocation table entries are marked as “read-only” if they were resolved before execution begun.
• Immediate binding: Whether the runtime linker resolves all relocations before starting program execution (this is equivalent to full RELRO).

Are the above hardening mechanisms enough? Unfortunately, not at all. There are known techniques to bypass all the above defenses, but the more hardening defenses are in place, the higher the bar for the attacker. For instance, techniques to bypass RELRO mitigations are hindered if PIE and immediate binding are present. Likewise, full ASLR requires attackers to perform additional work to construct a working exploit. However, for sophisticated attackers, the above mitigations are something to expect — their absence in essence opens the door to your system much faster. Therefore, it is critical that at minimum they are put in place.

For our analysis, we wanted to examine what percentage of the binaries in the distributions under examination had these mitigations in place, as well as the presence of three more mitigations:

• Non executable bit ( NX ), which prevents execution on any region that should not be executable such as the stack heap etc.
• RPATH/RUNPATH , which designates the run-time search path used by the dynamic loader to find the appropriate libraries. The former is a sine qua non for any modern system – its absence allows attackers to arbitrarily write a payload in memory and execute it as is. For the latter, non-sane runtime path configurations are responsible for introducing untrusted code that may lead to series of problems (e.g., privilege escalation as well as other issues ).
• Stack Clash protection, which provides protection against attacks that cause the stack to overlap with other memory regions (e.g., heap). Given the recent exploits abusing stack clash vulnerabilities insystemd, we thought it relevant to include this mitigation in our dataset.

So, without further ado, let us jump into the numbers: Tables 4 and 5 present the summary of our analysis for the executables and libraries of the different distributions, respectively.:

• We notice that with respect to NX enforcement, it is almost omnipresent, with few exceptions. Particularly, we notice slightly lower adoption in Ubuntu and Debian distributions compared to CentOS, RHEL and OpenSUSE.
• When it comes to stack canaries, we notice that they are missing from significant portions of the total executables, especially in distributions with older kernels, however there has been progress in latest Centos, RHEL, Debian and Ubuntu distributions.
• With the exception of Debian and Ubuntu 18.04, PIE support is poor in most distributions.
• Stack clash mitigations are scarce in OpenSUSE, Centos 7 and RHEL 7 and practically non-existent elsewhere.
• All distributions with modern kernels have some support for RELRO, with Ubuntu 18.04 leading the way, and Debian at second place.

As mentioned earlier, the metrics presented in this table are averages across all different versions of a binary. Thus, we observe differences in say, PIE statistics compared to examining only the latest version of a binary (see for instance Debian’s progress with PIE packages). Moreover, whereas most distributions generally examine if at least some functions are fortified in a binary when counting the percentage of fortified binaries, for our analysis we measure a true percentage of the fortified functions. Therefore, if for a binary 5 out of 50 fortifiable functions are fortified, we will assign it a score of 0.1, as 10% of the functions are fortified.

OS

Canaries

Stack

Clash

NX

PIE

RELRO (full)

RELRO (partial)

RUNPATH

FORTIFIED

SLES 12.4

55.34

1.15

99.29

6.33

7.30

89.84

85.62

41.04

CentOS 6.10

55.73

0.00

99.66

2.18

2.96

6.26

98.73

41.11

CentOS 7

84.55

0.15

99.83

5.83

11.80

88.00

98.44

41.51

RHEL 6.10

54.32

0.00

99.43

2.20

2.85

6.87

98.78

37.31

RHEL 7.6

82.26

0.13

99.56

5.57

10.93

88.89

98.45

36.66

Debian 9

73.03

0.01

97.78

38.04

35.34

62.15

88.48

31.10

Ubuntu 14.04

53.85

0.01

96.12

4.04

9.53

85.82

99.08

44.12

Ubuntu 16.04

75.58

0.01

95.84

4.76

12.01

85.09

97.54

43.54

Ubuntu 18.04

85.62

0.01

93.43

37.77

56.81

41.46

88.54

41.00

Table 4: Hardening Properties of Fig. 3 Executables

(Percentage of Adoption)

OS

Canaries

Stack

Clash

NX

RELRO (full)

RELRO (partial)

RUNPATH

FORTIFIED

SLES 12.4

49.46

0.37

99.49

6.41

90.41

85.05

35.29

CentOS 6.10

53.15

0.00

99.91

2.72

5.37

98.66

38.07

CentOS 7

83.12

0.06

99.84

10.49

89.39

99.06

38.10

RHEL 6.10

52.28

0.00

99.91

2.51

5.96

98.71

38.06

RHEL 7.6

82.83

0.07

99.89

9.45

90.47

98.88

38.31

Debian 9

74.46

0.01

96.94

33.17

63.88

89.02

33.00

Ubuntu 14.04

43.37

0.01

94.60

9.40

85.00

98.79

38.89

Ubuntu 16.04

67.37

0.01

93.29

12.23

83.76

96.73

35.95

Ubuntu 18.04

81.96

0.01

89.57

32.01

65.97

88.35

33.81

Table 5:Hardening Properties of Fig. 3 Libraries

(Percentage of Adoption)

So are things progressing? They definitely are: this can be seen from statistics on individual distributions (e.g., Debian ), as well as from the above tables. As a use case, we present a summary of the hardening mechanism adoption for three consecutive Ubuntu LTS distributions in Figure 5 (we omit stack-clash statistics due to the low adoption). We notice that consistently more binaries have stack canaries from version to version, whereas significantly more binaries in 18.04 are shipped with full RELRO compared to previous versions.

Ubuntu Executables Ubuntu Libraries

Figure 6

Unfortunately, multiple executables are present still in the different distributions, having none of the above mitigations. For instance, taking a quick look at Ubuntu 18.04, we notice the ngetty binary (which is a getty replacement) ships without canaries, NX, PIE and fortified source, and so do the mksh and lksh shells, the picolisp interpreter as well as the packages nvidia-cuda-toolkit (which is a popular package for creating GPU-accelerated applications like machine learning frameworks) and klibc-utils. Similarly, the mandos-client binary (which is an administrative tool allowing unattended reboots with encrypted root filesystems), as well as the rsh-redone-client (a re-implementation of rsh and rlogin), ship without NX and have SUID set :(. Likewise, several suid-binaries lack basic protections like stack canaries (e.g., the Xorg.wrap binary of the Xorg package).

Summary & Closing Remarks

In this post, we highlighted several characteristics of modern Linux distributions when it comes to hardening. Our analysis demonstrated that the latest Ubuntu LTS distribution (18.04) has on average the strongest OS-level and application mitigations amongst the distributions with newer kernels that we examined such as Ubuntu 14.04, 12.04 and Debian 9. The examined CentOS, RHEL and OpenSUSE distributions in our dataset, however, ship a tighter set of packages by default, and in their latest versions (for CentOS and RHEL) have higher numbers of stack clash adoption compared to their debian-based rivals (Debian and Ubuntu). Comparing CentOS and RedHat versions, we notice large improvements in deployment of stack canaries and RELRO from versions 6 to 7, however on average more functions are fortified in CentOS than in RHEL. Overall, we notice that the most progress to be made across all distributions is with regards to adoption of PIE, which, with the exception of Debian 9 and Ubuntu 18.04 is below 10% for the binaries of our dataset.

Finally we should note that while custom analysis such the above is possible, there is a plethora of security tools (e.g., Lynis , Tiger , Hubble ) that may be used for analysis of a system and to prevent insecure OS and application configurations. Unfortunately, even with hardening mitigations and sane configurations, vulnerabilities eventually will lead to exploitation. This is why we strongly believe that it is vital to have solid real-time monitoring and prevention of attacks , by focusing and preventing exploitation patterns. We will explore such exploitation patterns in future posts, as well as how one defend against them. Stay tuned!

Notes

1. Static analysis tools examine code properties without executing the software, by only examining the source code or generated binary. Dynamic techniques require execution of a piece of software and perform runtime analysis.

2. W3techs reports that some UNIX variant is used by 69.2% of all websites, and the percentage is even larger when it comes to supercomputers . A multitude of other sources (e.g., 1 , 2 , 3 ) verify this trend, which is especially true for container deployments — even back in 2016, Ubuntu alone was reported to have over 60 million images launched by docker users.

3. For a thorough explanation of the utility of the above settings, the interested reader may refer to the KSPP project or to our kernel configuration glossary . In future posts, we will elaborate more on why many of these came to exist, as well as on possible ways to exploit the system in their absence.

4. We should note that the numbers presented include all binaries across different versions of a software package. For instance, a dynamic module used by Python will be analyzed multiple times, separately for python-2.6, 2.7 etc. Our averages are presented across all packages.

Theofilos Petsios ( Research Scientist )

Theofilos Petsios is a Research Scientist at Capsule8. His interests include systems and application security, binary analysis and privacy. Over the years, he has presented his research at various conferences internationally, including IEEE Security & Privacy, ACM CCS, ACM EuroSys and others.

https://github.com/nettrino

Capsule8 Labs