25 septiembre 2017

This article is part of a series of blog posts. If you haven’t already, we recommend that you read the introduction article prior to this one.

During our research, we ended up wanting to analyse a large number of Cisco ASA firmware files. Most importantly, we needed to mine exploit targets for both CVE-2016-1287 and CVE-2016-6366 and to apply patches or changes to specific versions prior to booting. We did most of our analysis on a collection of approximately 170 Cisco firmware files ranging from 8.0.x to 9.7.x. Note that some similar automated target analysis and retrieval from lina was done by RiskSense [1] in their work related to adding 9.x targets to the Equation Group EXTRABACON exploit.

This post explains the general approach of manually extracting and repacking firmware files. It also describes a small collection of Python and bash scripts we are releasing to help automate some of this process and ease the handling of a large corpus of firmware files. We refer to these scripts as asafw. If you want to have a summary of these tools, you can go to the end of this article.

Collecting firmware

Cisco allows you to download firmware files from their official website [2] if you have bought a router with a specific license. For instance, you can get 32-bit firmware for an ASA5505 [3], 64-bit firmware for an ASA 5512-X [4] or 64-bit firmware for ASA GNS3 [5]. You will get files named similar to:

asa841-k8.bin
asa924-k8.bin
asa924-smp-k8.bin
asa961-smp-k8.bin
asav961.qcow2
asav981.qcow2

You can refer to introductory article of our blog series for an idea as to what the naming conventions of these firmware files indicates.

Note that many firmware files can be found on your favorite search engine with some smart pattern matching. You might have to dig through a few different results to flesh out your collection, though. If you do so, you should check the MD5 against Cisco’s website to avoid any backdoor or hilarity. 🙂

Manual firmware extraction

The first step is to extract interesting files from the firmware. The asa*.bin files are a proprietary format and we don’t need to understand it all for now. We use the usual firmware analysis tool binwalk as detailed by Exodus Intel [6], Silent Signal [7] and the RiskSense article noted earlier. binwalk works by recognizing file formats inside the binary firmware a.k.a. file carving.

Note that unlike the .bin files, the qcow2 format is documented as part of the QEMU project and it can be mounted on Linux using the qemu-nbd tool. Inside you will find a .bin file and the following extraction applies:

$ binwalk -e asa924-k8.bin

DECIMAL       HEXADECIMAL       DESCRIPTION
--------------------------------------------------------------------------------
75000         0x124F8           SHA256 hash constants, little endian
144510        0x2347E           gzip compressed data, maximum compression, from Unix, last modified: 2015-07-15 04:53:23
1501296       0x16E870          gzip compressed data, has original file name: "rootfs.img", from Unix, last modified: 2015-07-15 05:19:52
27168620      0x19E8F6C         MySQL ISAM index file Version 4
28192154      0x1AE2D9A         Zip archive data, at least v2.0 to extract, name: com/cisco/webvpn/csvrjavaloader64.dll
28773362      0x1B70BF2         Zip archive data, at least v2.0 to extract, name: AliasHandlerWrapper-win64.dll

You’ll have different names depending on the firmware you extract, but the important files here are 2347E and rootfs.img.

$ file _asa924-k8.bin.extracted/*
_asa924-k8.bin.extracted/1AE2D9A.zip: Zip archive data, at least v2.0 to extract
_asa924-k8.bin.extracted/2347E:       ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, BuildID[sha1]=9a30de919df0b52db3a0d02bf4dbe7260281c6c7, stripped
_asa924-k8.bin.extracted/com:         directory 
_asa924-k8.bin.extracted/rootfs.img:  ASCII cpio archive (SVR4 with no CRC)

2347E is vmlinux i.e. the non-compressed Linux kernel executable.

strings _asa924-k8.bin.extracted/2347E | grep linux | wc -l
164

rootfs.img is initrd i.e. the initial root file system mounted at boot. Note that initrd contains the whole filesystem. It can be extracted using the cpio utility:

_asa924-k8.bin.extracted$ mkdir rootfs
_asa924-k8.bin.extracted$ cd rootfs
_asa924-k8.bin.extracted/rootfs$ cpio -id < ../rootfs.img
_asa924-k8.bin.extracted/rootfs$ ls
asa bin boot config dev etc home init lib lib64 linuxrc mnt opt proc root sbin share sys tmp usr var

We see the folder hierarchy of a regular Linux installation except with the addition of an asa/ folder made by Cisco.

_asa924-k8.bin.extracted/rootfs$ du -sh asa
44M asa

The interesting files from a target-hunting and analysis perspective are:

/asa/bin/lina: ELF executable containing all of the ASA features
/asa/bin/lina_monitor: ELF executable responsible for loading lina and monitoring it
/asa/scripts/rcS: bash script responsible for building the commands to start lina_monitor and hence lina
/lib/libc.so.6: symlink to the glibc version used

The lina binary is quite large (42MB for this particular firmware):

_asa924-k8.bin.extracted/rootfs$ ls -lh asa/bin/lina
-rwxr-xr-x 1 user user 42M May 25 10:30 asa/bin/lina

By looking at the strings in the lina ELF, we can see it supports most of the ASA features:

_asa924-k8.bin.extracted/rootfs$ strings asa/bin/lina|grep -i webvpn|wc -l
3081
_asa924-k8.bin.extracted/rootfs$ strings asa/bin/lina|grep -i anyconnect|wc -l
446
_asa924-k8.bin.extracted/rootfs$ strings asa/bin/lina|grep -i ike|wc -l
1948

As detailed by the Breaking Bricks paper and Exodus Intelligence, rcS contains commented lines that allow you to ask lina_monitor to attach gdb to lina when starting it.

# Use -g to have system await gdb connect during boot.
#echo "/asa/bin/lina_monitor -l -g -d" >> /tmp/run_cmd
# Use -s to specify a serial device other than the default /dev/ttyS1
#echo "/asa/bin/lina_monitor -l -g -s /dev/ttyUSB0 -d" >> /tmp/run_cmd

lina_monitor‘s usage gives hints about how to attach gdb:

-a : Number of Lina instances to run
-l : Start Lina in pthread mode (not process)
-m : SM or AM (*SM Default for 7.3*) 
-g : start ASA in gdb via serial device 
-n : gdb ethernet device, eg, 'eth0' 
-s : gdb serial device '/dev/ttySx' 
-d : generate debug messages 
-c : control C will exit. 
-h : help

We need to specify -g to enable gdb and -s /dev/ttyS0 (note the difference with /dev/ttyUSB0) to ask gdbserver to listen on the serial line.

qcow2 format

The qcow2 [8] format is used by QEMU and is natively supported by Linux. We first need to load the Linux nbd driver. Then we mount it using the qemu-nbd command:

# modprobe nbd
# lsmod | grep nbd
nbd                      17642 0 
# qemu-nbd --connect=/dev/nbd0 asav971.qcow2

We see that it contains two partitions:

# fdisk /dev/nbd0 -l
Disk /dev/nbd0: 9126 MB, 9126805504 bytes
128 heads, 32 sectors/track, 4352 cylinders, total 17825792 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

    Device   Boot     Start        End     Blocks     Id     System
/dev/nbd0p1   *        4096    1048575     522240      c     W95 FAT32 (LBA)
/dev/nbd0p2         1052672   17825791    8386560      c     W95 FAT32 (LBA)

Now we mount the partitions. The first partition contains the ASA general firmware (asa*.bin) and the ASDM firmware (asdm*.bin), as well as a few other files such as the grub configuration:

# mkdir /mnt/qcow2
# mount /dev/nbd0p1 /mnt/qcow2
# tree /mnt/qcow2
/mnt/qcow2
├── asa971-smp-k8.bin
├── asa-restapi-132-lfbff-k8.SPA
├── asdm-77170.bin
├── backup_key
├── boot
│   ├── grub
│   │   ├── fat_stage1_5
│   │   ├── stage1
│   │   └── stage2
│   └── grub.conf
└── primary_key
# cat /mnt/qcow2/boot/grub.conf 
timeout 10

title bootflash:/asa971-smp-k8.bin
rootnoverify (hd0,0)
kernel (hd0,0)/
title bootflash:/asa971-smp-k8.bin with no configuration load

rootnoverify (hd0,0)
kernel (hd0,0)/ passwd_recover

end

Note that the asa*.bin inside the qcow2 used by GNS3 is in fact different from the one you may download for a real 64-bit hardware:

# md5sum /mnt/qcow2/asa971-smp-k8.bin
4b7c2ff288c6c231d29dd1b6848741d1 /mnt/qcow2/asa971-smp-k8.bin
# md5sum asa971-smp-k8.bin
9e484ed19848d321dd1c9bbf9844a285 asa971-smp-k8.bin

Once we have pulled out the files we are interested in, we unmount everything:

# umount /mnt/qcow2
# qemu-nbd --disconnect /dev/nbd0

By mounting the second partition, we notice it is initially empty. However, by mounting a qcow2 that was loaded and used in GNS3, we notice that it logically corresponds to the flash on a real device:

# qemu-nbd --connect=/dev/nbd0 asav962-7-modified.qcow2
# mount /dev/nbd0p2 /mnt/qcow2
# ls /mnt/qcow2/
anyconnect-win-2.5.2014-k9.pkg config-asav9627 coredumpinfo crash.txt csco_config FSCK0000.REC log smart-log

Automating unpacking/repacking

ASAs support lots of branches and versions. We are going to detail how we automated unpacking/repacking firmware for analysis and testing. Note that a couple of years after the original Breaking Bricks presentation, Alec Stuart released some shell scripts that did basic repacking [9].

Since we are interested in repacking the firmware, we need to understand where the rootfs is stored inside the asa*.bin. We manually worked out that the size of the vmlinuz (compressed vmlinux) and the size of the compressed rootfs are two DWORDs before a static string "quiet loglevel=0". If these two sizes are part of a documented file format, feel free to contact us, and we will update this article.

The compressed rootfs contains the "rootfs.img" string after the GZIP magic (1F 8B). vmlinuz contains the "Direct booting from..." string at the beginning of it.

Locating vmlinuz and compressed rootfs

We use a Python script we developed to unpack the firmware:

$ bin.py -u -f asa924-k8.bin 
[bin] Unpacking...
[bin] Writing asa924-k8-initrd-original.gz (29013841 bytes)...
[bin] unpack: Writing asa924-k8-vmlinuz (1368176 bytes)...
$ file asa924-k8-initrd-original.gz asa924-k8-vmlinuz
asa924-k8-initrd-original.gz: gzip compressed data, was "rootfs.img", from Unix, last modified: Wed Jul 15 06:19:52 2015
asa924-k8-vmlinuz:                 x86 boot sector

We see that the result is the same rootfs that we got earlier using binwalk.

$ gunzip asa924-k8-initrd-original.gz 
$ file asa924-k8-initrd-original 
asa924-k8-initrd-original: ASCII cpio archive (SVR4 with no CRC)
$ md5sum asa924-k8-initrd-original _asa924-k8.bin.extracted/rootfs.img
dc7b654cc65c4b6dc50e457fd7893c4c asa924-k8-initrd-original
dc7b654cc65c4b6dc50e457fd7893c4c _asa924-k8.bin.extracted/rootfs.img

We extract the rootfs using the following cpio.sh wrapper around the cpio command. Note that we need to be root here as we want all the existing file permissions/ownership values to remain when we repack the rootfs.

# cpio.sh -e -i $PWD/asa924-k8-initrd-original -d rootfs
# ls rootfs
asa bin boot config dev etc home init lib lib64 linuxrc mnt opt proc root sbin share sys tmp usr var

As an example of following the previous references [10] to enable gdb, we patch the script so lina is started with gdb attached:

# sed -i 's/#(.*)ttyUSB0(.*)/1ttyS02/' rootfs/asa/scripts/rcS

We get the following modified script:

# Use -g to have system await gdb connect during boot.
#echo "/asa/bin/lina_monitor -l -g -d" >> /tmp/run_cmd
# Use -s to specify a serial device other than the default /dev/ttyS1
echo "/asa/bin/lina_monitor -l -g -s /dev/ttyS0 -d" >> /tmp/run_cmd

We repack the rootfs:

# cpio.sh -c -d rootfs -o $PWD/rootfs.img.gz

Then we inject the modified rootfs into the new asa*.bin. Note that to ease the process of re-injecting the modified rootfs, we need to make sure the compressed rootfs is smaller than the original one:

$ bin.py -r -f asa924-k8.bin -g rootfs.img.gz
[bin] Repacking...
[bin] repack: Writing asa924-k8-repacked.bin (30597120 bytes)...

Finally, we check that our modifications were successfully applied by booting the modified firmware and making sure that gdbserver is waiting for us to connect to:

SMFW PID: 511, started gdbserver on member: 513//asa/bin/lina
SMFW PID: 511, created member ASA BLOB, PID=513
SMFW PID: 513, Starting /asa/bin/lina under gdbserver /dev/ttyS0
Process /asa/bin/lina created; pid = 516
Remote debugging using /dev/ttyS0

If it is not waiting, there might have been a problem with the injection. In order to automate the enabling of gdb and any changes that we might make to lina or other files, we wrote a shell script for unpacking/repacking firmware files. This wraps bin.py, cpio.sh and other tools such as binwalk:

# unpack_repack_bin.sh -i asa924-k8.bin --free-space --enable-gdb
[unpack_repack_bin] Single firmware detected
[unpack_repack_bin] unpack_one: asa924-k8.bin
[bin] Unpacking...
[bin] Writing asa924-k8-initrd-original.gz (29013841 bytes)...
[bin] unpack: asa924-k8-vmlinuz (1368176 bytes)...
[unpack_repack_bin] modify_one: asa924-k8.bin
[unpack_repack_bin] ENABLE GDB
[unpack_repack_bin] FREE SPACE IN .BIN
[unpack_repack_bin] repack_one: asa924-k8.bin
[bin] Repacking...
[bin] repack: Writing ./asa924-k8-repacked.bin (30597120 bytes)...
[unpack_repack_bin] MD5: 5aeb5d76c6a3851b9bd8ea9dce87701c ./asa924-k8-repacked.bin
[unpack_repack_bin] CLEANUP

The unpack_repack_bin.sh [11] script supports the following features:

Remove unused files to free some space and make the rootfs smaller (so it can be injected back into the asa*.bin)
Enable/disable ASLR
Inject a gdbserver (for firmware files that do not have it already)
Inject an SSH-triggered debug shell to help analysis e.g. allowing access to /proc//maps. This will be detailed in our future blog post focused on asadbg
Root the firmware to get a root shell at boot or directly enable gdb at boot instead of the regular boot

We are releasing a similar tool called unpack_repack_qcow2.sh [12], which wraps the QEMU nbd tools so it can deal with GNS3-enabled firmware files.

Datamining lina mitigations

We leveraged the unpacking firmware tools to analyse mitigations on all ASA firmware files. First, let’s detail what kind of information we want to collect.

We use file to check if the architecture is 32-bit or 64-bit:

$ file _asa924-k8.bin.extracted/rootfs/asa/bin/lina
_asa924-k8.bin.extracted/rootfs/asa/bin/lina: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.29, stripped

We obtain the exact Linux version using strings:

$ strings _asa924-k8.bin.extracted/2347E |grep "Linux version"
Linux version 2.6.29.6 (builders@releng02) (gcc version 4.3.4 (crosstool-NG-1.5.0) ) #1 PREEMPT Tue Jul 14 21:53:19 PDT 2015

We use the checksec.sh [13] script to get mitigations supported by the lina ELF. As we see below, asa924 lina binary does not support ASLR, NX or stack cookies.

$ checksec.sh --file _asa924-k8.bin.extracted/rootfs/asa/bin/lina
RELRO           STACK CANARY       NX            PIE      RPATH    RUNPATH    FILE
Partial RELRO   No canary found    NX disabled   No PIE   RPATH    RUNPATH _asa924-k8.bin.extracted/rootfs/asa/bin/lina

We check if ASLR is explicitly enabled or disabled at the Linux level. Here, we see it is disabled for asa924 but enabled for asav9627:

$ grep va_space _asa924-k8.bin.extracted/rootfs/asa/scripts/rcS.common
echo 0 > /proc/sys/kernel/randomize_va_space
$ grep va_space _asav962-7.qcow2.extracted/rootfs/asa/scripts/rcS.common
echo 2 > /proc/sys/kernel/randomize_va_space

Finally, we check if symbols are available for the executable. On an old 32-bit firmware you only have a few symbols available and they aren’t useful. So, when we refer to 32-bit firmware files, we say that they don’t have symbols.

$ readelf -s _asa924-k8.bin.extracted/rootfs/asa/bin/lina | grep .dynsym
Symbol table '.dynsym' contains 435 entries:

We’ve noticed the presence of some symbols on most of the recent 64-bit firmware and it looks like it affects both ASAv and SMP.

$ readelf -s _asav962-7.qcow2.extracted/rootfs/asa/bin/lina | grep .dynsym
Symbol table '.dynsym' contains 96469 entries:

We also collect some additional information like the glibc version and the base address of non-PIE lina binaries.

Mitigations table

Since we can unpack firmware files corresponding to multiple ASA versions, we decided to save all the mitigation-related information into a JSON database. This is achieved with the following script:

$ info.sh -h
Display/save mitigations and additional info for all firmware in the current folder
Usage: info.sh [--save-result --db-name ]

This gives the following output:

{
    "ASLR": false, 
    "glibc_version": "2.9",
    "fw": "asa924-k8.bin", 
    "imagebase": 134512640, 
    "RELRO": false, 
    "Canary": false, 
    "PIE": false, 
    "NX": false, 
    "uname": "Linux version 2.6.29.6 (builders@releng02) (gcc version 4.3.4 (crosstool-NG-1.5.0) ) #1 PREEMPT Tue Jul 14 21:53:19 PDT 2015", 
    "stripped": false, 
    "arch": 32, 
    "exported_symbols": false
}

We then print the results for all firmware in an easily referenceable table. We use a separate script called info.py to read out the JSON database.

Note that in the table below, dlmalloc has been shortened to dl and ptmalloc has been shortened to pt.

Version

Arch

ASLR

PIE

Canary

RELRO

Symbol

Strip

Linux

Glibc

Heap

Firmware

000

8.0.2

2.6.17.8

asa802-k8.bin

001

8.0.3

2.6.17.8

asa803-k8.bin

018

8.2.3

2.6.29.6

2.3.2

asa823-k8.bin

019

8.2.3

2.6.29.6

2.3.2

asa823-smp-k8.bin

106

9.1.6

2.6.29.6

2.9

dl 2.8.3

asa916-k8.bin

107

9.1.6

2.6.29.6

2.9

dl 2.8.3

asa916-smp-k8.bin

129

9.2.4

2.6.29.6

2.9

dl 2.8.3

asa924-k8.bin

130

9.2.4

2.6.29.6

2.9

dl 2.8.3

asa924-smp-k8.bin

131

9.3.1

2.6.29.6

2.9

dl 2.8.3

asa931-smp-k8.bin

132

9.3.2.200

3.10.19

2.18

pt 2.x

asa932-200-smp-k8.bin

154

9.3.2.200

3.10.19

2.18

pt 2.x

asav932-200.qcow2

144

9.4.3

3.10.55

2.18

pt 2.x

asa943-smp-k8.bin

145

9.4.4

3.10.55

2.18

pt 2.x

asa944-smp-k8.bin

146

9.5.1

3.10.62

2.18

pt 2.x

asa951-smp-k8.bin

147

9.5.2

3.10.62

2.18

pt 2.x

asa952-smp-k8.bin

152

9.6.2

3.10.62

2.18

pt 2.x

asa962-smp-k8.bin

168

9.6.2

3.10.62

2.18

pt 2.x

asav962.qcow2

153

9.7.1

3.10.62

2.18

pt 2.x

asa971-smp-k8.bin

169

9.7.1

3.10.62

2.18

pt 2.x

asav971.qcow2

Some interesting takeaways from mining this information are:

ASA <= 9.3.1 uses Linux 2.6.29.6, dlmalloc-2.8.x and does not have safe-unlinking enabled
ASA > 9.3.1 uses Linux 3.10.19, glibc’s ptmalloc2 and does have safe-unlinking enabled
smp generally indicates a 64-bit device (except for a few firmware in the early days)
Symbols starting to appear in the latest 64-bit firmware
ASLR and NX appear to have been introduced independently in different branches and were eventually supported together in a 3rd branch
ASA >= 9.5.3 supports ASLR, NX and ptmalloc2 safe unlinking, but still lacks stack canaries

The result of the datamining is available on our GitHub [14]; feel free to contribute.

Testing across numerous firmware

On a real device, the flash is usually a CompactFlash card or eUSB (flash chip with a USB interface). As we noted in the introduction blog post, one way to easily run tools across numerous firmware files is to store them on the flash and leverage the ROMMON boot command to boot the one you want at boot.

We use the unpacking/repacking tools to enable gdb on all the firmware we need and store them on the flash in advance. Then we use a separate set of scripts (which we generally refer to as asadbg) to load the one we want to analyse/debug on reboot. asadbg will be detailed in a future blog post.

Release branches

Another interesting aspect of looking at a large corpus of ASA firmware files is trying to understand how the release branches work, as you quickly get lost looking at so many files. By looking at a security advisory [15], we infer that Cisco supports quite a few different release branches for the ASA. Each branch has its own patch level and some branches may not be officially supported or patched by Cisco anymore (referred to as end-of-life, or EOL).

Example of fixed releases for CVE-2016-1287 [18]

This is useful for understanding if an ASA version is impacted by a given vulnerability. For instance, an ASA 9.1.6 is vulnerable to the IKE heap overflow, whereas an ASA 9.0.4.38 is patched, as detailed above.

What versions are EOL?

To our knowledge there isn’t any summary of ASA branches being EOL, though an official list of EOL devices is here [16]. Feel free to contact us if we are missing something. However, it is possible to use the Cisco ASA advisories [17][18][19][20] to deduce branch EOL status.

Cisco ASA Branch	Date	EOL?	Notes
7.2	<= Feb 2016	Yes
8.0	<= Feb 2016	Yes
8.1	<= Feb 2016	Yes
8.2	<= Feb 2016	Yes	Exceptional patch for IKE heap overflow
8.3	<= Feb 2016	Yes
8.4	<= May 2016	Yes
8.5	<= Feb 2016	Yes
8.6	<= Feb 2016	Yes
8.7	<= May2016	Yes
9.0	<= Feb 2017	Yes
9.1		No
9.2		No
9.3	<= Feb 2017	Yes
9.4		No
9.5		No
9.6		No
9.7		No
9.8		No

asafw at a glance

As demonstrated earlier, the collection of convenience scripts we simply refer to as asafw is quite useful in automating analysis and manipulating firmware:

unpack_repack_bin.sh: unpack an asa*.bin, modify it automatically and repack it. Also allows unpacking the rootfs only
unpack_repack_qcow2.sh: mount a QEMU image, unpack an asa*.bin, modify it automatically and repack everything. Also allows unpacking the rootfs only
bin.py, cpio.sh: directly manipulate asa*.bin or initrd CPIO file. They are used by unpack_repack_*.sh scripts above
lina.py: patch lina to, for example, inject a debug shell. Used by unpack_repack_*.sh
info.sh: analyze mitigations in all extracted firmware from a directory
info.py: display mitigations in a beautiful table
linabins.sh: copy all lina files from all extracted firmware directories so they can easily be accessed

Conclusion

We detailed asafw which is a framework to manipulate Cisco ASA firmware files. You can access the tool on GitHub [21]. In a future blog post we will detail how to debug Cisco ASA devices leveraging a new tool called asadbg.

We would appreciate any feedback or corrections. If you would like to contact us we can be reached by email or twitter: aaron(dot)adams(at)nccgroup(dot)trust / @fidgetingbits and cedric(dot)halbronn(at)nccgroup(dot)trust / @saidelike.