Return to main page

MikroTik Router Security Analysis: Uncovering a hidden kernel module in a binary

(started 28/8/2007, added 11/10/2007)

The MikroTik Wireless Router is a Linux embedded wireless router, focusing on various functionality such as bandwidth management, Firewalling, VPN server/client, and various other things. As with all embedded linux based software, it is interesting to pull it apart :)

It has been around for a while now… a couple of years ago when I analysed the software / pulling it apart, it had drivers/firmware to turn standard Orinoco wireless cards into an Access Point (which as far as I know isn't possible otherwise, at least not when I was looking at it.)

For the purposes of this article, I am looking at mikrotik-2.9.46.iso (MD5sum: 65aa908dd748ccf72ad9f588613dfe31, SHA1sum: 5e5ed13498db8d9745a701f75e58da3ef6701e58). For the most part, I have used QEMU to emulate the hardware/software environment to install it on. This has several advantages, such as being able to edit the "disk" it's using easily, amongst other things.

Performing active analysis of MikroTik router components

To perform more active analysis of the MikroTik components, we could copy the applicable binaries and associated libraries to another linux platform. This would allow us to strace the binary, debug it (which is incredibly useful for exploit development), and monitor the activities it performs in general. Furthermore, we can copy the kernel and applicable modules to perform further analysis on them, and to allow the environment to be replicated a lot better.

The analysis environment / setup

For this article, I have done a basic network install of Debian 4rc1. After performing the installation and installing a bunch of generic tools (strace/gdb/gcc/ltrace/openssh-server/nasm/etc), I then extracted the Mikrotik kernel and modules, and put the applicable files into their place.

[box] # wget http://felinemenace.org/~andrewg/MikroTik_Router_Security_Analysis_Part1/MikroTik-2.9.46-kernel-initrd.tgz
--08:58:00-- http://felinemenace.org/~andrewg/MikroTik_Router_Security_Analysis_Part1/MikroTik-2.9.46-kernel-initrd.tgz
           => `MikroTik-2.9.46-kernel-initrd.tgz'
Resolving felinemenace.org... 69.55.233.10
Connecting to felinemenace.org|69.55.233.10|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1,832,960 (1.7M) [application/x-gzip]

100%[====================================>] 1,832,960    327.68K/s    ETA 00:00

08:58:06 (330.42 KB/s) - `MikroTik-2.9.46-kernel-initrd.tgz' saved [1832960/1832960]
[box] # tar xzf MikroTik-2.9.46-kernel-initrd.tgz
[box] # mv lib/modules/2.4.31/ /lib/modules/2.4.31
[box] # mv boot/vmlinuz /boot/vmlinuz-2.4.31
[box] # cd /boot/grub/
[box] # cat >>menu.lst <<_EOF_
> title MikroTik 2.4.31 / 2.9.46
> root            (hd0,0)
> kernel /boot/vmlinuz-2.4.31
>
> _EOF_
[box] # sync ; reboot

While the QEMU image was rebooting, I wondered if it would work, due to differences in 2.6 and 2.4 kernels.. Firstly though, it appeared I may have to experiment with initrd images to load the applicable drivers for it and generally mess around as when booting it displayed:

  Booting 'MikroTik 2.4.31 / 2.9.46'

root                    (hd0,0)
 Filesystem type is ext2fs, partition type is 0x83
kernel /boot/vmlinuz-2.4.31
    [Linux-bzImage, setup=0x1400, size=0xa044d]

Uncompressing Linux... Ok, booting the kernel.
Kernel panic: VFS: Unable to mount root fs on 09:00

Booting back into the standard Debian kernel:

[box] # cd /boot/
[box] # cp /root/boot/initrd.rgz mikro-initrd.rgz
[box] # cd grub/
[box] # echo initrd /boot/mikro-initrd.rgz >>menu.lst
[box] # sync ; reboot

Unfortunately, this gives a similar error message before. We know previously from mounting the mikrotik root filesystem it was ext3, but we can (attempt) to verify what filesystems they support. Looking over the original find output for the kernel modules, we don't see any filesystem modules for the default ext2 filesystem.

We can verify the filesystems supported by analysing the vmlinuz file. To summarise, basically, this file contains some bootup to get the machine into a decent state, a gzip decompression routine, and a heap of compressed data. The information we're interested in is in the compressed data, so we have to decompress it. As it's not a standard gzip file, we can't just run gunzip on it and be done with it, we need extract the compressed data.

Fortunately, this can be done rather easily because the the gzip header / magic bytes, which allows us to find suitable offsets to attempt decompression. The gzip magic bytes can be found by doing:

[box] # cd /tmp
[box] # cp /etc/passwd .
[box] # gzip passwd
[box] # xxd passwd.gz  | head
0000000: 1f8b 0808 2061 d346 0003 7061 7373 7764  .... a.F..passwd
0000010: 0065 93c1 6ec2 300c 86ef 3c05 c74d 0285  .e..n.0...<..M..
0000020: 5260 90e3 3469 9771 d99e c06d 4289 d626  R`..4i.q...mB..&
0000030: 55d2 5278 fbd9 71a0 4593 adc8 7ffc 2536  U.Rx..q.E.....%6
0000040: 0ef5 ce75 f22a 5768 9e42 c16b 61ac 2820  ...u.*Wh.B.ka.(
0000050: 9c67 0a74 e32c 1219 5a12 a20f 5e04 4498  .g.t.,..Z...^.D.
0000060: 438a e2ab 5ca3 dd77 1fa9 700b 98ca d128  C...\..w..p....(
0000070: 124a 5f26 295b 626e 2377 db6d be91 514e  .J_&)[bn#w.m..QN
0000080: cea2 9c55 d068 3abf 95bb 9564 11ab a730  ...U.h:....d...0
0000090: 5dd4 0095 dfc9 6c2d 2914 17f0 a284 f2ac  ].....l-).......

For the purpose on hand, we'll use 0x1f8b as our marker.

[box] # xxd /boot/vmlinuz-2.4.31 | egrep "\b1f8b|1f\b \b8b" | head -n 5
00049a0: a9d0 0900 1f8b 0800 b533 bc46 0203 ec5d  .........3.F...]
0004f20: 3613 e31f 8b6d a730 ef6f 1078 d415 4401  6....m.0.o.x..D.
001c6c0: 0a1f 8bab 69a2 a7e5 533b 4d60 764f 93bc  ....i...S;M`vO..
00381b0: c3ba d727 7964 631f 8baf 810c 2704 206f  ...'ydc.....'. o
003fdf0: 972f d2d6 d50e 5d37 180b 771f 8bc5 b43d  ./....]7..w....=

To extract the the compressed data from the vmlinuz-2.4.31 file, dd does the trick easily. We'll start from our first match and work our way down.

[box] # dd if=/boot/vmlinuz-2.4.31 of=vmlinuz.gz bs=1 skip=$((0x49a4))
662093+0 records in
662093+0 records out
662093 bytes (662 kB) copied, 16.0954 seconds, 41.1 kB/s
[box] # file vmlinuz.gz
vmlinuz.gz: gzip compressed data, from Unix, last modified: Fri Aug 10
19:45:25 2007, max compression
[box] # gunzip vmlinuz.gz
[box] # strings -a vmlinuz
... skip ...

After seeing various ext2 related things, I realised it was probably a problem with something else. Reviewing grub's menu.lst, it becomes obvious:

title           Debian GNU/Linux, kernel 2.6.18-5-686
root            (hd0,0)
kernel          /boot/vmlinuz-2.6.18-5-686 root=/dev/hda1 ro
initrd          /boot/initrd.img-2.6.18-5-686
savedefault
.... skip...
title MikroTik 2.4.31 / 2.9.46
root            (hd0,0)
kernel /boot/vmlinuz-2.4.31

initrd /boot/mikro-initrd.rgz

The kernel line for the MikroTik entry misses out on some parameters. Fixing the applicable line, (kernel /boot/vmlinuz-2.4.31 root=/dev/hda1 ro single) and rebooting, it works. At least the decompressing of the kernel image can come in use for further investigation work. One last thing to note is that it requires the debian modutils package to work with 2.4 kernels.

Anyways, moving on, the debian image boots up and works reasonably with their kernel / modules. I had to load the ne2k-pci module manually (via modprobe ne2k-pci) to bring up networking under QEMU.

Another issue I had under the debian/mikrotik hybrid I created, was that the MikroTik kernel does not have AF_UNIX/AF_FILE support built-in, so useful programs like sysklogd and sshd would not run by default… However, it ships this as a module, so modprobe unix took care of this issue.

In order to run the MikroTik binaries (/nova/bin/), I needed to copy various files. I copied /nova/ over, and made a directory called /lib_mikro/ where I could copy various libary files over that resided in the /lib directory on the MikroTik installation.

In order to use these libraries in a non-standard directory location, the environment variable LD_LIBRARY_PATH can be set. This way only the applicable MikroTik binaries can be ran with correct library versions.

Examining the fileman binary

Doing some prelimiary analysis on the fileman binary shows that it appears to be expecting a network file descriptor on fd 3.

[box] $ LD_LIBRARY_PATH=/lib_mikro/ strace -f ./fileman
execve("./fileman", ["./fileman"], [/* 14 vars */]) = 0
uname({sys="Linux", node="debian", ...}) = 0
brk(0)                                  = 0x805861c
...
rt_sigaction(SIGFPE, {0x4002c8ca, [], SA_RESTORER|SA_RESTART|SA_SIGINFO,
0x4009d4c0}, NULL, 8) = 0
getsockname(3, 0xbffffd18, [110])       = -1 EBADF (Bad file descriptor)
socket(PF_FILE, SOCK_STREAM, 0)         = -1 EAFNOSUPPORT (Address family not
supported by protocol)
exit_group(1)                           = ?
Process 2147 detached

Unfortunately, debian's version of bash does not have /dev/tcp/ support, so unfortunately it's not as easy as nc -l 12121 on one terminal, and … ./fileman 3</dev/tcp/blah/.

After quickly writing some code to do what's needed, we can run fileman (and probably others). It's available here if you would like it. Usage is simple, python fd3.py <program to execute> <arguments for program>. Note that the first argument you specify for the program is argv[0] - not argv[1]. An example of a command line would be python fd3.py /usr/bin/strace strace -f /path/to/program.

Another issue appeared when trying to run fileman with a valid file descriptor on fd 3 - it wanted /tmp/novasock to be a valid file descriptor:

[box] $ LD_LIBRARY_PATH=/lib_mikro fd3 `which strace` strace -f ./fileman
getsockname(3, {sa_family=AF_INET, sin_port=htons(31313), sin_addr=inet_addr("192.168.254.3")}, [16]) = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 4
connect(4, {sa_family=AF_FILE, path="/tmp/novasock"}, 110) = -1 ENOENT (No such file or directory)
close(4)                                = 0
exit_group(1)                           = ?

Looking for /tmp/novasock we find the the loader binary seems to have what we're after:

[box] $ strings -f * | grep novasock
loader: /tmp/novasock
Analysing kernel module and userland interaction

Performing some initial analysis via strace on loader reveals an interesting / startling behaviour in loader:

[box] $ LD_LIBRARY_PATH=/lib_mikro strace -f ./loader
...
[pid  2361] create_module("qwink", 1430) = 0xc8831000
[pid  2361] init_module(0x80553fc, 134587376, umovestr: Input/output error
0xa7) = 0
[pid  2361] delete_module("qwink")      = 0
...

The interesting thing about this particular piece of code is that it is loading a linux kernel module (LKM), and immediately removes the kernel module. This is particularly interesting as it would appear to be a kernel module that's meant to be out of sight.

To dump the module, we could hook init_module to perform our required actions, however, first we have to verify it's easily possible:

[box[ $ objdump -R loader  | grep -i module
[box] $

This is interesting, as it appears to not be importing the various module library calls, performing some more analysis:

[box] $ objdump -dtrs loader | grep module
...
 804fb69:       e8 3a 42 00 00          call   8053da8 <delete_module+0x336>
08053a20 <create_module>:
 8053a36:       76 0c                   jbe    8053a44 <create_module+0x24>
08053a49 <init_module>:
...

The interesting thing about this output is the sections where the module names are surrounded by the angle brackets; this indicates that those functions exist in the .text of loader:

08053a49 <init_module>:
 8053a49:       55                      push   %ebp
 8053a4a:       b8 80 00 00 00          mov    $0x80,%eax
 8053a4f:       89 e5                   mov    %esp,%ebp
 8053a51:       53                      push   %ebx
 8053a52:       8b 4d 0c                mov    0xc(%ebp),%ecx
 8053a55:       8b 5d 08                mov    0x8(%ebp),%ebx
 8053a58:       cd 80                   int    $0x80
 8053a5a:       83 f8 82                cmp    $0xffffff82,%eax
 8053a5d:       89 c3                   mov    %eax,%ebx
 8053a5f:       76 0c                   jbe    8053a6d <init_module+0x24>
 8053a61:       f7 db                   neg    %ebx
 8053a63:       e8 54 70 ff ff          call   804aabc <__errno_location@plt>
 8053a68:       89 18                   mov    %ebx,(%eax)
 8053a6a:       83 cb ff                or     $0xffffffff,%ebx
 8053a6d:       89 d8                   mov    %ebx,%eax
 8053a6f:       5b                      pop    %ebx
 8053a70:       5d                      pop    %ebp
 8053a71:       c3                      ret

This appears to be a standard implementation of the _syscallX() macros in the asm/unistd.h. While we're staring at objdump -d output, we may as well look at where this code is (statically) being called from:

 8053ea6:       ff 75 dc                pushl  0xffffffdc(%ebp)
 8053ea9:       ff 35 20 64 05 08       pushl  0x8056420
 8053eaf:       e8 95 fb ff ff          call   8053a49 <init_module>

At 0x8053ea9 it pushes a static address (0x8056420) which does not correspond to our strace output. Having a brief look at where that variable is used before hand:

[box] $ objdump -dtrsRS loader | grep -C 2 8056420
...
 8053db4:       83 ec 18                sub    $0x18,%esp
 8053db7:       c7 45 f0 00 00 00 00    movl   $0x0,0xfffffff0(%ebp)
 8053dbe:       8b 3d 20 64 05 08       mov    0x8056420,%edi
 8053dc4:       ba 10 00 00 00          mov    $0x10,%edx
 8053dc9:       f2 ae                   repnz scas %es:(%edi),%al
--
 8053e19:       e8 3e 70 ff ff          call   804ae5c <memcpy@plt>
 8053e1e:       53                      push   %ebx
 8053e1f:       ff 35 20 64 05 08       pushl  0x8056420
 8053e25:       56                      push   %esi
 8053e26:       e8 31 70 ff ff          call   804ae5c <memcpy@plt>
 8053e2b:       ff 75 ec                pushl  0xffffffec(%ebp)
 8053e2e:       ff 35 20 64 05 08       pushl  0x8056420
 8053e34:       e8 e7 fb ff ff          call   8053a20 <create_module>
 8053e39:       83 c4 24                add    $0x24,%esp
--
 8053ea1:       e8 57 fd ff ff          call   8053bfd <delete_module+0x18b>
 8053ea6:       ff 75 dc                pushl  0xffffffdc(%ebp)
 8053ea9:       ff 35 20 64 05 08       pushl  0x8056420
 8053eaf:       e8 95 fb ff ff          call   8053a49 <init_module>
 8053eb4:       83 c4 10                add    $0x10,%esp
--
 8053ee0:       83 c4 0c                add    $0xc,%esp
 8053ee3:       8b 55 f0                mov    0xfffffff0(%ebp),%edx
 8053ee6:       ff 35 20 64 05 08       pushl  0x8056420
 8053eec:       39 c2                   cmp    %eax,%edx
 8053eee:       0f 94 c3                sete   %bl

So, it appears it's used a bit to set it up. Being the somewhat lazy type, it would be easy enough to write a dynamic library to modify the .text segment to insert a hook. The mechanisms used for this is discussed in a paper I wrote available here.

The hook code aim is simple, which is to dump the applicable information sent to init_module. It also appears that the init_module function declaration changes between 2.4 and 2.6 kernel versions:

2.4:
 *`int init_module(const char *name, struct module *image);`
.2.6:
 * `long sys_init_module (void *umod, unsigned long len, const char *uargs);`

The strace output above is for 2.6, not 2.4. After locating a suitable 2.4 init_module man page, we see that the second parameter is a pointer to a structure:

The module image begins with a module structure and is followed by code and data as appropriate.  The module structure is defined as follows:

struct module {
  unsigned long         size_of_struct;
  struct module        *next;
  const char           *name;
  unsigned long         size;
  long                  usecount;
  unsigned long         flags;
  unsigned int          nsyms;
  unsigned int          ndeps;
  struct module_symbol *syms;
  struct module_ref    *deps;
  struct module_ref    *refs;
  int                 (*init)(void);
  void                (*cleanup)(void);
  const struct exception_table_entry *ex_table_start;
  const struct exception_table_entry *ex_table_end;
#ifdef __alpha__
  unsigned long gp;
#endif
};

At least the required information is available to make it easier. After writing the required the code (which is available here)

Running our hooking library code (which is available , we get:

[box] $ LD_LIBRARY_PATH=/lib_mikro LD_PRELOAD=/tmp/hook-loader.so ./loader
forked
creating loader
--> In an int3 handler
--> Create module return address is 0xc8831000
--> In an int3 handler
--> working our magic for qwink

Matching the dumped header information with the struct module output, we get:

[box] $ xxd /tmp/module_header
0000000: 3c00 0000 0000 0000 9015 83c8 9605 0000  <...............
0000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000020: 0000 0000 0000 0000 0000 0000 4010 83c8  ............@...
0000030: 0000 0000 0000 0000 0000 0000            ............
... matching up with the above struct module ...
size_of_struct: 0x0000003c      (0x0000)
next: NULL                      (0x0004)
name: 0xc8831590                (0x0008)
size: 0x00000596                (0x000c)
usecount: NULL                  (0x0010)
flags: 0                        (0x0014)
nsyms: 0                        (0x0018)
ndeps: 0                        (0x001c)
syms: NULL                      (0x0020)
deps: NULL                      (0x0024)
refs: NULL                      (0x0028)
init: 0xc8831040                (0x002c)
cleanup: NULL                   (0x0030)
ex_table_start: NULL            (0x0034)
ex_table_end: NULL              (0x0038)

So, our output in /tmp/module_text starts at 0xc883103c, and is 1430 bytes long. Disassembling the binary dump in /tmp/module_text, looking for the init function pointer that gets called:

[box] $ ndisasm -b 32 -o $((0xc883103c)) /tmp/module_text
...
C8831040  83EC18            sub esp,byte +0x18
C8831043  53                push ebx
C8831044  E8F7040000        call 0xc8831540
C8831049  8D5818            lea ebx,[eax+0x18]
C883104C  C7430400000000    mov dword [ebx+0x4],0x0
C8831053  C7401800000000    mov dword [eax+0x18],0x0
...

We can combine the gdb debugging functionality in qemu in conjunction with a disassembler such as IDA Pro , we can use the gdb stub functionality of qemu, we can disassemble and follow what is happening when the init code is executed.

Testing our analysis against the init function by setting break points on:

Just to make sure we catch everything.. I don't expect 0xc053ffd0 to be hit, but it is a valid kernel address :)

Testing the theory out under gdb/qemu doesn't pan out how it was expected to be:

(gdb) break *0xc053ffd0
Breakpoint 4 at 0xc053ffd0
(gdb) break *0xc8831040
Breakpoint 5 at 0xc8831040
(gdb) c
Continuing.

... Run the loader program ...

Breakpoint 5, 0xc8831040 in ?? ()
(gdb) x/10i $eip
0xc8831040:     sub    $0x18,%esp
0xc8831043:     push   %ebx
0xc8831044:     call   0xc8831540
0xc8831049:     lea    0x18(%eax),%ebx
0xc883104c:     movl   $0x0,0x4(%ebx)
0xc8831053:     movl   $0x0,0x18(%eax)
0xc883105a:     mov    0xc0252024,%eax
0xc883105f:     inc    %eax
0xc8831060:     mov    %eax,0x8(%ebx)
0xc8831063:     call   0xc8831090
(gdb) i r ebx
ebx            0xc8831000       -930934784

Now that we're able to trace this code, we should work out what information would be desirable to make this process far easier:

Another couple of things we could try to help our reversing efforts:

[box] $ cat /proc/version
Linux version 2.4.31 (build@builder2) (gcc version 2.95.4 20011002 (Debian prerelease)) #1 Fri Aug 10 12:43:55 EEST 2007

After some researching, it was determined that the gcc version 2.95.4 20011002 is available if a default install of Debian 3.0r1 is installed. I only needed cd1 and cd2 from the set of cd's to install the required tools.

After performing a compile of the default kernel configuration shipped in 2.4.31 sans PCMCIA support, I used the 2pelf tool available here to generate the signatures off the .o files in the linux tree, then used the IDA tool sigmake to generate the signature information. (I love bash scripting to automate these tasks).

If needed, the signatures could be regenerated several times with different compilation parameters in order to increase success of signature matching.

Since it would be helpful to have a full kernel text image in IDA Pro, we can use the memsave function in QEMU to generate an appropriate memory dump. To get the most useful memory dump, I'll generate the dump when qemu has hit the breakpoint on the "qwink" init function.

The vmlinux file generated when I compiled the 2.4.31 kernel indicated that the kernel got loaded at 0xc0100000, so we'll dump from there to 0xd0000000… This actually turned out to be a bad idea, as IDA couldn't analyse such large files. After some experimentation and looking at the output of objdump -fp on the compiled vmlinux file, I dumped from 0xc0100000 to 0xc1000000.

(qemu) memsave 3222274048 15728640 d
*pause*
(qemu)

The first number is 0xc0100000 and the second is 0xc1000000 - 0xc0100000.

After dumping the kernel memory to disk, then loading it into IDA, and applying the signatures that was generated before, we see a lot of function names pop up. Sometimes they're correct, sometimes they don't appear to be, sometimes they don't find functions we find interesting, but in general they save us a lot of work :)

The init code is as follows from IDA:

seg001:C8831040 module_init_code:
seg001:C8831040                 sub     esp, 18h
seg001:C8831043                 push    ebx
seg001:C8831044                 call    GET_C8831550    ; eax = 0xc8831550
seg001:C8831049                 lea     ebx, [eax+18h]  ; ebx = eax + 0x18 = 0xc8831568, this is struct timer_list data
seg001:C883104C                 mov     dword ptr [ebx+4], 0 ; sets next pointer to null
seg001:C8831053                 mov     dword ptr [eax+18h], 0 ; sets prev pointer to null ?
seg001:C883105A                 mov     eax, ds:jiffies
seg001:C883105F                 inc     eax             ; schedules the code to be executed in one jiffie
seg001:C8831060                 mov     [ebx+8], eax    ; sets the scheduler expires
seg001:C8831063                 call    GET_C88310a0
seg001:C8831068                 mov     [ebx+10h], eax  ; function pointer used (execd_by_timer) below
seg001:C883106B                 mov     eax, 0FFFFE000h
seg001:C8831070                 and     eax, esp        ; get current
seg001:C8831072                 mov     [ebx+0Ch], eax  ; data pointer for call back. current task
seg001:C8831075                 add     esp, -0Ch       ; no idea what it's doing here. possibly aligning stack to paragraph boundary?
seg001:C8831078                 mov     eax, offset add_timer
seg001:C883107D                 push    ebx             ; push offset to struct timer_list data
seg001:C883107E                 call    eax ; add_timer
seg001:C8831080                 xor     eax, eax        ; return NULL
seg001:C8831082                 add     esp, 10h
seg001:C8831085                 pop     ebx
seg001:C8831086                 add     esp, 18h
seg001:C8831089                 retn

Having a look at the IDA loader binary, we see:

Example: Loader: load_module_into_kernel
.text:08053DA8 load_module_into_kernel proc near       ; CODE XREF: sub_804DD20+25p
.text:08053DA8                                         ; sub_804F572+10Dp ...
.text:08053DA8
.text:08053DA8 constructed_image= dword ptr -24h
.text:08053DA8 roundup_size    = dword ptr -20h
.text:08053DA8 start_of_module_data= dword ptr -18h
.text:08053DA8 total_size      = dword ptr -14h
.text:08053DA8 kernel_modified_variable= dword ptr -10h
.text:08053DA8 var_C           = dword ptr -0Ch
.text:08053DA8 tv_usecs_challenge= dword ptr  8
.text:08053DA8
.text:08053DA8                 push    ebp
.text:08053DA9                 cld                     ; clear direction flag
.text:08053DAA                 mov     ebp, esp
.text:08053DAC                 push    edi
.text:08053DAD                 xor     eax, eax
.text:08053DAF                 push    esi
.text:08053DB0                 or      ecx, 0FFFFFFFFh
.text:08053DB3                 push    ebx
.text:08053DB4                 sub     esp, 18h
.text:08053DB7                 mov     [ebp+kernel_modified_variable], 0 ; initialise the variable written to by access_process_vm to 0
.text:08053DBE                 mov     edi, module_name
.text:08053DC4                 mov     edx, 10h
.text:08053DC9                 repne scasb
.text:08053DCB                 mov     ebx, ecx
.text:08053DCD                 mov     eax, 3Ch
.text:08053DD2                 not     ebx             ; embedded strlen()
.text:08053DD4                 call    sub_8053D95     ; calcuate if additional space is needed in a memory copy later on
.text:08053DD9                 mov     [ebp+roundup_size], eax ; save the calcutation for later use
.text:08053DDC                 lea     edx, [eax+ebx+1360] ; calcuate total size needed, + 1360
.text:08053DE3                 push    edx             ; size
.text:08053DE4                 mov     [ebp+total_size], edx
.text:08053DE7                 call    _malloc         ; allocate the required space
.text:08053DEC                 mov     [ebp+constructed_image], eax ; save the results of malloc()
.text:08053DEF                 mov     ecx, [ebp+total_size]
.text:08053DF2                 cld                     ; clear direction flag
.text:08053DF3                 mov     edi, eax
.text:08053DF5                 xor     eax, eax        ; write nulls
.text:08053DF7                 rep stosb               ; embedded memset
.text:08053DF9                 push    510h            ; size_t
.text:08053DFE                 mov     edi, [ebp+constructed_image] ; get the allocated memory
.text:08053E01                 add     edi, [ebp+roundup_size] ; move past the module structure
.text:08053E04                 push    offset module_init_code ; void *
.text:08053E09                 lea     edx, [edi+510h] ; edx now points PAST the module_init_code and all that
.text:08053E0F                 push    edi             ; void *
.text:08053E10                 lea     esi, [edi+1360] ; esi = module name pointer
.text:08053E16                 mov     [ebp+start_of_module_data], edx
.text:08053E19                 call    _memcpy         ; copy the module_init_code (of length 0x510)
.text:08053E19                                         ; to the allocated memory
.text:08053E1E                 push    ebx             ; size_t. this is calcuated previously from strlen(module_name)
.text:08053E1F                 push    module_name     ; void *
.text:08053E25                 push    esi             ; points to module name
.text:08053E26                 call    _memcpy         ; setup module name
.text:08053E2B                 push    [ebp+total_size] ; int
.text:08053E2E                 push    module_name     ; name
.text:08053E34                 call    create_module
.text:08053E39                 add     esp, 24h
.text:08053E3C                 cmp     eax, 0FFFFFFFFh ; eax contains base of the allocated kernel memory
.text:08053E3F                 jnz     short loc_8053E48
.text:08053E41                 push    offset aFailedToCreate ; "failed to create module"
.text:08053E46                 jmp     short loc_8053EC0
.text:08053E48 ; ---------------------------------------------------------------------------
.text:08053E48
.text:08053E48 loc_8053E48:                            ; CODE XREF: load_module_into_kernel+97j
.text:08053E48                 mov     ecx, [ebp+constructed_image]
.text:08053E4B                 mov     dword ptr [ecx], 3Ch ; set size of module header to 0x3c
.text:08053E51                 mov     ecx, [ebp+roundup_size]
.text:08053E54                 lea     edx, [eax+ecx]  ; edx = start of module code, in kernel space
.text:08053E57                 mov     ecx, [ebp+constructed_image]
.text:08053E5A                 lea     eax, [edx+1360] ; eax = end of module code, start of "qwink"
.text:08053E60                 mov     [ecx+2Ch], edx  ; set module size
.text:08053E63                 mov     edx, [ebp+tv_usecs_challenge] ; arg_0 = tv.tv_usecs ?
.text:08053E66                 mov     [ecx+8], eax    ; write the name pointer, points to end of module, "qwink"
.text:08053E69                 mov     eax, [ebp+total_size]
.text:08053E6C                 mov     [edi+510h], edx ; write challenge to kernel image (data section)
.text:08053E72                 mov     [ecx+0Ch], eax  ; write size of complete module to header
.text:08053E75                 mov     ecx, [ebp+start_of_module_data]
.text:08053E78                 mov     eax, kernel_ptr_c0105000
.text:08053E7D                 mov     [ecx+4], eax
.text:08053E80                 mov     eax, dword_8056A68 ; some mystical value
.text:08053E85                 mov     [ecx+8], eax
.text:08053E88                 mov     eax, dword_8056A6C ; 0x3f
.text:08053E8D                 mov     [ecx+0Ch], eax
.text:08053E90                 lea     eax, [ebp+kernel_modified_variable] ; get the address of the variable
.text:08053E93                 mov     [ecx+10h], eax  ; variable that is going to be written to
.text:08053E96                 call    alloc_rc4_t
.text:08053E9B                 push    [ebp+start_of_module_data]
.text:08053E9E                 mov     esi, eax
.text:08053EA0                 push    eax
.text:08053EA1                 call    rc4_init_key_encrypt ; (rc4_t, start_of_module_data)
.text:08053EA6                 push    [ebp+constructed_image] ; image
.text:08053EA9                 push    module_name     ; name
.text:08053EAF                 call    init_module
.text:08053EB4                 add     esp, 10h
.text:08053EB7                 test    eax, eax
.text:08053EB9                 jz      short loc_8053EC9
.text:08053EBB                 push    offset aFailedToLoadMo ; "failed to load module"
.text:08053EC0
.text:08053EC0 loc_8053EC0:                            ; CODE XREF: load_module_into_kernel+9Ej
.text:08053EC0                 call    _puts
.text:08053EC5                 xor     eax, eax
.text:08053EC7                 jmp     short loc_8053EFE
.text:08053EC9 ; ---------------------------------------------------------------------------
.text:08053EC9
.text:08053EC9 loc_8053EC9:                            ; CODE XREF: load_module_into_kernel+111j
.text:08053EC9                                         ; load_module_into_kernel+126_j
.text:08053EC9                 mov     eax, [ebp+kernel_modified_variable]
.text:08053ECC                 test    eax, eax
.text:08053ECE                 jz      short loc_8053EC9 ; while the variable hasn't been modified, loop. Ugh.
.text:08053ED0                 push    offset unk_8056960 ; not sure yet
.text:08053ED5                 xor     ebx, ebx
.text:08053ED7                 push    [ebp+tv_usecs_challenge] ; challenge
.text:08053EDA                 push    esi             ; rc4 structure, returned from rc4_init
.text:08053EDB                 call    check_challenge_response
.text:08053EE0                 add     esp, 0Ch
.text:08053EE3                 mov     edx, [ebp+kernel_modified_variable]
.text:08053EE6                 push    module_name     ; name
.text:08053EEC                 cmp     edx, eax        ; result from check_challenge_response
.text:08053EEE                 setz    bl              ; set return code if they're the same
.text:08053EF1                 call    delete_module
.text:08053EF6                 push    esi
.text:08053EF7                 call    free_rc4_t_tailcall
.text:08053EFC                 mov     eax, ebx        ; set return value (based on check_challenge_response)
.text:08053EFE
.text:08053EFE loc_8053EFE:                            ; CODE XREF: load_module_into_kernel+11Fj
.text:08053EFE                 lea     esp, [ebp-0Ch]
.text:08053F01                 pop     ebx
.text:08053F02                 pop     esi
.text:08053F03                 pop     edi
.text:08053F04                 pop     ebp
.text:08053F05                 retn
.text:08053F05 load_module_into_kernel endp
.text:08053F05
Example: Loader: check_challenge_response
.text:08053CEA check_challenge_response proc near      ; CODE XREF: load_module_into_kernel+133p
.text:08053CEA
.text:08053CEA local_rc4_structure= dword ptr -14h
.text:08053CEA challenge_copy  = dword ptr -10h
.text:08053CEA var_C           = dword ptr -0Ch
.text:08053CEA rc4_t           = dword ptr  8
.text:08053CEA challenge       = dword ptr  0Ch
.text:08053CEA unknown_data    = dword ptr  10h
.text:08053CEA
.text:08053CEA                 push    ebp
.text:08053CEB                 mov     ebp, esp
.text:08053CED                 push    edi
.text:08053CEE                 push    esi
.text:08053CEF                 xor     esi, esi
.text:08053CF1                 push    ebx
.text:08053CF2                 push    ecx
.text:08053CF3                 push    ecx
.text:08053CF4                 mov     edi, [ebp+rc4_t]
.text:08053CF7                 mov     eax, [ebp+challenge]
.text:08053CFA                 push    edi
.text:08053CFB                 mov     [ebp+challenge_copy], eax
.text:08053CFE                 call    rc4_dup_struct
.text:08053D03                 mov     [ebp+local_rc4_structure], eax ; duplicated structure
.text:08053D06                 mov     eax, [ebp+unknown_data]
.text:08053D09                 push    dword ptr [eax+100h] ; offset in 256 bytes
.text:08053D0F                 push    edi             ; rc4 structure passed in
.text:08053D10                 call    rc4_set_key
.text:08053D15                 add     esp, 0Ch
.text:08053D18
.text:08053D18 loc_8053D18:                            ; CODE XREF: check_challenge_response+91j
.text:08053D18                 push    edi
.text:08053D19                 call    rc4_get_byte
.text:08053D1E                 mov     edx, [ebp+local_rc4_structure]
.text:08053D21                 mov     bl, al          ; gets a byte from the rc4 structure which was passed in (and initialised later on)
.text:08053D23                 mov     ecx, esi        ; esi is a loop counter
.text:08053D25                 and     ecx, 3          ; index into key material
.text:08053D28                 movzx   eax, byte ptr [edx+esi] ; index into the duplicated structure
.text:08053D2C                 mov     edx, [ebp+unknown_data]
.text:08053D2F                 movzx   eax, byte ptr [edx+eax] ; get a byte of the unknown data
.text:08053D33                 pop     edx             ; edx = local rc4 structure
.text:08053D34                 mov     edx, ebx        ; get the byte that was generated via rc4_get_byte
.text:08053D36                 and     edx, 3          ; perform a switch on the last 4 bytes
.text:08053D39                 cmp     edx, 1
.text:08053D3C                 jz      short loc_8053D5A
.text:08053D3E                 jg      short loc_8053D46
.text:08053D40                 test    edx, edx
.text:08053D42                 jz      short loc_8053D52
.text:08053D44                 jmp     short loc_8053D74
.text:08053D46 ; ---------------------------------------------------------------------------
.text:08053D46
.text:08053D46 loc_8053D46:                            ; CODE XREF: check_challenge_response+54j
.text:08053D46                 cmp     edx, 2
.text:08053D49                 jz      short loc_8053D62
.text:08053D4B                 cmp     edx, 3
.text:08053D4E                 jz      short loc_8053D6A
.text:08053D50                 jmp     short loc_8053D74
.text:08053D52 ; ---------------------------------------------------------------------------
.text:08053D52
.text:08053D52 loc_8053D52:                            ; CODE XREF: check_challenge_response+58j
.text:08053D52                 xor     al, bl
.text:08053D54                 add     byte ptr [ebp+ecx+challenge_copy], al
.text:08053D58                 jmp     short loc_8053D74
.text:08053D5A ; ---------------------------------------------------------------------------
.text:08053D5A
.text:08053D5A loc_8053D5A:                            ; CODE XREF: check_challenge_response+52j
.text:08053D5A                 add     al, bl
.text:08053D5C                 xor     byte ptr [ebp+ecx+challenge_copy], al
.text:08053D60                 jmp     short loc_8053D74
.text:08053D62 ; ---------------------------------------------------------------------------
.text:08053D62
.text:08053D62 loc_8053D62:                            ; CODE XREF: check_challenge_response+5Fj
.text:08053D62                 xor     bl, byte ptr [ebp+ecx+challenge_copy]
.text:08053D66                 add     al, bl
.text:08053D68                 jmp     short loc_8053D70
.text:08053D6A ; ---------------------------------------------------------------------------
.text:08053D6A
.text:08053D6A loc_8053D6A:                            ; CODE XREF: check_challenge_response+64j
.text:08053D6A                 add     al, byte ptr [ebp+ecx+challenge_copy]
.text:08053D6E                 xor     al, bl
.text:08053D70
.text:08053D70 loc_8053D70:                            ; CODE XREF: check_challenge_response+7Ej
.text:08053D70                 mov     byte ptr [ebp+ecx+challenge_copy], al
.text:08053D74
.text:08053D74 loc_8053D74:                            ; CODE XREF: check_challenge_response+5Aj
.text:08053D74                                         ; check_challenge_response+66j ...
.text:08053D74                 inc     esi
.text:08053D75                 cmp     esi, 100h
.text:08053D7B                 jnz     short loc_8053D18
.text:08053D7D                 push    [ebp+local_rc4_structure]
.text:08053D80                 call    free_rc4_t_tailcall
.text:08053D85                 mov     eax, [ebp+challenge_copy]
.text:08053D88                 lea     esp, [ebp-0Ch]
.text:08053D8B                 pop     ebx
.text:08053D8C                 or      eax, 80000000h  ; ret |= 0x80000000l
.text:08053D91                 pop     esi
.text:08053D92                 pop     edi
.text:08053D93                 pop     ebp
.text:08053D94                 retn
.text:08053D94 check_challenge_response endp
.text:08053D94
.text:08053D95
.text:08053D95 ; ??????????????? S U B R O U T I N E ???????????????????????????????????????
.text:08053D95
.text:08053D95 ; Attributes: bp-based frame
.text:08053D95
.text:08053D95 sub_8053D95     proc near               ; CODE XREF: load_module_into_kernel+2Cp
.text:08053D95                 push    ebp
.text:08053D96                 dec     edx             ; 16 -> 15
.text:08053D97                 test    eax, edx        ; eax = 0x3c
.text:08053D99                 mov     ebp, esp
.text:08053D9B                 mov     ecx, eax
.text:08053D9D                 jz      short loc_8053DA4
.text:08053D9F                 or      edx, eax
.text:08053DA1                 lea     ecx, [edx+1]
.text:08053DA4
.text:08053DA4 loc_8053DA4:                            ; CODE XREF: sub_8053D95+8j
.text:08053DA4                 pop     ebp
.text:08053DA5                 mov     eax, ecx
.text:08053DA7                 retn
.text:08053DA7 sub_8053D95     endp
.text:08053DA7
.text:08053DA8
.text:08053DA8 ; ??????????????? S U B R O U T I N E ???????????????????????????????????????
.text:08053DA8
.text:08053DA8 ; Attributes: bp-based frame
.text:08053DA8
Example: Loader: rc4_init_key_encrypt
.text:08053BFD rc4_init_key_encrypt proc near          ; CODE XREF: load_module_into_kernel+F9p
.text:08053BFD
.text:08053BFD var_8           = dword ptr -8
.text:08053BFD rc4_t           = dword ptr  8
.text:08053BFD start_of_module_data= dword ptr  0Ch
.text:08053BFD
.text:08053BFD                 push    ebp
.text:08053BFE                 mov     ebp, esp
.text:08053C00                 push    esi
.text:08053C01                 mov     esi, [ebp+rc4_t]
.text:08053C04                 push    ebx
.text:08053C05                 mov     ebx, [ebp+start_of_module_data]
.text:08053C08                 push    esi
.text:08053C09                 call    init_rc4_t
.text:08053C0E                 push    dword ptr [ebx] ; push the encryption key
.text:08053C10                 add     ebx, 4          ; move the data along
.text:08053C13                 push    esi             ; push the rc4_t context
.text:08053C14                 call    rc4_set_key
.text:08053C19                 push    1000            ; length
.text:08053C1E                 push    esi             ; rc4 structure
.text:08053C1F                 call    rc4_prevent_weak_bytes
.text:08053C24                 push    10h             ; size
.text:08053C26                 push    ebx             ; data
.text:08053C27                 push    esi             ; rc4 structure
.text:08053C28                 call    rc4_crypt       ; encrypts 16 bytes that go to the kernel module
.text:08053C2D                 lea     esp, [ebp-8]
.text:08053C30                 pop     ebx
.text:08053C31                 pop     esi
.text:08053C32                 pop     ebp
.text:08053C33                 retn
.text:08053C33 rc4_init_key_encrypt endp

In general, the feeling is that the loader binary tries to ensure it's running on the same kernel the module was written for, because it is using hard coded offsets to the kernel data. I was disappointed that this module the loader binary inserted didn't appear to be malicious.

To bypass this somewhat artifical restriction the binary imposes is somewhat easily done. The previous code written to hook the create_module and init_module code, can be modified to:

xor eax, eax
inc eax
ret

to avoid this restriction. I haven't tested this, but it should work :p

Continuing the analysis of loader

Before we can continue the analysis of the fileman binary, we still need to get the loader binary running. Running a strace reveals the current issue with loader:

[box] # LD_LIBRARY_PATH=/lib_mikro strace -f ./loader
execve("./loader", ["./loader"], [/* 15 vars */]) = 0
uname({sys="Linux", node="debian", ...}) = 0
...
[pid  1727] rt_sigaction(SIGSEGV, {0x4002c8ca, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x400774c0}, NULL, 8) = 0
[pid  1727] rt_sigaction(SIGILL, {0x4002c8ca, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x400774c0}, NULL, 8) = 0
[pid  1727] rt_sigaction(SIGABRT, {0x4002c8ca, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x400774c0}, NULL, 8) = 0
[pid  1727] rt_sigaction(SIGBUS, {0x4002c8ca, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x400774c0}, NULL, 8) = 0
[pid  1727] rt_sigaction(SIGFPE, {0x4002c8ca, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x400774c0}, NULL, 8) = 0
[pid  1727] gettimeofday({1189098093, 745332}, NULL) = 0
[pid  1727] create_module("qwink", 1430) = 0xc8833000
[pid  1727] init_module(0x80553fc, 134587376, umovestr: Input/output error
0x7c) = 0
[pid  1726] waitpid(1727, Process 1726 suspended
 <unfinished ...>
[pid  1727] delete_module("qwink")      = 0
...
[pid  1727] open("/dev/panics", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  1727] open("/proc/cmdline", O_RDONLY) = 3
[pid  1727] fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid  1727] mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40170000
[pid  1727] read(3, "root=/dev/hda1 ro single\n", 4096) = 25
[pid  1727] close(3)                    = 0
[pid  1727] munmap(0x40170000, 4096)    = 0
[pid  1727] exit_group(1)               = ?

It appears some functionality that is provided elsewhere needs to be initialised first.

Having a look in IDA at loader (cross referencing on /proc/cmdline), indicates that the /proc/cmdline must contain MBR= followed by a pattern that matches %x.

.text:0804DBD4                 push    offset name     ; "/proc/cmdline"
.text:0804DBD9                 call    _fopen          ; open the file, read only
.text:0804DBDE                 pop     ebx
.text:0804DBDF                 mov     esi, eax
.text:0804DBE1                 test    esi, esi
.text:0804DBE3                 pop     eax
.text:0804DBE4                 jz      short loc_804DC34 ; if opening the file failed, jump down to exit(1)
.text:0804DBE6                 push    esi             ; FILE *
.text:0804DBE7                 lea     ebx, [ebp+var_408]
.text:0804DBED                 push    1024            ; int
.text:0804DBF2                 push    ebx             ; char *
.text:0804DBF3                 call    _fgets          ; read in 0x400 bytes
.text:0804DBF8                 push    esi             ; FILE *
.text:0804DBF9                 call    _fclose
.text:0804DBFE                 push    offset aMbr     ; "MBR="
.text:0804DC03                 push    ebx             ; char *
.text:0804DC04                 call    _strstr         ; look for MBR= in the string
.text:0804DC09                 add     esp, 18h
.text:0804DC0C                 test    eax, eax
.text:0804DC0E                 mov     edx, eax
.text:0804DC10                 jz      short loc_804DC34 ; don't find it, exit out
.text:0804DC12                 mov     [ebp+var_40C], 0 ; initialse the number read in
.text:0804DC1C                 lea     eax, [ebp+var_40C] ; load the address in
.text:0804DC22                 push    eax
.text:0804DC23                 push    offset aMbrX    ; "MBR=%x"
.text:0804DC28                 push    edx             ; char *
.text:0804DC29                 call    _sscanf         ; parse the string
.text:0804DC2E                 add     esp, 0Ch
.text:0804DC31                 dec     eax
.text:0804DC32                 jz      short loc_804DC3B
.text:0804DC34
.text:0804DC34 loc_804DC34:                            ; CODE XREF: sub_804DBC4+20j
.text:0804DC34                                         ; sub_804DBC4+4Cj
.text:0804DC34                 push    1               ; status
.text:0804DC36                 call    _exit
.text:0804DC3B ; ---------------------------------------------

Looking up information on Master boot records (MBR), indicates that the MBR starts at 0x1BE. However, looking where the code is called, there is a comparision to see if it's above a certain size. Restarting with an updated kernel line with MBR=0, lets the binary run, and listen on a network socket.

Summary

This article has shown usage QEMU with GDB to debug Linux kernel modules, along with static disassembly provided by IDA Pro. It has covered analysing an obscured kernel module that was bound tightly to a specific, vendor, kernel.