In the last post I brought up an Ampere Altra as a headless ARM64 host. Now I put it to work. This is my setup for fuzzing an ARM64 Linux kernel with syzkaller under QEMU, all of it native. The big win over doing this from an x86 box is that nothing gets cross-compiled. The kernel, syzkaller and the guest all build straight on the host.

Prerequisites

  • ARM64 Linux host, tested on Ubuntu 24.04
  • KVM support, check with ls /dev/kvm
  • GCC 13+, tested with gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0

Dependencies

sudo apt update
sudo apt install -y git make gcc flex bison libssl-dev libelf-dev bc \
    qemu-system-arm debootstrap libgmp-dev libmpfr-dev

Go

wget https://dl.google.com/go/go1.23.6.linux-arm64.tar.gz
sudo tar -C /usr/local -xzf go1.23.6.linux-arm64.tar.gz
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
echo 'export GOPATH=$HOME/go' >> ~/.bashrc
source ~/.bashrc
go version

Build syzkaller

git clone https://github.com/google/syzkaller.git
cd syzkaller
make TARGETOS=linux TARGETARCH=arm64

The binaries land in bin/linux_arm64/.

Build the kernel

Clone and start from defconfig:

git clone --depth=1 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git upstream-vanilla
cd upstream-vanilla
make defconfig

Now turn on what syzkaller needs. I use scripts/config rather than hand editing .config:

# Coverage
scripts/config -e KCOV
scripts/config -e KCOV_INSTRUMENT_ALL
scripts/config -e KCOV_ENABLE_COMPARISONS

# Debug info
scripts/config -e DEBUG_INFO
scripts/config -e DEBUG_INFO_DWARF4
scripts/config -e DEBUG_FS

# KASAN, software tag-based is faster on ARM64 than generic
scripts/config -e KASAN
scripts/config -e KASAN_SW_TAGS
scripts/config -d KASAN_GENERIC

# If your CPU has MTE you can use hardware tag-based KASAN, which is faster still
# scripts/config -e KASAN_HW_TAGS

# Virtio, required for QEMU and faster than emulated hardware on ARM64
scripts/config -e VIRTIO
scripts/config -e VIRTIO_PCI
scripts/config -e VIRTIO_BLK
scripts/config -e VIRTIO_NET
scripts/config -e VIRTIO_MMIO

# 9P for syzkaller file transfer
scripts/config -e NET_9P
scripts/config -e NET_9P_VIRTIO

# Networking
scripts/config -e CONFIGFS_FS
scripts/config -e SECURITYFS

# Optional, disable KASLR for easier debugging
scripts/config -d RANDOMIZE_BASE

Regenerate and build:

make olddefconfig
make -j$(nproc)

The image is at arch/arm64/boot/Image. Note that it is Image, not the bzImage you get on x86.

Create the disk image

syzkaller ships a helper for this:

cd ~/syzkaller/tools
./create-image.sh

That gives you a Debian image with SSH set up:

  • trixie.img, the disk image
  • trixie.id_rsa, the SSH key

Move them somewhere sensible:

mkdir -p ~/wdir/trixie
mv trixie.img trixie.id_rsa ~/wdir/trixie/

Test the VM by hand first

Before letting syzkaller drive, make sure a single VM boots. This is where the ARM64 differences bite:

#!/bin/bash
KERNEL=~/wdir/upstream-vanilla
IMAGE=~/wdir/trixie

qemu-system-aarch64 \
    -m 2G \
    -smp 2 \
    -cpu host \
    -machine virt,gic-version=3 \
    -kernel $KERNEL/arch/arm64/boot/Image \
    -append "console=ttyAMA0 root=/dev/vda earlyprintk=serial net.ifnames=0" \
    -drive file=$IMAGE/trixie.img,format=raw,if=virtio \
    -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \
    -net nic,model=virtio \
    -enable-kvm \
    -nographic \
    -pidfile vm.pid

What changes from an x86 command line:

  • -cpu host and -machine virt,gic-version=3 are needed for KVM
  • console=ttyAMA0 instead of console=ttyS0
  • if=virtio for the disk, faster than emulated IDE or SATA here
  • model=virtio for the NIC, faster than e1000
  • the kernel is Image, not bzImage

Then check SSH works:

ssh -i ~/wdir/trixie/trixie.id_rsa -p 10021 root@localhost

syzkaller config

Create arm64.cfg:

{
    "target": "linux/arm64",
    "http": "0.0.0.0:56700",
    "workdir": "/mnt/ramdisk/workdir",
    "kernel_obj": "/home/user/wdir/upstream-vanilla",
    "image": "/mnt/ramdisk/workdir/trixie/trixie.img",
    "sshkey": "/home/user/wdir/trixie/trixie.id_rsa",
    "syzkaller": "/home/user/syzkaller",
    "reproduce": true,
    "procs": 4,
    "type": "qemu",
    "vm": {
        "count": 52,
        "kernel": "/home/user/wdir/upstream-vanilla/arch/arm64/boot/Image",
        "cpu": 2,
        "mem": 2048,
        "qemu_args": "-machine virt,gic-version=3 -cpu host -enable-kvm",
        "cmdline": "console=ttyAMA0 root=/dev/vda earlyprintk=serial net.ifnames=0"
    }
}

This is where the 128 core machine pays off. To size the fleet:

  • count, number of VMs, roughly (total_cores / cpu_per_vm) - overhead
  • cpu, vCPUs per VM
  • procs, fuzzing processes per VM, roughly match cpu
  • mem, RAM per VM in MB

On the 128 core, 256GB box that works out to:

  • 52 VMs times 2 CPUs is 104 cores
  • 52 VMs times 2GB is 104GB of RAM
  • 80GB ramdisk for the workdir
  • the rest left as headroom for the host

Point the workdir and image paths at the ramdisk from the previous post so this heavy I/O never touches the SSD.

Run it

cd ~/syzkaller
bin/syz-manager -config=arm64.cfg

The dashboard is at http://<server-ip>:56700.

KASAN variants on ARM64

ARM64 gives you a few KASAN modes. Fastest first:

  1. Hardware tag-based, CONFIG_KASAN_HW_TAGS, needs ARMv8.5-A MTE
  2. Software tag-based, CONFIG_KASAN_SW_TAGS, my default on ARM64
  3. Generic, CONFIG_KASAN_GENERIC, slowest but most compatible

Check for MTE:

cat /proc/cpuinfo | grep -i mte

If MTE is there, use HW_TAGS. Otherwise SW_TAGS.

Extracting syscalls

Only needed if you touch the descriptions:

cd ~/syzkaller
bin/syz-extract -sourcedir=../upstream-vanilla -arch=arm64 <syscall_file.txt>

USB fuzzing

To fuzz the USB subsystem, add to the kernel config:

scripts/config -e USB
scripts/config -e USB_GADGET
scripts/config -e USB_DUMMY_HCD
scripts/config -e USB_RAW_GADGET
scripts/config -e SND_USB_AUDIO

And in the syzkaller config:

"enable_syscalls": [
    "syz_usb*"
],
"cmdline": "console=ttyAMA0 root=/dev/vda dummy_hcd.num=4 net.ifnames=0"

Troubleshooting

VM fails to boot, or “failed to read from qemu: EOF”. Run the manual QEMU command above, confirm virtio is built in with grep VIRTIO .config, and check KVM permissions with ls -la /dev/kvm.

No coverage in your target subsystem. Make sure it is built into the kernel and not a module, confirm KCOV with grep KCOV .config, and check the syscalls are listed under enable_syscalls.

syz-extract complains about KASAN_SHADOW_SCALE_SHIFT. A known issue on bleeding-edge kernels, 6.17 and up. Use a stable kernel, 6.12 LTS works well, update syzkaller, or just test anyway since the extract errors may not matter.

Slow boot waiting on entropy. ARM64 QEMU can stall here. Install haveged in the guest image or add random.trust_cpu=on to the kernel cmdline.

KVM permission denied. Add yourself to the kvm group:

sudo usermod -aG kvm $USER
newgrp kvm

That is a full ARM64 fuzzing pipeline with no cross-compilation anywhere. Native host, native kernel, native syzkaller, and a VM count that actually uses the hardware.