Most of my kernel fuzzing runs on ARM64 targets, and cross-compiling everything from an x86 box gets old fast. I wanted a native ARM64 machine with enough cores to run a large fleet of QEMU VMs under syzkaller. The answer was an Ampere Altra on an ASRock Rack board. This post covers taking it from a bare board to a running Ubuntu host, all of it headless over the BMC.

The hardware

  • Motherboard: ASRock Rack ALTRAD8UD-1L2T
  • CPU: Ampere Altra, 128 cores, Neoverse-N1
  • RAM: 256GB, 8x 32GB DDR4 RDIMMs
  • Storage: 2x 1TB NVMe, one for Ubuntu and one kept for Windows
  • GPU: NVIDIA RTX 5060 Ti

128 cores and 256GB is the whole point. It lets me run around 50 VMs with 2 vCPUs each and still leave headroom for the host.

Getting into the BMC without a monitor

The board has no VGA that I care about, so everything goes through the BMC. There is a debug serial header on the bottom-left of the board. Pinout, left to right:

  • Rx, wire this to your adapter Tx
  • Tx, wire this to your adapter Rx
  • 3.3V
  • NC, no pin
  • GND

Use a 3.3V USB-TTL adapter at 115200 8N1.

Default BMC credentials:

  • Username: root
  • Password: 0penBmc (that is a zero, not the letter O)

The BMC is a DHCP client on the MGMT port. Once it boots, find its address:

ip addr

The web UI then lives at https://<BMC_IP>. There are also SSH consoles on a few ports, which is what I actually use day to day:

  • Port 2200: Console
  • Port 2201: TF-A console
  • Port 2202: SCP console, SMpro and PMpro logs
  • Port 2203: Host console, UEFI and OS

Firmware first

I moved from BMC 2.07 / BIOS 2.06 to BMC 3.06 / BIOS 3.06 before doing anything else. Grab both from the ASRock Rack Beta Zone for the board.

The order matters. BMC first, then BIOS.

  1. Upload the BMC .tar in the web UI under Operations, Firmware
  2. Wait for the BMC to reboot, around 5 minutes
  3. Upload the BIOS .tar
  4. Power cycle the host so the BIOS takes effect

Installing Ubuntu 24.04 over virtual media

Write the arm64 server ISO to a USB stick, or skip the stick entirely and mount it as virtual media. I did the latter.

wget https://cdimage.ubuntu.com/releases/24.04/release/ubuntu-24.04-live-server-arm64.iso

In the web UI, go to Operations, Virtual Media, add the ISO and click Start. Then power cycle the host from the BMC console:

obmcutil poweroff
obmcutil poweron

Watch the host console on port 2203 and spam ESC for the boot menu if it does not pick the ISO on its own.

A few install choices that saved me pain:

  • Skip network during install and configure it later
  • Do not pick the HWE kernel, it wants network
  • Enable the OpenSSH server
  • Install onto one NVMe and leave the other alone

Network after install

Since I skipped it during setup, I drop in a small netplan config:

sudo nano /etc/netplan/01-netcfg.yaml

network:
  version: 2
  ethernets:
    enP3p3s0f0:
      dhcp4: true

sudo netplan apply

NVIDIA driver

sudo apt update
sudo apt install nvidia-driver-580-open
sudo reboot
nvidia-smi

Ramdisk for syzkaller

Syzkaller does heavy I/O on its workdir and I would rather not grind an SSD to dust. I keep the workdir on a tmpfs ramdisk and back it up on a schedule and around reboots.

Mount it and make room for the workdir:

sudo mount -t tmpfs -o size=80G tmpfs /mnt/ramdisk
mkdir -p /mnt/ramdisk/workdir

The backup script at /usr/local/bin/backup-ramdisk.sh:

#!/bin/bash
rsync -a --no-owner --no-group /mnt/ramdisk/workdir/ /home/user/wdir/syzkaller-ramdisk/workdir/
echo "$(date): Backup completed" >> /home/user/ramdisk-backup.log

sudo chmod +x /usr/local/bin/backup-ramdisk.sh

A cron job every 6 hours, via sudo crontab -e:

0 */6 * * * /usr/local/bin/backup-ramdisk.sh

Since a ramdisk is gone after a reboot, I restore it on boot with a service at /etc/systemd/system/ramdisk-restore.service:

[Unit]
Description=Restore ramdisk after boot
After=local-fs.target

[Service]
Type=oneshot
ExecStartPre=/bin/mount -t tmpfs -o size=80G tmpfs /mnt/ramdisk
ExecStart=/usr/bin/rsync -a /home/user/wdir/syzkaller-ramdisk/workdir/ /mnt/ramdisk/workdir/
ExecStartPost=/usr/bin/rsync -a /home/user/wdir/trixie/ /mnt/ramdisk/workdir/trixie/

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable ramdisk-restore.service

And a matching one that flushes the ramdisk back to disk on shutdown, at /etc/systemd/system/ramdisk-backup.service:

[Unit]
Description=Backup ramdisk before shutdown
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup-ramdisk.sh

[Install]
WantedBy=halt.target reboot.target shutdown.target

sudo systemctl daemon-reload
sudo systemctl enable ramdisk-backup.service

Fix ownership after boot and make sure the user can touch KVM:

sudo chown -R $USER:$USER /mnt/ramdisk
sudo usermod -aG kvm $USER

Commands I keep reaching for

Power control from the BMC:

obmcutil poweroff
obmcutil poweron
obmcutil state

Temperature, because 128 cores under load get warm:

sudo apt install freeipmi-tools
sudo ipmi-sensors | grep -i temp

Ramdisk usage:

df -h /mnt/ramdisk

That is the machine ready. The next post covers building the ARM64 kernel and running syzkaller across the VM fleet natively, no cross-compilation involved.