Troubleshooting

Quick Diagnostics

Run these commands to quickly diagnose issues:

Check Firecracker process
ps aux | grep firecracker

Check KVM availability
ls -la /dev/kvm

Check vhost_vsock
ls -la /dev/vhost-vsock

Check vsock socket exists
ls -la /tmp/fc-*.vsock

View Firecracker logs
journalctl -u firecracker -f

Common Issues

VM Won't Start

KVM Error

Error:

Error message
Error creating VM: cannot create KVM instance

Cause: KVM module not loaded or no access to /dev/kvm

Solution:

Load KVM module
sudo modprobe kvm
sudo modprobe kvm_intel  # or kvm_amd

Check permissions
ls -la /dev/kvm

Add user to kvm group
sudo usermod -aG kvm $USER

Vhost Vsock Error

Error:

Error message
Error configuring vsock: cannot open /dev/vhost-vsock

Cause: vhost_vsock module not loaded

Solution:

Load vhost_vsock module
sudo modprobe vhost_vsock

Verify
ls -la /dev/vhost-vsock

Make permanent
echo "vhost_vsock" | sudo tee -a /etc/modules

Missing Kernel

Error:

Error message
Error: cannot open kernel file: /srv/firecracker/vmlinux

Solution:

Download kernel
task s3:download

Verify
ls -la /srv/firecracker/images/vmlinux

Missing Rootfs

Error:

Error message
Error: cannot open drive file: /srv/firecracker/rootfs.ext4

Solution:

Download rootfs images
task s3:download

Verify
ls -la /srv/firecracker/images/rootfs-*.ext4

Socket In Use

Error:

Error message
Error: cannot bind to socket /tmp/fc-12345.sock: Address already in use

Solution:

Find and kill existing Firecracker process
ps aux | grep firecracker
kill -9 <PID>

Remove stale sockets
rm -f /tmp/fc-*.sock /tmp/fc-*.vsock

Connection Issues

Socket Not Found

Error:

Error message
dial unix /tmp/fc-12345.vsock: no such file or directory

Causes:

VM not started
vsock not configured
Wrong socket path

Solution:

Verify VM is running
ps aux | grep firecracker

Check vsock was configured
curl --unix-socket /tmp/fc-12345.sock \
  http://localhost/vsock 2>/dev/null | jq

Verify socket exists
ls -la /tmp/fc-12345.vsock

Connection Refused

Error:

Error message
NO 5000

Causes:

infra.operator (guest mode) not started
Wrong port number
VM not fully booted

Solution:

Wait longer for VM to boot
sleep 5

Timeout Error

Error:

Error message
read: deadline exceeded

Causes:

infra.operator (guest mode) crashed
Code execution taking too long
Deadlock in guest

Solution:

Increase timeout
job := Job{
    Timeout: 60,  // Increase from default
}

Check if VM is still running
curl --unix-socket /tmp/fc.sock http://localhost/

Execution Errors

Execution Timeout

Error:

Response
{
  "exit_code": -1,
  "error": "execution timeout"
}

Causes:

Infinite loop in code
Timeout too short
Heavy computation

Solution:

Increase execution timeout
job := Job{
    Code:    "heavy_computation()",
    Timeout: 60,  // seconds
}

Out of Memory

Error:

Response
{
  "exit_code": 137,
  "stderr": "Killed"
}

Cause: Code exceeded memory limit

Solution:

Increase VM memory
{
  "vcpu_count": 1,
  "mem_size_mib": 1024
}

Command Not Found

Error:

Response
{
  "exit_code": 127,
  "stderr": "python3: command not found"
}

Cause: Wrong rootfs or language not installed

Solution:

Verify using correct rootfs
ls -la /srv/firecracker/images/rootfs-python.ext4

Mount and check contents
sudo mount -o loop rootfs-python.ext4 /mnt
ls /mnt/usr/bin/python3
sudo umount /mnt

Permission Denied

Error:

Response
{
  "exit_code": 126,
  "stderr": "permission denied"
}

Cause: Script file not executable

Solution:

Set executable permission
err := os.WriteFile(scriptPath, []byte(code), 0755)

Build Errors

Package Not Found

Error:

Error message
cannot find package syscall (using -tags vsock)

Solution:

Build with correct tags
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 \
  go build -o bin/infra.operator-linux ./cmd/infra.operator

Wrong Architecture

Error:

Error message
cannot execute binary file: Exec format error

Cause: Wrong architecture

Solution:

Build for x86_64
GOOS=linux GOARCH=amd64 go build -o bin/infra.operator-linux ./cmd/infra.operator

Build for ARM64
GOOS=linux GOARCH=arm64 go build -o bin/infra.operator-arm64 ./cmd/infra.operator

Verify binary
file bin/infra.operator-linux

S3 Issues

Access Denied

Error:

Error message
An error occurred (AccessDenied) when calling the GetObject operation

Solution:

Check AWS credentials
aws sts get-caller-identity

Verify bucket policy allows access
aws s3 ls s3://llm-infra-operator-rootfs/

Check IAM policy
aws iam get-user-policy --user-name <user> --policy-name <policy>

Bucket Not Found

Error:

Error message
An error occurred (NoSuchBucket) when calling the ListObjects operation

Solution:

Create bucket
aws s3 mb s3://llm-infra-operator-rootfs --region us-east-1

Verify
aws s3 ls

Debug Commands

Check VM Status

Check VM via Firecracker API
curl --unix-socket /tmp/fc.sock http://localhost/ | jq
curl --unix-socket /tmp/fc.sock http://localhost/machine-config | jq
curl --unix-socket /tmp/fc.sock http://localhost/vsock | jq

Monitor Serial Console

Connect to serial console
screen /dev/pts/X

Test vsock Connection

Manual vsock test
exec 3<>/dev/tcp/localhost/5000 || echo "Failed"

Test with netcat
echo -e "CONNECT 5000\n" | nc -U /tmp/fc-12345.vsock

Check Guest Status

Send test job via vsock
{
  echo "CONNECT 5000"
  sleep 0.5
  echo -ne '\x00\x00\x00\x2B{"trace_id":"test","lang":"bash","code":"echo OK","timeout":10}'
} | nc -U /tmp/fc-12345.vsock

Log Locations

Component	Log Location
infra.operator (host)	stdout / systemd journal
Firecracker	stderr / serial console
infra.operator (guest)	/var/log/infra.operator.log (in VM)
System	/var/log/syslog

Enable Debug Logging

Enable debug in infra.operator
log.SetLevel(log.DebugLevel)

Firecracker Debug Mode

Start Firecracker with debug logging
firecracker --api-sock /tmp/fc.sock \
  --log-path /var/log/firecracker.log \
  --level Debug

Health Checks

Automated Health Check Script

Check KVM
if [ ! -c /dev/kvm ]; then
    echo "FAIL: /dev/kvm not available"
    exit 1
fi

Check vhost_vsock
if [ ! -c /dev/vhost-vsock ]; then
    echo "FAIL: /dev/vhost-vsock not available"
    exit 1
fi

Check kernel
if [ ! -f /srv/firecracker/images/vmlinux ]; then
    echo "FAIL: kernel not found"
    exit 1
fi

Check rootfs images
for lang in python nodejs go rust bash; do
    if [ ! -f /srv/firecracker/images/rootfs-${lang}.ext4 ]; then
        echo "FAIL: rootfs-${lang}.ext4 not found"
        exit 1
    fi
done

Check disk space
FREE=$(df /srv -m | tail -1 | awk '{print $4}')
if [ "$FREE" -lt 5000 ]; then
    echo "WARN: Low disk space: ${FREE}MB"
fi
echo "OK: All checks passed"

Run Health Check

Run health check
task aws:healthcheck

Quick Fixes Cheatsheet

Problem	Quick Fix
VM won't start	`sudo modprobe kvm kvm_intel vhost_vsock`
Socket in use	`rm -f /tmp/fc-.sock /tmp/fc-.vsock`
vsock refused	Wait longer, check infra.operator service
Timeout	Increase timeout, check for loops
OOM	Increase `mem_size_mib`
Wrong rootfs	Check language → rootfs mapping
S3 access denied	Check AWS credentials
Binary format	Cross-compile with correct GOARCH

Quick Diagnostics​

Common Issues​

VM Won't Start​

KVM Error​

Vhost Vsock Error​

Missing Kernel​

Missing Rootfs​

Socket In Use​

Connection Issues​

Socket Not Found​

Connection Refused​

Timeout Error​

Execution Errors​

Execution Timeout​

Out of Memory​

Command Not Found​

Permission Denied​

Build Errors​

Package Not Found​

Wrong Architecture​

S3 Issues​

Access Denied​

Bucket Not Found​

Debug Commands​

Check VM Status​

Monitor Serial Console​

Test vsock Connection​

Check Guest Status​

Log Locations​

Enable Debug Logging​

Firecracker Debug Mode​

Health Checks​

Automated Health Check Script​

Run Health Check​

Quick Fixes Cheatsheet​

Quick Diagnostics

Common Issues

VM Won't Start

KVM Error

Vhost Vsock Error

Missing Kernel

Missing Rootfs

Socket In Use

Connection Issues

Socket Not Found

Connection Refused

Timeout Error

Execution Errors

Execution Timeout

Out of Memory

Command Not Found

Permission Denied

Build Errors

Package Not Found

Wrong Architecture

S3 Issues

Access Denied

Bucket Not Found

Debug Commands

Check VM Status

Monitor Serial Console

Test vsock Connection

Check Guest Status

Log Locations

Enable Debug Logging

Firecracker Debug Mode

Health Checks

Automated Health Check Script

Run Health Check

Quick Fixes Cheatsheet