This question became longer than I expected. Since BPF is very new, it's possible there is no precise answer.
I am trying to load a "hello world" eBPF program onto the XDP attachment point. I want to do this using raw syscalls and without using libbpf, bpftool, xdp-loader, etc.
My kernel release is 6.5.0-9-generic.
The BPF program is:
#include <linux/types.h>
#include <bpf/bpf_helpers.h>
#include <linux/bpf.h>
#include <linux/version.h>
SEC("xdp")
int xdp_prog_simple(struct xdp_md *ctx)
{
bpf_printk("In xdp_prog_simple\n");
return XDP_PASS;
}
char _license[] SEC("license") = "GPL";
__u32 _version SEC("version") = LINUX_VERSION_CODE;
And compiled with:
clang -O2 -g -Wall -target bpf -c bpf.c -o bpf.o
And the output of llvm-objdump --no-show-raw-insn --dr bpf.o is:
Disassembly of section xdp:
0000000000000000 <xdp_prog_simple>:
0: r1 = 0x0 ll
0000000000000000: R_BPF_64_64 .rodata
2: r2 = 0x14
3: call 0x6
4: r0 = 0x2
5: exit
I'm trying to load this bpf.o file using a Rust program. I extract the bytes from the xdp section and reference these in the attribute passed to the BPF_PROG_LOAD command of the SYS_bpf syscall:
(This code is not the problem. Including it for context, and because I haven't seen many Rust examples on the internet, so it could be helpful for future readers)
use std::mem::size_of;
use crate::ffi::bpf;
use crate::ffi::syscall::check_err;
use crate::Error;
use object::{Object, ObjectSection};
const LOG_BUF_SIZE: usize = 65536;
static mut BPF_LOG_BUF: [u8; LOG_BUF_SIZE] = [0; LOG_BUF_SIZE];
pub fn load_xdp_program(elf: &[u8], section: &str) -> Result<i32, Error> {
let obj = object::File::parse(elf).map_err(|err| Error::Boxed(Box::new(err)))?;
// Extract bytes from the program section ("xdp" in this example)
let text = obj
.section_by_name(section)
.ok_or_else(|| Error::NotFound("program section not found"))?
.data()
.map_err(|err| Error::Boxed(Box::new(err)))?;
// Extract the license bytes
let license = obj
.section_by_name("license")
.ok_or_else(|| Error::NotFound("section 'license' not found"))?
.data()
.map_err(|err| Error::Boxed(Box::new(err)))?;
// Extract the kernel version bytes
let version = {
let data: [u8; 4] = obj
.section_by_name("version")
.ok_or_else(|| Error::NotFound("section 'version' not found"))?
.data()
.map_err(|err| Error::Boxed(Box::new(err)))?
.try_into()
.map_err(|err| Error::Boxed(Box::new(err)))?;
u32::from_le_bytes(data)
};
let attr = &bpf::ProgramAttr {
prog_type: XDP_PROG_TYPE,
insn_cnt: (text.len() / size_of::<bpf::Insn>()) as u32,
insns: ptr_to_u64(text),
license: ptr_to_u64(license),
log_level: 1,
log_size: LOG_BUF_SIZE as u32,
log_buf: ptr_to_u64(unsafe { &BPF_LOG_BUF }),
kern_version: version,
prog_flags: 0,
prog_name: *b"xdp_prog_simple\0",
prog_ifindex: 0,
expected_attach_type: 37, // BPF_XDP
prog_btf_fd: 0,
func_info_rec_size: 0,
func_info: 0,
func_info_cnt: 0,
line_info_rec_size: 0,
line_info: 0,
line_info_cnt: 0,
attach_btf_id: 0,
attach: bpf::AttachFd { object_fd: 0 },
core_relo_cnt: 0,
fd_array: 0,
core_relos: 0,
core_relo_rec_size: 0,
};
let prog_fd = check_err(unsafe {
libc::syscall(
libc::SYS_bpf,
bpf::cmd::PROG_LOAD,
attr as *const _ as *const libc::c_void,
size_of::<bpf::BpfAttr>(),
) as i32
});
match prog_fd {
Ok(fd) => Ok(fd),
Err(err) => {
let msg = String::from_utf8_lossy(unsafe { &BPF_LOG_BUF });
eprintln!("Failed to load BPF program. Log buffer:\n\n{msg}");
Err(err)
}
}
}
I build and call this as sudo to make sure it has root permissions:
cargo build --release --bin xdp-loader
sudo ./target/release/xdp-loader bpf.o
Which returns an error from the log buffer:
0: R1=ctx(off=0,imm=0) R10=fp0
0: (18) r1 = 0x0 ; R1_w=0
2: (b7) r2 = 20 ; R2_w=20
3: (85) call bpf_trace_printk#6
R1 type=scalar expected=fp, pkt, pkt_meta, map_key, map_value, mem, ringbuf_mem, buf, trusted_ptr_
processed 3 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
And the BPF syscall return with error code 13: permission denied.
This seems to be implying that the register r1 is a type scalar when it should be type fp. This is confusing to me. As far as I understand, r1 should be carrying the address to context structure struct xdp_md *ctx, according to these docs. I am not performing any pointer arithmetic, so it should not be converted to a scalar.
I think this is because struct xdp_md *ctx is not used, so it is compiled out. It is instead used for BPF calling convention to invoke the bpf_printk helper, which is actually a macro that is replaced by bpf_trace_printk.
When invoked, register r1 is used to point at the "In xdp_prog_simple\n" string. It loaded an immediate 64-bit 0x0 address which is a zero-offset address into the .rodata section, which contains that string. This is the first argument to bpf_printk. Register r2 is also loaded the immediate value 20 because that is the length of the "In xdp_prog_simple\n" string – it's the second argument to bpf_trace_printk.
But this doesn't explain the error:
R1 type=scalar expected=fp, pkt, pkt_meta, map_key, map_value, mem, ringbuf_mem, buf, trusted_ptr_
It expect r1 to be a frame pointer, but it's a scalar.
I think this is because while I'm loading the xdp section containing the program byte code, I'm not loading the .rodata section, so this doesn't point anywhere, and the verifier interprets the 0x0 address as a scalar.
It seems like there needs to be an intermediate linking/loading step in order to make .rodata accessible by the BPF JIT. This is corroborated by this SO answer, which implies that libbpf does it automatically. If I rewrite my code:
SEC("xdp")
int xdp_prog_simple(struct xdp_md *ctx)
{
char msg[] = "In xdp_prog_simple\n";
bpf_trace_printk(msg, sizeof(msg));
return XDP_PASS;
}
The string is stored inline, and not in a different section:
Disassembly of section xdp:
0000000000000000 <xdp_prog_simple>:
0: r1 = 0xa656c
1: *(u32 *)(r10 - 0x8) = r1
2: r1 = 0x706d69735f676f72 ll
4: *(u64 *)(r10 - 0x10) = r1
5: r1 = 0x705f706478206e49 ll
7: *(u64 *)(r10 - 0x18) = r1
8: r1 = r10
9: r1 += -0x18
10: r2 = 0x14
11: call 0x6
12: r0 = 0x2
13: exit
And the program loads successfully!
On my kernel version, bpf_printk doesn't seem to do this restructure automatically:
/* Helper macro to print out debug messages */
#define bpf_printk(fmt, args...) ___bpf_pick_printk(args)(fmt, ##args)
My question: how can I do this automatically without need to split the declaration of the string and invocation of bpf_trace_printk? I believe I need to somehow link/load the string in the .rodata section as identified by the R_BPF_64_64 relocation entry in the disassembly:
(Snippet from earlier)
0000000000000000 <xdp_prog_simple>:
0: r1 = 0x0 ll
0000000000000000: R_BPF_64_64 .rodata
So that it's stored inline with the bytecode. Is that correct? If so, how do I do this?
(My thinking is that I'll need scan the ELF file for relocation entries and use these to locate items in the .rodata section somehow, and then insert them into the bytecode directly)
I understand BPF is rather new and has sharp edges. Happy with any answer that points me in the right direction.