mmap query on linux platform

Question

On Linux machine, trying to write driver and trying to map some kernel memory to application for performance gains. Checking driver implementation for mmap online, finding different varieties of implementation. As per man pages, mmap - creates new mapping in virtual address space of calling process.

1) Who allocates physical address space during mmap calling? kernel or device driver?

seen following varieties of driver mmap implementation.

a) driver creates contiguous physical kernel memory and maps it with process address space.

static int driver_mmap(struct file *filp, struct vm_area_struct *vma)
{
    unsigned long size = vma->vm_end - vma->vm_start;

    pos = kmalloc(size); //allocate contiguous physical memory.
    while (size > 0) {
        unsigned long pfn;
        pfn = virt_to_phys((void *) pos) >> PAGE_SHIFT; // Get Page frame number
        if (remap_pfn_range(vma, start, pfn, PAGE_SIZE, PAGE_SHARED)) // creates mapping
            return -EAGAIN;
        start += PAGE_SIZE;
        pos += PAGE_SIZE;
        size -= PAGE_SIZE;
    }
}

b) driver creates virtual kernel memory and maps it with process address space.

static struct vm_operations_struct dr_vm_ops = {
    .open = dr_vma_open,
    .close = dr_vma_close,
};
static int driver_mmap(struct file *filp, struct vm_area_struct *vma)
{
    unsigned long size = vma->vm_end - vma->vm_start;
    void *kp = vmalloc(size);
    unsigned long up;

    for (up = vma->vm_start; up < vma->vm_end; up += PAGE_SIZE) {
        struct page *page = vmalloc_to_page(kp); //Finding physical page from virtual address
        err = vm_insert_page(vma, up, page); //How is it different from remap_pfn_range?
        if (err)
            break;
        kp += PAGE_SIZE;
    }

    vma->vm_ops = &dr_vm_ops;
    ps_vma_open(vma);
}

c) not sure who allocates memory in this case.

static int driver_mmap(struct file *filp, struct vm_area_struct *vma)
{
    if (remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
                vma->vm_end - vma->vm_start,
                vma->vm_page_prot)) // creates mapping
        return -EAGAIN;
}

2) If kernel allocates memory for mmap, is n't memory wasted in a&b cases?

3) remap_pfn_range is to map multiple pages where as vm_insert_page is just for single page mapping. is it the only difference b/w these two APIs?

Thank You,

Gopinath.

Answer 1

Which you use depends on what you're trying to accomplish.

(1) A device driver is part of the kernel so it doesn't really make sense to differentiate that way. For these cases, the device driver is asking for memory to be allocated for its own use from the (physical) memory resources available to the entire kernel.

With (a), a physically contiguous space is being allocated. You might do this if there is some piece of external hardware (a PCI device, for example) that will be reading or writing that memory. The return value from kmalloc already has a mapping to kernel virtual address space. remap_pfn_range is being used to map the page into the user virtual address space of the current process as well.

For (b), a virtually contiguous space is being allocated. If there is no external hardware involved, this is what you would typically use. There is still physical memory being allocated to your driver, but it isn't guaranteed that the pages are physically contiguous -- hence fewer constraints on which pages can be allocated. (They will still be contiguous in kernel virtual address space.) And then you are simply using a different API to implement the same kind of mapping into user virtual address space.

For (c), the memory being mapped is allocated under control of some other subsystem. Thevm_pgoff field has already been set to the base physical address of the resource. For example, the memory might correspond to a PCI device's address region (a network interface controller's registers, say) where that physical address is determined/assigned by your BIOS (or whatever mechanism your machine uses).

(2) Not sure I understand this question. How can the memory be "wasted" if it's being used by the device driver and a cooperating user process? And if the kernel needs to read and write the memory, there must be kernel virtual address space allocated and it needs to be mapped to the underlying physical memory. Likewise, if the user space process is to access the memory, there must be user virtual address space allocated and that must be mapped to the physical memory as well.

"Allocating virtual address space" essentially just means allocating page table entries for the memory. That is done separately from actually allocating the physical memory. And it's done separately for kernel space and user space. And "mapping" means setting the page table entry (the virtual address of the beginning of the page) to point to the correct physical page address.

(3) Yes. They are different APIs that accomplish much the same thing. Sometimes you have a struct page, sometimes you have a pfn. It can be confusing: there are often several ways to accomplish the same thing. Developers typically use the one most obvious for the item they already have ("I already have a struct page. I could calculate its pfn. But why do that when there's this other API that accepts a struct page?").