According to the "Windows Vista and Later Display Driver Model Architecture", there is a channel between application and display miniport driver like this:
Application <---> user-mode Win32 GDI <---> kernel-mode Win32K.sys <---> Dxgikrnl.sys <---> Display Miniport Driver
which doesn't need the user-mode display driver.
I am wondering that is it possible to supply a miniport driver only to support a simple framebuffer-based graphic card?
The answer to your question is YES. Since WDDM1.2 it provides you the option that just support a display only driver, which is basically a framebuffer driver without any 3D support. However, you definitely need a user mode driver to interact with d3d runtime and implement a full set of miniport DDI functions, if you would like to see D3D application running.