I know that PCI has feature called ATS to translate virtual address to physical address, but I am not sure whether it is enabled by default in current x86 platform.
If it is, how should I generate virtual address and corresponding page table in Linux system?
This is what I guess now. First, DMA API like pci_map_* will call into intel iommu driver to setup page table and get io virtual address. Second, PCI device will use that virtual io address to start transaction, and vt-d will translate that virtual io address to physical address. Am I right about that?
There are not yet any Intel CPUs that support ATS. However, the Intel VT-d spec is available at http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf. It specifies how to enable ATS and how to set up the page tables. See chapter 4 in particular. There are also some diagrams and explanation at https://01.org/blogs/ashokraj/2018/recent-enhancements-intel-virtualization-technology-directed-i/o-intel-vt-d, which may be helpful.