In OpenGL, is it possible to use multisample anti-aliasing only on edge pixels and sample each other pixel only once in order to improve performance?
In other words, I want to treat edge pixels and non-edge pixels differently, and sample multiple points for each edge pixel, while only sampling the center point for each non-edge pixel.
I have tried to Google this but I have not found any sources that discuss this; however, this video about the graphics in Manifold Garden suggests that they use a form of MSAA only on pixels that are close to an edge.
That's what multisampling is. The fragment evaluation happens at a lower rate than the number of samples in the framebuffer. The results of the evaluation are written to a number of samples that fit the pixel coverage area of the primitive rasterization. While implementations differ on what the ratio should be (some always sample the fragment shader once, others do it at a rate of 1/4th that of the sample count, etc), that is the basic idea.
Multisampling already does what you want. If it did a 1:1 ratio of fragment evaluation to samples, that would just be "supersampling".