I am working on a GCC plugin for inserting custom assembly instructions at specific positions of each function, e.g. at the very beginning or right before return statements. The plugin is registered to be executed during the RTL passes and works just fine in principle.
Now my problem is that using this approach I can only instrument programs where I have the full source code available, i.e. no libraries, because those are only linked into the application and not compiled by me (this is only about static linking; dynamic linking is a different topic).
Is there a way to perform optimization at linker level using a custom plugin, similarly to what I am doing at compiler level? I know that certain optimizations take place during linking, but I could not find any resources on the question if/how what I want to do is possible at all.
Thanks!
There are two possibilities to optimize at linker level:
The first one is the lto. https://gcc.gnu.org/wiki/LinkTimeOptimization. In this case, it will be an rtl pass which will be called on one module.
The second option is to use linker relaxation if is supported for your target. Based on the relaxation information you can insert or delete instructions