I am using service fabric locally.
If I stopped debug mode for a couple of mins or restart the computer and continued my work it redeploys the application again from scratch.
How to make it only deploy the changed libraries (which I think it what it should do in the first place)
There is the Application Debug Mode property you can use (source):
I think you are looking for option 1 or 3.
You can set the property using the Service Fabric Application project properties.