Filtering image in a way as in overlay apps

Question

I am creating an application to give the functionality of overlay as in photoshop.

I have two images,

1. The background image on which the overlay would be added,

2. The Foreground image to which the filtration will be applied

Note: Both the images are in .jpeg format

I tried to use the Chroma Key effect, but it has some smoothness issues and not working properly on different color removal.

This is the image after applying the ChromaKey effect by removing black color

The above image has smoothness issues

There are images like this also when I try to remove its background color, it doesn't work properly

How can I achieve this functionality

Thanks!