Smooth jagged pixels

Question

I've created a pinch filter/effect on canvas using the following algorithm:

// iterate pixels
for (var i = 0; i < originalPixels.data.length; i+= 4) {

    // calculate a pixel's position, distance, and angle
    var pixel = new Pixel(affectedPixels, i, origin);

    // check if the pixel is in the effect area
    if (pixel.dist < effectRadius) {

        // initial method (flawed)
        // iterate original pixels and calculate the new position of the current pixel in the affected pixels
        if (method.value == "org2aff") {

            var targetDist = ( pixel.dist - (1 - pixel.dist / effectRadius) * (effectStrength * effectRadius) ).clamp(0, effectRadius);
            var targetPos  = calcPos(origin, pixel.angle, targetDist);

            setPixel(affectedPixels, targetPos.x, targetPos.y, getPixel(originalPixels, pixel.pos.x, pixel.pos.y));

        } else {

            // alternative method (better)
            // iterate affected pixels and calculate the original position of the current pixel in the original pixels
            var originalDist = (pixel.dist + (effectStrength * effectRadius)) / (1 + effectStrength);
            var originalPos  = calcPos(origin, pixel.angle, originalDist);

            setPixel(affectedPixels, pixel.pos.x, pixel.pos.y, getPixel(originalPixels, originalPos.x, originalPos.y));

        }

    } else {
        // copy unaffected pixels from original to new image
        setPixel(affectedPixels, pixel.pos.x, pixel.pos.y, getPixel(originalPixels, pixel.pos.x, pixel.pos.y));
    }
}

I've struggled a lot to get it to this point and I'm quite happy with the result. Nevertheless, I have a small problem; jagged pixels. Compare the JS pinch with Gimp's:

I don't know what I'm missing. Do I need to apply another filter after the actual filter? Or is my algorithm wrong altogether?

---

I can't add the full code here (as a SO snippet) because it contains 4 base64 images/textures (65k chars in total). Instead, here's a JSFiddle.

Answer 1

One way to clean up the result is supersampling. Here's a simple example: https://jsfiddle.net/Lawmo4q8/

Basically, instead of calculating a single value for a single pixel, you take multiple value samples within/around the pixel...

let color =
    calcColor(x - 0.25, y - 0.25) + calcColor(x + 0.25, y - 0.25) +
    calcColor(x - 0.25, y + 0.25) + calcColor(x + 0.25, y + 0.25);

...and merge the results in some way.

color /= 4;

Answer 2

In stead of applying a successive filter it is best to do oversampling directly in the pinch algorithm. (instead of calculating a single pixel do it to more slightly shifted pixels and average)

Another option is to upscale the original image, run your algorithm, and scale down the image interpolating the pixels.