ScrollableControl resets ScrollBar Value after user scrolled

Question

I'm writing an image viewer control in C# inheriting from ScrollableControl. It allows zooming and panning the view on the image and has scroll bars reflecting the visible image area.

I'm using the integrated HorizontalScroll and VerticalScroll and could implement updating them to appear as expected (setting Visible, Maximum, LargeChange andValue most prominently):

SizeF viewSize = new SizeF(ClientSize.Width / Zoom, ClientSize.Height / Zoom);
SizeF freeTopLeft = new SizeF(-_offset.X, -_offset.Y);
SizeF freeBotRight = new SizeF(
    viewSize.Width - ImageSize.Width - freeTopLeft.Width,
    viewSize.Height - ImageSize.Height - freeTopLeft.Height);
// Horizontal
if (freeTopLeft.Width < 0 || freeBotRight.Width < 0)
{
    HorizontalScroll.Visible = true;
    HorizontalScroll.Maximum = (int)ImageSize.Width;
    HorizontalScroll.LargeChange = (int)viewSize.Width;
    HorizontalScroll.Value = (int)-freeTopLeft.Width;
}
else
{
    HorizontalScroll.Visible = false;
}
// Vertical same as horizontal using Height, removed for brevity.

When the user drags the scrollbar thumb, I correctly update the displayed part of the image as follows in OnScroll(ScrollEventArgs se):

switch (se.ScrollOrientation)
{
    case ScrollOrientation.HorizontalScroll:
        _offset.X = se.NewValue;
        break;
    case ScrollOrientation.VerticalScroll:
        _offset.Y = se.NewValue;
        break;
}
Refresh();

However, as soon as the user releases the mouse button he dragged the thumb with, both scroll bars jump back to Value 0. It does not create another OnScroll event, so my image still shows the expected part, but the scroll bars "desynced" with what is displayed:

I checked many times if my code somehow manipulates the Value unexpectedly while and after scrolling, but I never set that property erroneously. I also tried setting the ScrollBar values manually in OnScroll, but it did not help.

What do I need to do to correctly keep the thumb in place? What am I doing wrong?