RxSeekBar onNext called before debounce timeout (very inconsistent)

Question

I'm working on AndroidTv where I'm using the standard seekBar for playback.

This is how I'm using the RxSeekBar

RxSeekBar.changeEvents(seekBar)
            .debounce(SEEKBAR_DEBOUNCE_TIME, TimeUnit.MILLISECONDS)
            .observeOn(AndroidSchedulers.mainThread())
            .subscribe(new Observer<SeekBarChangeEvent>() {
                @Override
                public void onSubscribe(Disposable disposable) {
                    compositeDisposable.add(disposable);
                }

                @Override
                public void onNext(SeekBarChangeEvent seekBarChangeEvent) {
                    if (seekBarChangeEvent instanceof SeekBarProgressChangeEvent) {
                        if (seekBar.hasFocus()) {
                            DebugLog.d(TAG, "(PROGRESS) SeekBarProgressChangeEvent");
                            seekBarSeekProgress = seekBarChangeEvent.getView().getProgress();
                            onSeek();
                        }
                    }
                }

                @Override
                public void onError(Throwable e) {
                    DebugLog.d(TAG, "RxSeek error - " + e.getMessage());
                }

                @Override
                public void onComplete() {

                }
            });

Basically, I want to update things after the debounce timeout. This works fine 7/10 times but sometimes onNext is called prematurely and things go off hand.

Am I doing something wrong here?