Items get reshuffled on scrolling when using StaggeredGridLayoutManager

Question

I'm using StaggeredGridLayoutManager with recylerview and passing a list of images in adapter. When I scroll down and then up, the order of image items gets shuffled. I googled and went through similar SO answers, suggesting setGapStrategy() to GAP_HANDLING_MOVE_ITEMS_BETWEEN_SPANS and GAP_HANDLING_NONE, I used both but no got no change in the shuffle behavior.

Below is my code for layoutmanager:

NewActivitiesAdapter adapter = new NewActivitiesAdapter(this, listImages, optionList);
        SpacesItemDecoration decoration = new SpacesItemDecoration(16);

        StaggeredGridLayoutManager manager = new StaggeredGridLayoutManager(4, StaggeredGridLayoutManager.VERTICAL);
        manager.setGapStrategy(StaggeredGridLayoutManager.GAP_HANDLING_MOVE_ITEMS_BETWEEN_SPANS);
        mRecyclerView.setLayoutManager(manager);
        mRecyclerView.setAdapter(adapter);
        mRecyclerView.setHasFixedSize(true);
        mRecyclerView.addItemDecoration(decoration);

My item_image_tile xml:

<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:card_view="http://schemas.android.com/apk/res-auto"
    android:id="@+id/rootView"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_margin="6dp"
    android:background="@color/pink_bg"
    android:clipChildren="false"
    android:clipToPadding="false"
    android:focusable="true"
    android:paddingStart="2dp"
    android:paddingEnd="2dp">

    <androidx.cardview.widget.CardView
        android:id="@+id/cardview"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginBottom="3dp"
        android:background="@android:color/transparent"
        android:clipChildren="false"
        android:clipToPadding="false"
        android:elevation="1dp"
        card_view:cardBackgroundColor="@android:color/transparent"
        card_view:cardCornerRadius="6dp"
        card_view:cardElevation="1dp"
        card_view:cardPreventCornerOverlap="false"
        card_view:cardUseCompatPadding="true">

        <ImageView
            android:id="@+id/iv_content"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:scaleType="fitXY" />

    </androidx.cardview.widget.CardView>

    <TextView
        android:id="@+id/tvTitle"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_below="@id/cardview"
        android:layout_centerHorizontal="true"
        android:fontFamily="@font/open_sans"
        android:paddingBottom="2dp"
        android:text="Image sample title"
        android:textColor="@color/black_color"
        android:textSize="12sp" />

</RelativeLayout>

Thank you in advance!