How to disable the auto scroll of a RecyclerView (ListAdapter) that happens when an item is updated?

Question

BACKGROUND

I have a UI that shows a list of users' fullnames with a like/dislike button for each item. I am using a ListAdapter that under the hood uses DiffUtil and AsyncListDiffer APIs. The list of users is received as a LiveData from a Room database and it's ordered by "isLiked".

PROBLEM

Whenever the like button is tapped, Room as I am using a LiveData will re-submit the new data to the adapter. The problem is that as the list is ordered by "isLiked", the liked user will change its position and the RecyclerView will always sroll to the new position.

I don't want to see the new position of the updated item. So, how can I disable the auto scroll behavior?

WHAT I TRIED

MainActivity.kt

    ..
    val userAdapter = UsersAdapter(this)
    val ll = LinearLayoutManager(this, LinearLayoutManager.VERTICAL, false)
   
    recyclerView.apply {
        layoutManager = ll
        adapter = userAdapter
        itemAnimator = null
        setHasFixedSize(true)
    }
    
    viewModel.users.observe(this, {
        // This will force the recycler view to scroll back to the previous position
        // But it's more of a workaround than a clean solution.
        val pos = ll.findFirstVisibleItemPosition()
        userAdapter.submitList(it) {
            recyclerView.scrollToPosition(pos)
        }
    })
    ..

UsersAdapter.kt

class UsersAdapter(
    private val clickListener: UserClickListener
) : ListAdapter<UserEntity, UsersAdapter.UserViewHolder>(DIFF_CALLBACK) {

    override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): UserViewHolder {
        val view = LayoutInflater.from(parent.context).inflate(R.layout.item_user, parent, false)
        return UserViewHolder(view)
    }

    override fun onBindViewHolder(holder: UserViewHolder, position: Int) {
        val userEntity = getItem(position)
        holder.bind(userEntity, clickListener)
    }

    class UserViewHolder(view: View) : RecyclerView.ViewHolder(view) {
        private val textView: TextView = view.findViewById(R.id.fullName)
        private val fav: ImageButton = view.findViewById(R.id.fav)

        fun bind(user: UserEntity, clickListener: UserClickListener) {
            textView.text = user.fullName

            val favResId = if (user.favorite) R.drawable.like else R.drawable.dislike
            fav.setImageResource(favResId)

            fav.setOnClickListener {
                val newFav = !user.favorite
                val newFavResId = if (newFav) R.drawable.like else R.drawable.dislike
                fav.setImageResource(newFavResId)
                clickListener.onUserClicked(user, newFav)
            }
        }
    }

    interface UserClickListener {
        fun onUserClicked(user: UserEntity, isFavorite: Boolean)
    }

    companion object {
        private val DIFF_CALLBACK = object : DiffUtil.ItemCallback<UserEntity>() {
            override fun areItemsTheSame(
                oldUser: UserEntity,
                newUser: UserEntity
            ) = oldUser.id == newUser.id

            override fun areContentsTheSame(
                oldUser: UserEntity,
                newUser: UserEntity
            ) = oldUser.fullName == newUser.fullName && oldUser.favorite == newUser.favorite
        }
    }
}

I tried using a regular RecyclerView adapter and DiffUtil with detect moves set to false.
I added the AsyncListDiffer as well.
I tried the ListAdapter, and even tried the paging library and used the PagedListAdapter.
DiffUtil's callback changes the auto scrolling, but i couldn't get the desired behavior.

Any help is greatly appreciated!