Is there possibility in YANG language to select one option in bit type leaf, which will deselect all other bits ? What I would like to do is, by default Whole-Platform is selected, but when user selects some other option, the Whole-Platform will be deselected. And otherwise when user selects Whole-Platform, all other options should be deselected.
leaf check-type {
type bits {
bit Whole-platform;
bit NSO-checks;
bit ESC-checks;
}
default "Whole-platform";
}

No, YANG bits type will not let you do that, since no relation between individual bits ever exists by default - other than the notion of all of them belonging to the same set of "configurable flags".
This is where a union type would be a viable modeling decision.
Doing this implies an XOR (mutual exclusivity) between value
Whole-platformand the set of remaining bit values to the reader.Valid values:
Invalid:
You could leave your type as is and handle the one bit to rule them all in a description, since that is just human readable normative text.
Note: what you are trying to achieve is an implementation detail.