I did a simple pcsctest and the smartcard was verified sucessfully. Also did a system_profiler SPSmartCardsDataType and show the certificate is there. It's just the browser is not prompting for the certificate for selection. I am in macOS Ventura version 13.5.2 with M1 Pro chip.
I found similar issues for mac users at some point but they seem to have been fixed in later updates. So I think this might be new or probably re-introduced again in the new updates. How can I work around this problem?