Reduce size of large matrix populated with only few kinds of elements in R

Question

I have a matrix sdist in R as follows.

library("babynames")
library("stringdist")
d <- babynames
sdist <- stringdistmatrix(d$name[1:1000], d$name[1001:10000], method="lv", useBytes = T)
sdist[sdist > 3] <- Inf

The size of this matrix is large even when only few kinds of elements are there.

object.size(sdist)
72000112 bytes

dim(sdist)
[1] 1000 9000
length(sdist)
[1] 9000000

table(sdist)
sdist
      0       1       2       3     Inf 
   4093   12125   72937  381084 8529761 

Is there any way to reduce the size of these kinds of matrices. I can't use sparse matrices as the number of 0s is less.