How to deal with singular matrix in Local linear embedding?

Question

I'm getting singular matrix error while solving for the local weights using the equation

Ci*Wi = 1

where Ci is the local co-variance matrix constructed from neighbors of point i and wi is the weight vector needed to construct the point i from its neighbors.

I'm using the IRIS dataset and number of neighbors is 20.

Answer 1

For someone who comes here looking for answer.
This problem arises because weight gets concentrated to only few members. Few neighbors contribute heavily while other have zero contribution. Add a regularizing term to penalize the larger weights. If your covariance matrix is C, regularization value is 0.0005 and I is an identity matrix of same order as C, update C using the formula below

C = C + 0.0005*I(n)