SVM's support vectors decision function representation

Question

I am currently using SVM for my project with 'rbf' kernel.

What i understand from the theory is that the decision function value for the support vectors must be either +1 or -1. (if i use clf.decision_function(x))

But i find some support vectors, the decision function value is even 0.76, -0.88, 0.93 and so on.. (its not even closer to +1 or -1 like 0.99 nor -0.99).

What is wrong in this scenario? Or is my understanding wrong?

Answer 1

I guess there is no range limitation for the decision function value output in SVM.

The value of the decision function for those points, will be a high positive number for high-confidence positive decisions and have a low absolute value (near 0) for low-confidence decisions.

Source here

Code Example:

import numpy as np
from sklearn.svm import SVC

X = np.array([[-1, -1], [-2, -1], [0, 0], [0, 0], [1, 1], [2, 1]])
y = np.array([1, 1, 2, 2, 3, 3])

clf = SVC()
clf.fit(X, y)

print(clf.decision_function(X))
print(clf.predict(X))

Output:

# clf.decision_function(X)
array([[ 2.21034835,  0.96227609, -0.20427163],
       [ 2.22222707,  0.84702504, -0.17843569],
       [-0.16668475,  2.22222222,  0.83335142],
       [-0.16668475,  2.22222222,  0.83335142],
       [-0.20428472,  0.96227609,  2.21036024],
       [-0.17841683,  0.84702504,  2.22221737]])

# clf.predict(X)
array([1, 1, 2, 2, 3, 3])

Answer 2

What SVM is interested is the sign of the decision. e.g., if the sign is negative, the point lies (say) left of the hyperplane. Similarly if the sign is positive, the point lies right of the hyperplane. The value determines how far is it from the hyperplane. Therefore -0.88 means the point is left of the hyperplane and having a distance 0.88. Near the point to the hyperplane, the chances of mis-classification can be considered higher.

Have a look here

To quote from scikit-learn:

the function values are proportional to the distance of the samples X to the separating hyperplane.

Answer 3

sklearn uses soft margin SVMs. From the User Guide:

In general, when the problem isn’t linearly separable, the support vectors are the samples within the margin boundaries.

See also this sklearn example, where depending on C the margin changes and more or fewer points act as support vectors.

So support vectors can have a decision_function score of anything from -1 to +1. (In fact, misclassified points outside the margin will still be support vectors, and will have a score outside even that range. See https://stats.stackexchange.com/a/585832/232706)