Handwritten character recognition with Gabor Filter and SVM with PCA

Question

I am trying to build a simple program in Python to classify handwritten digits (numbers) with the Gabor filter.

I am using the MNIST database (http://yann.lecun.com/exdb/mnist/) for the training and the way I do it is by:

1. Calculating the phase of the Gabor filter for different orientations and scales.
2. Performing a PCA to reduce the dimensionality of the features (i.e. the phases).
3. Performing an SVM classification with a radial basis function as the kernel.

The code that I use is the following:

# Import required libraries
import numpy as np
from sklearn.decomposition import PCA
from sklearn import svm
from skimage.filters import gabor
import matplotlib.pyplot as plt
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import train_test_split
import idx2numpy
from sklearn.model_selection import KFold
from sklearn.multiclass import OneVsOneClassifier

# Load NMIST dataset.
trainX = idx2numpy.convert_from_file("data/train-images-idx3-ubyte")
labelsX = idx2numpy.convert_from_file("data/train-labels-idx1-ubyte")

features = np.empty((1, 2))
labels = np.empty((1,))
mxm = 100
for i in range(mxm):
    image = trainX[i]
    for freq in np.arange(0.2,1.2,0.2):
        for orientation in range(0,360,45):
            filt_real, filt_imag = gabor(image,frequency=freq,theta=orientation) # Compute Gabor Filter on image i for a given scale and orientation
            phase = np.angle(filt_real + 1.0j*filt_imag) # Compute phase with real and complex part
            pca = PCA(n_components=2) # Initialize a 2-component PCA
            pca.fit(phase) # Fit phase in PCA
            curr = pca.transform(phase) # Feature reduction
            features = np.append(features,curr,axis=0) # Append features
            labels = np.append(labels,[labelsX[i]]*len(curr)) # Append labels

kf = KFold(n_splits=mxm)

counter = 1
for train, test in kf.split(features):
    print('Fold',counter)
    counter = counter + 1
    X_train, X_test, y_train, y_test = features[train], features[test], labels[train], labels[test]
    clf = OneVsOneClassifier(svm.SVC(kernel='rbf'))
    clf.fit(X_train,y_train)
    print('Accuracy:',clf.score(X_test,y_test))

But for some reason, it gives me a very low accuracy (around 0.1). So, my question is: Is SVM with PCA a good strategy for feature extraction of the Gabor filter?

I do not want to use neural networks.