Error Backpropagation - Neural network

Question

I am trying to write a code for error back-propagation for neural network but my code is taking really long time to execute. I know that training of Neural network takes long time but it is taking long time for a single iteration as well.

Multi-class classification problem!

Total number of training set = 19978
Number of inputs = 513
Number of hidden units = 345
Number of classes = 10

Below is my entire code:

X=horzcat(ones(19978,1),inputMatrix);         %Adding bias
M=floor(0.66*(513+10));         %Taking two-third of imput+output

Wji=rand(513,M);
aj=X*Wji;

zj=tanh(aj);                      %Hidden Layer output

Wkj=rand(M,10);
ak=zj*Wkj; 

akTranspose = ak';
ykTranspose=softmax(akTranspose);      %For multi-class classification
yk=ykTranspose';                       %Final output

error=0;

%Initializing target variables
t = zeros(19978,10);
t(1:2000,1)=1;
t(2001:4000,2)=1;
t(4001:6000,3)=1;
t(6001:8000,4)=1;
t(8001:10000,5)=1;
t(10001:12000,6)=1;
t(12001:14000,7)=1;
t(14001:16000,8)=1;
t(16001:18000,9)=1;
t(18001:19778,10)=1;

errorArray=zeros(100000,1); %Stroing error values to keep track of error iteration 
errorDiff=zeros(100000,1);
for nIterations=1:5
    errorOld=error;

    aj=X*Wji;                %Forward propagating in each iteration
    zj=tanh(aj);

    ak=zj*Wkj;

    akTranspose = ak';
    ykTranspose=softmax(akTranspose);
    yk=ykTranspose';
    error=0;
    %Calculating error
    for n=1:19978                                      %for 19978 training samples
        for k=1:10                                      %for 10 classes
            error = error + t(n,k)*log(yk(n,k));         %using cross entropy function  
        end
    end
    error=-error;

    Ediff = error-errorOld;

    errorArray(nIterations,1)=error;     
    errorDiff(nIterations,1)=Ediff;

    %Calculating dervative of error wrt weights wji
    derEWji=zeros(513,345);
    derEWkj=zeros(345,10);

    for i=1:513
        for j=1:M;
            derErrorTemp=0;
            for k=1:10
                for n=1:19978
                    derErrorTemp=derErrorTemp+Wkj(j,k)*(yk(n,k)-t(n,k));
                    Calculating derivative of E wrt Wkj%
                    derEWkj(j,k) = derEWkj(j,k)+(yk(n,k)-t(n,k))*zj(n,j);
                end
            end
            for n=1:19978
                Calculating derivative of E wrt Wji
                derEWji(i,j) = derEWji(i,j)+(1-(zj(n,j)*zj(n,j)))*derErrorTemp;
            end
        end
    end


    eta = 0.0001;                    %learning rate

    Wji = Wji - eta.*derEWji;          %updating weights
    Wkj = Wkj - eta.*derEWkj;
end

Answer 1

for-loop is very time-consuming in Matlab even with the help of JIT. Try to modify your code by vectorize them rather than organizing them in a 3-loop or even 4-loop. For example,

for n=1:19978                                      %for 19978 training samples
        for k=1:10                                      %for 10 classes
            error = error + t(n,k)*log(yk(n,k));         %using cross entropy function  
        end
end

can be changed to:

error = sum(sum(t.*yk)); % t and yk are both n*k arrays that you construct

You may try to do similar jobs for the rest of your code. Use dot product or multiplication operations on arrays for different cases.