Removing nested loops within python

Question

I am trying to speed up my code. The biggest problem is a few nested loops I have (they have to iterate over 25000 cells). However, when I try to get rid of these nested loops, I get a different result and I don't seem to get why.

This is one of the nested loop:

for i in range(N):
    for j in range(N):
        # value added in sector i (month k+1)
        VA[i,k+1]= VA[i,k+1] - IO[j,i]*(Produc[i,k+1]/Produc[i,0])

This is what I did to get rid of the inner loop:

for in range(N):
    VA[i,k+1]=VA[i,k+1] - np.sum(IO[:,i])*(Produc[i,k+1]/Produc[i,0])

Thank you for very much your help.

Answer 1

The problem is that assigning to VA constricts the type to VA.dtype, so you can lose accuracy if VA.dtype is less precise than the result from VA[i,k+1] - IO[j,i]*(Produc[i,k+1]/Produc[i,0]).

To keep this rounding you'd want:

for i in range(N):
    # value added in sector i (month k+1)
    VA[i,k+1] -= (IO[:,i]*(Produc[i,k+1]/Produc[i,0])).astype(VA.dtype).sum()

...assuming you're not more happy with the more accurate version!

Some more painstaking research has shown that if the subtractions take the data through 0, the behaviour isn't perfectly emulated. I wouldn't bother though, because emulating subtle bugs is a waste of time ;).

---

Note that if you're happy with

for in range(N):
    VA[i,k+1]=VA[i,k+1] - np.sum(IO[:,i])*(Produc[i,k+1]/Produc[i,0])

you can also do

VA[:,k+1] -= IO.sum(axis=0) * Produc[:,k+1] / Produc[:,0]

which I think is equivalent.

---

Note that this assumes that N is the perfect fit for a lot of these. It could be that VA[:N, :N] is a subset of VA, in which case that's the problem and you should crop everything to N within the calculations.