Imshow add secondary y-axis and keep format

Question

I am trying to add a secondary y-axis to my imshow plot:

Original without secondary y axis

But it wont keep the original format, i.e. i want the secondary y-axis scaled the the same as my axis on the left side.

Trying to add secondary y axis

Code:

...
axes.imshow(zi, cmap='viridis', extent=(0, 500, 0, 250), origin='lower')
ax2 = axes.twinx()
ax2.set_ylim(0, 320)

I tried different settings of the imshow kwargs.