When I convert pixels to physical units using "extent" inside of plt.imshow(), my subplots that rely on units of pixels break

Question

I'm trying to plot an image using physical units instead of pixels (I use extent to imitate this). On the top and side of my plot is a plot of the lineout of my image. Everything is originally in units of pixels, but I am trying to convert it to physical units. I use an "extent" inside of my imshow() to change my main plot to physical units, but my lineouts on the sides rely on units of pixels as they rely on the index of the main plot. A quick fix would be an extent inside of my lineout plots but I can't find an equivalent function. I attached a few pictures to visualize whats going on.

This is my imshow and extent for my main plot:

img1 = ax1.imshow(I, cmap='Greys_r',extent=(-ex2/(2*um),ex2/(2*um),-ex2/(2*um),ex2/(2*um)))

These are my lineouts on the side:

top_ax = divider.append_axes("top", 1.0, pad=0.2, sharex=ax1)  # Create top lineout
right_ax = divider.append_axes("right", 1.0, pad=0.2, sharey=ax1)  # Create right lineout
xvals = np.linspace(0, (gridsize-1), gridsize)
yvals = np.linspace(0, (gridsize-1), gridsize)

right_ax.plot(I[:, gridsize//2], yvals, 'red', lw=1)  #gridsize is the total number of pixels in I
top_ax.plot(xvals, I[gridsize//2, :], 'green', lw=1)  

This is how I want the plot to look, but in physical units

This is how my plot looks with physical units. Notice that the top and side plot are units of pixels to index and the main plot is using units of microns