WPF Losing caret position when moving mouse over adorner in richtextbox

Question

I have an adorner layer in my richtextbox. Whenever a word is misspelled I add an adorner to the richtextbox's adornerlayer. I want to be able to right click on the adorner and capture that event but I don't want to lose the caretposition of the richtextbox when the mouse is over the adorned element.

I've tried setting IsHitTestVisible=False. This keeps the mouse from changing to a pointer when I'm over the adorned element like I want so that I can click on the underlying richtextbox, but it complicates the rightmousebutton click capturing.

Protected Sub Editor_PreviewRightMouseButtonUp(ByVal sender As Object, ByVal e As MouseEventArgs)

    ' Retreive the coordinates of the mouse button event.
    Dim pt As Point = e.GetPosition(CType(sender, UIElement))

    ' Initiate the hit test by setting up a hit test result callback method.
    VisualTreeHelper.HitTest(Me, Nothing, New HitTestResultCallback(AddressOf HitTestCallBack_PreviewRightMouseButtonUp), New PointHitTestParameters(pt))
    e.Handled = True
End Sub

Public Function HitTestCallBack_PreviewRightMouseButtonUp(ByVal result As HitTestResult) As HitTestResultBehavior
    If result.VisualHit.GetType() Is GetType(HighLightAdorner) Then
        Dim adornerControl As HighLightAdorner = DirectCast(result.VisualHit, HighLightAdorner)
        Dim e = New MouseButtonEventArgs(Mouse.PrimaryDevice, Environment.TickCount, MouseButton.Right)
        e.RoutedEvent = Mouse.PreviewMouseUpEvent
        adornerControl.RaiseEvent(e)
        Return HitTestResultBehavior.Stop
    End If
    Return HitTestResultBehavior.Continue
End Function

This allows me to capture the rightmousebutton click but it only seems to work when I click right at the bottom of the adornedElement. If I try to click in the body of the adornedElement it doesn't find it in the hitTesting.

Am I going about this correctly or is there a better way to allow an underlying control to maintain it cursor when an element above it needs to be able to accept mouse events.

Thanks for the help!