Knockout Sortables, Draggables and Droppables - issues moving between droppables

Question

We are trying to build something where a user has a bank of letters that they will drag down to a particular answer spot. So the idea is the user would spell out a word for example by dragging 4 letters down from the bank and putting them in the correct answer bucket. I was able to get something working with the answers being a draggable inside a droppable, however when I allow the user to drag from one droppable/draggable to another I'm seeing some weird results. It seems like it tries to clone the move still, and it actually screws up the HTML a bit.

What we were doing was we had the initial letter bank set up like this:

    <div id="phonemeOptions" data-bind="foreach: $root.Page.PhonemeOptions" class="dragableSource center-block text-center phonemeOptions">
        <div class="canclick sortablePhrase ui-draggable phonemeOption" data-bind="draggable: { data: $data, options: { helper: 'clone' } }">
            <div class="abbr" data-bind="text: abbreviation"></div>
            <div class="bucketText" data-bind="text: text"></div>
        </div>
    </div>

Then the answer "buckets" were set up like this:

<div class="phoneme-answer-box bucket studentBox" id="phonemeContainer1" data-bind="droppable: $root.Answers.phonemeDrop1">
    <span class="phoneme-x fa fa-times-circle-o" title="Remove Phoneme" data-bind="visible: $root.Answers.phonemeDrop1() != undefined, click: function() { $root.ClearPhoneme(1); }"></span>
    <div class="sortableSource sortableDestination" style="list-style-type:none; padding-left: 0;">
        <!-- ko if: $root.Answers.phonemeDrop1 -->
        <div style="position:relative" class="canclick sortablePhrase phoneme-answer-box-li" data-bind="draggable: { data: $root.Answers.phonemeDrop1 }">
            <div id="phoneme-box-inner-1" class="phoneme-answer-box-inner" data-bind="text: abbreviation">
            </div>
            <div class="phoneme-answer-box-sub" data-bind="text: text">
            </div>
        </div>
        <!-- /ko -->
    </div>
</div>

Here is the result when you try to move from one droppable to another. Notice how the left bucket still has an X and when you debug you can see that both boxes still have an item in them. The HTML is getting moved from the left bucket over top of the right bucket's droppable