Abstract factory design pattern : issue handling abstractions

Question

In a project, I have decided to try to hide all the widget toolkit details behind an abstract factory pattern. The concrete widget toolkit is gtkmm. Here is simple diagram showing the architecture I have so far:

(So for now, I'm simply trying to abstract stuff away. The factory is not there yet. Eventually, it will construct the Gtkmm3* classes and return the corresponding interface.)

I started trying to abstract away the "containers" or "layout". In my code, I need to operate on the layouts. For example, I need to be able to register widgets in them. Using plain gtkmm, I would use:

void Gtk::Grid::attach(Gtk::Widget& child, int left, int top, int width, int height);

which in the case of my abstractions would translate to something like this:

void ILayout::attach(IWidget& child, int left, int top, int width, int height);

Implementing ILayout::attach is easy. I can simply dynamic_cast the child argument to the Gtk::Widget type and call Gtk::Grid::attach. This is possible because the concrete Gtkmm3Widget class also inherits from Gtk::Widget. In my code, I also need to access the specific elements inside de layout. In gtkmm I used the

Gtk::Widget* Gtk::Grid::get_child_at(int left, int top);

call. In the case of my abstraction, this would translate to:

IWidget* ILayout::get_child_at(int left, int top);

In this case, the implementation is not so simple, because I access a bare Gtk::Widget* but must return a (non owning) IWidget*, of which Gtk::Widget knowns nothing about. Here is the code:

IWidget* Gtkmm3Layout::get_child_at(int left, int top)
{
    Gtk::Widget* gtkWidget = get_child_at(left, top);

    return ???
}

In the above code, if I construct a Gtkmm3Widget instance and return its address, then I will get a dangling pointer because it will be destroyed at the end of the scope.

Questions:

1. Is there a way to return the needed non-owning Gtk::Widget*?
2. I don't see any and I'm starting to think I have a design flaw*. If this is the case, what would be a better way to approach this?

* I have found no "real life" example of an implementation of this design pattern I could study. All I find are toy examples.