How to index a converted userdata value?

Question

I tried to convert C++ class to a void pointer using lua_touserdata() and then convert it back to C++ class using lua_pushlightuserdata().

However, I can't index variables in a class once I do the conversion.

Here's my test code:

MyBindings.h

class Vec2
{
public:
    Vec2():x(0), y(0){};
    Vec2(float x, float y):x(x), y(y){};
    float x, y;
};

void *getPtr(void *p)
{
    return p;
}

MyBindings.i

%module my
%{
    #include "MyBindings.h"
%}

%typemap(typecheck) void* 
{
    $1 = lua_isuserdata(L, $input);
}
%typemap(in) void* 
{
    $1 = lua_touserdata(L, $input);
}

%typemap(out) void* 
{
    lua_pushlightuserdata(L, $1);
    ++SWIG_arg;
}

%include "MyBindings.h"

main.cpp

#include "lua.hpp"

extern "C"
{
    int luaopen_my(lua_State *L);
}

int main()
{
    lua_State *L = luaL_newstate();
    luaL_openlibs(L);
    luaopen_my(L);
    lua_settop(L, 0);
    const int ret = luaL_dostring(L, "local vec = my.Vec2(3, 4)\n"
                                     "local p = my.getPtr(vec)\n"
                                     "print(p.x)");
    if (ret)
    {
        std::cout << lua_tostring(L, -1) << '\n';
    }
    lua_close(L);
}

The Result I get :

[string "local vec = my.Vec2(3, 4)..."]:3: attempt to index a userdata value (local 'p')

The Result I expect :

3

What should I do to get the result I expect?

Answer 1

If you want to do it like this you have to adapt you design. First of all the function getPtr cannot work because it is too generic. There is no way SWIG will magically guess the type and do the right thing. You will have to at least fix the type of the input.

MyBindings.h

struct Vec2 {
    Vec2() : x(0), y(0){};
    Vec2(float x, float y) : x(x), y(y){};
    float x, y;
};

void *getPtr(Vec2 &p) { return &p; }

Again, are you really sure you want to do this? Because it is going to get ugly!

You need at least two metamethods, __index and __newindex, to get and set elements of the vector through the pointer. I implemented these in the literal block (%{ ... %}) of the interface file but you can also move them to a header and include this header in the literal block.

Now you have to let Lua know about the metamethods you defined and insert them in a named metatable, so that you can distinguish pointers of type Vec2 from other pointers. Therefore you have to put a bit in the %init section of the interface file to register the metatable when the interpreter starts up.

Because you had to get rid of the void* input argument to getPtr, the typecheck and in typemaps can be removed. The out typemap has to be adapted. We have to allocate memory for a userdata which fits a pointer to Vec2. We set the userdata to this pointer and slap the Vec2 metatable onto it. Now that was super easy wasn't it?_(sarcasm)

MyBindings.i

%module my
%{
    #define SWIG_FILE_WITH_INIT
    #include <string>
    #include "MyBindings.h"

    static int setVec2(lua_State *L) {
        Vec2 *v = *static_cast<Vec2 **>(luaL_checkudata(L, 1, "Vec2"));
        luaL_argcheck(L, v != nullptr, 1, "invalid pointer");
        std::string index = luaL_checkstring(L, 2);
        luaL_argcheck(L, index == "x" || index == "y", 2, "index out of range");
        luaL_argcheck(L, lua_isnumber(L, 3), 3, "not a number");
        float record = lua_tonumber(L, 3);

        if (index == "x") {
            v->x = record;
        } else if (index == "y") {
            v->y = record;
        } else {
            assert(false); // Can't happen!
        }

        return 0;
    }

    static int getVec2(lua_State *L) {
        Vec2 *v = *static_cast<Vec2 **>(luaL_checkudata(L, 1, "Vec2"));
        luaL_argcheck(L, v != nullptr, 1, "invalid pointer");
        std::string index = luaL_checkstring(L, 2);
        luaL_argcheck(L, index == "x" || index == "y", 2, "index out of range");

        if (index == "x") {
            lua_pushnumber(L, v->x);
        } else if (index == "y") {
            lua_pushnumber(L, v->y);
        } else {
            assert(false); // Can't happen!
        }

        return 1;
    }

    static const struct luaL_Reg Vec2_meta[] = {
        {"__newindex", setVec2},
        {"__index", getVec2},
        {nullptr, nullptr} // sentinel
    };
%}

%init %{
    luaL_newmetatable(L, "Vec2");
    luaL_setfuncs(L, Vec2_meta, 0);
    lua_pop(L, 1);
%}

%typemap(out) void* 
{
    void * udata = lua_newuserdata(L, sizeof(Vec2 *));
    *static_cast<void **>(udata) = $1;
    luaL_getmetatable(L, "Vec2");
    lua_setmetatable(L, -2);
    ++SWIG_arg;
}

%include "MyBindings.h"

Let's see whether it works.

test.lua

local my = require("my")
local vec = my.Vec2(3, 4)
local p = my.getPtr(vec)
print(p.x, p.y)
p.x = 1.0
p.y = 2.0
print(p.x, p.y)
print(vec.x, vec.y)

$ swig -lua -c++ MyBindings.i
$ clang++ -Wall -Wextra -Wpedantic -I/usr/include/lua5.3 -fPIC -shared MyBindings_wrap.cxx -o my.so -llua5.3
$ lua5.3 test.lua
3.0 4.0
1.0 2.0
1.0 2.0

It might be a little easier if you used light userdata, but that has the downside that all light userdata will share the same metatable, so you can only do this for one kind of object.

---

Answer to the comment

Casting a pointer to a certain type to void* is called type erasure, because you lose all information about the data contained. Therefore you have to be careful when restoring the type, that you actually restore the correct type. Casting to an unrelated type is undefined behaviour and, if you are lucky, results in a program crash.

You probably don't want to be able to use void* like a Vec2. What's the purpose of casting to void* then, when you want to retain the original meaning anyway. Instead you want to have two functions, getPtr and getVec2. The getPtr function erases the type and gives you a void* object which is unusable in Lua, but is handy to pass to callback functions which accept arbitrary data as void*. The getVec2 function restores the type to Vec2 once you are done.

In the example the getVec2 function returns by reference, i.e. the returned object will be a reference to the object you called getPtr on. That also means that if the original object is garbage collected you have an invalid pointer, which will crash your application.

MyBindings.h

struct Vec2 {
    Vec2() : x(0), y(0){};
    Vec2(float x, float y) : x(x), y(y){};
    float x, y;
};

void *getPtr(Vec2 &p) { return &p; }
Vec2 &getVec2(void *p) { return *static_cast<Vec2 *>(p); }

MyBindings.i

%module my
%{
    #define SWIG_FILE_WITH_INIT
    #include "MyBindings.h"
%}

%include "MyBindings.h"

test.lua

local my = require("my")
local vec = my.Vec2(3, 4)
-- Erase the type of vec to pass it around
local p = my.getPtr(vec)
-- Then restore the type using getVec2
local v = my.getVec2(p)
-- Take care! v is a reference to vec
v.x = 1.0
v.y = 2.0
print(v.x, v.y)
print(vec.x, vec.y)

Example invocation:

$ swig -lua -c++ MyBindings.i
$ clang++ -Wall -Wextra -Wpedantic -I/usr/include/lua5.3 -fPIC -shared MyBindings_wrap.cxx -o my.so -llua5.3
$ lua5.3 test.lua 
1.0 2.0
1.0 2.0

To see the reference semantics fail, place vec = nil collectgarbage() after local p = my.getPtr(vec). It doesn't crash on my machine but Valgrind reports invalid reads and writes.