Calling SHGetKnownFolderPath from Python?

Question

I've written this minimal reproducible example to calculate the Desktop folder on Windows "the hard way" (using SHGetKnownFolderPath), but I seem to end up with a Success error code while the output buffer only yields b'C' when dereferenced via the .result property of c_char_p. What am I doing wrong?

My code does this:

1. Converts the desired GUID into the cursed _GUID struct format according to Microsoft's specification
2. Allocates result_ptr = c_char_p() which is initially a NULL pointer but will be overwritten with the pointer to the result
3. Calls SHGetKnownFolderPath with the desired GUID struct, no flags, on the current user, passing our result_ptr by reference so its value can be overwritten
4. If SHGetKnownFolderPath indicated success, dereferences result_ptr using .value

I'm getting a result which is only a single char long, but I thought that c_char_p is supposed to be the pointer to the start of a null-terminated string.

Is Windows writing a bogus string into my pointer, am I reading its value out wrongly, or have I made some other error in building my function?

import contextlib
import ctypes
import ctypes.wintypes
import functools
import os
import pathlib
import types
import uuid

try:
    wintypes_GUID = ctypes.wintypes.GUID
except AttributeError:
    class wintypes_GUID(ctypes.Structure):
        # https://learn.microsoft.com/en-us/windows/win32/api/guiddef/ns-guiddef-guid
        # https://github.com/enthought/comtypes/blob/1.3.1/comtypes/GUID.py
        _fields_ = [
            ('Data1', ctypes.c_ulong),
            ('Data2', ctypes.c_ushort),
            ('Data3', ctypes.c_ushort),
            ('Data4', ctypes.c_ubyte * 8)
        ]
        
        @classmethod
        def _from_uuid(cls, u):
            u = uuid.UUID(u)
            u_str = f'{{{u!s}}}'
            result = wintypes_GUID()
            errno = ctypes.oledll.ole32.CLSIDFromString(u_str, ctypes.byref(result))
            if errno == 0:
                return result
            else:
                raise RuntimeError(f'CLSIDFromString returned error code {errno}')

DESKTOP_UUID = 'B4BFCC3A-DB2C-424C-B029-7FE99A87C641'


def get_known_folder(uuid):
    # FIXME this doesn't work, seemingly returning just b'C' no matter what
    result_ptr = ctypes.c_char_p()
    with _freeing(ctypes.oledll.ole32.CoTaskMemFree, result_ptr):
        errno = ctypes.windll.shell32.SHGetKnownFolderPath(
            ctypes.pointer(wintypes_GUID._from_uuid(uuid)),
            0,
            None,
            ctypes.byref(result_ptr)
        )
        if errno == 0:
            result = result_ptr.value
            if len(result) < 2:
                import warnings
                warnings.warn(f'result_ptr.value == {result!r}')
            return pathlib.Path(os.fsdecode(result))
        else:
            raise RuntimeError(f'Shell32.SHGetKnownFolderPath returned error code {errno}')


@contextlib.contextmanager
def _freeing(freefunc, obj):
    try:
        yield obj
    finally:
        freefunc(obj)


assert get_known_folder(DESKTOP_UUID) ==\
       pathlib.Path('~/Desktop').expanduser(),\
       f'Result: {get_known_folder(DESKTOP_UUID)!r}; expcected: {pathlib.Path("~/Desktop").expanduser()!r}'

Answer 1

According to [MS.Learn]: SHGetKnownFolderPath function (shlobj_core.h) (emphasis is mine):

[out] ppszPath

Type: PWSTR*

When this method returns, contains the address of a pointer to a null-terminated Unicode string

Function returns the path as a WIDE (016bit) string which is wchar_t*, or [Python.Docs]: class ctypes.c_wchar_p.
Check [SO]: Passing utf-16 string to a Windows function (@CristiFati's answer) for more details.

So, all you have to change (at get_known_folder very beginning) is:

result_ptr = ctypes.c_wchar_p()  # :)

Other important aspects:

• Check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer) for a common pitfall when working with CTypes (calling functions). That is a problem in this case

• If you're not married to CTypes for resolving this issue (although it would be a good exercise), you can use [GitHub]: mhammond/pywin32 - Python for Windows (pywin32) Extensions which is a Python wrapper over WinAPIs. Documentation (WiP) can be found at [GitHub.MHammond]: Python for Win32 Extensions Help (or [ME.TimGolden]: Python for Win32 Extensions Help).
  Since it consists of lots of boilerplate code, you'd have to write much less:

  [cfati@CFATI-5510-0:e:\Work\Dev\StackExchange\StackOverflow\q078097730]> python
  Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] on win32
  Type "help", "copyright", "credits" or "license" for more information.
  >>>
  >>> import pathlib
  >>> from win32comext.shell import shell
  >>>
  >>>
  >>> print(str(pathlib.Path("~/Desktop").expanduser()))
  C:\Users\cfati\Desktop
  >>>
  >>> print(shell.SHGetKnownFolderPath("{B4BFCC3A-DB2C-424C-B029-7FE99A87C641}"))
  C:\Users\cfati\Desktop
  >>>
  

Answer 2

The main problem is the output parameter is PWSTR* and the object passed PSTR instead. When the right type is used the pathlib and fsdecode part is unneeded as well.

This single letter returned is a clue. Wide strings are encoded in UTF-16, so you get back 'C\x00:\x00\...' Instead of 'C:...' and getting the .value of a c_char_p stops at the first null. With c_wchar_p, ctypes will return the UTF-16-decoded Python str from .value.

It's a good habit to specify .argtypes and .restype for every function called by ctypes so it can type-check that proper parameters are passed, which would have caught the problem. .errcheck is also good from making functions raise exceptions on error.

Here's a fully-specified, working example:

import ctypes as ct
import ctypes.wintypes as w

class HResultError(Exception):
    pass

def hresultcheck(result, func, args):
    if result != S_OK:
        raise HResultError(f'{result} (0x{result & 0xFFFFFFFF:08X})')
    return None

class GUID(ct.Structure):
    # https://learn.microsoft.com/en-us/windows/win32/api/guiddef/ns-guiddef-guid
    # https://github.com/enthought/comtypes/blob/1.3.1/comtypes/GUID.py
    _fields_ = [('Data1', ct.c_ulong),
                ('Data2', ct.c_ushort),
                ('Data3', ct.c_ushort),
                ('Data4', ct.c_ubyte * 8)]
    
    @classmethod
    def from_uuid(cls, u):
        guid = GUID()
        CLSIDFromString(u, ct.byref(guid))
        return guid

# Definitions from Windows headers not included in ctypes.wintypes.
KNOWNFOLDERID = GUID
REFKNOWNFOLDERID = ct.POINTER(KNOWNFOLDERID)
PWSTR = w.LPWSTR
HRESULT = w.LONG
CLSID = GUID
LPCLSID = ct.POINTER(CLSID)
S_OK = 0

shell32 = ct.WinDLL('shell32', use_last_error=True)
ole32 = ct.WinDLL('ole32', use_last_error=True)

# Explicit argument and return types matching MSDN docs,
# and automatic return type checking.
SHGetKnownFolderPath = shell32.SHGetKnownFolderPath
SHGetKnownFolderPath.argtypes = REFKNOWNFOLDERID, w.DWORD, w.HANDLE, ct.POINTER(PWSTR)
SHGetKnownFolderPath.restype = HRESULT
SHGetKnownFolderPath.errcheck = hresultcheck
CLSIDFromString = ole32.CLSIDFromString
CLSIDFromString.argtypes = w.LPCOLESTR, LPCLSID
CLSIDFromString.restype = HRESULT
CLSIDFromString.errcheck = hresultcheck
CoTaskMemFree = ole32.CoTaskMemFree
CoTaskMemFree.argtypes = w.LPVOID,
CoTaskMemFree.restype = None

DESKTOP_UUID = GUID.from_uuid('{B4BFCC3A-DB2C-424C-B029-7FE99A87C641}')

def get_known_folder(guid):
    path = PWSTR()  # bug was here.  C equivalent is "wchar_t*" but was "char*".
    try:
        SHGetKnownFolderPath(ct.byref(guid), 0, None, ct.byref(path))
        return path.value
    finally:
        CoTaskMemFree(path)

print(get_known_folder(DESKTOP_UUID))

Output:

C:\Users\Mark\Desktop