Some comments:
- your ssl_convert_filename() function has a strange signature
- using PyByteArray_AsString() is wrong, we release the GIL later and the buffer could be reallocated, leading to a crash; the right approach is to use PyObject_GetBuffer() and later PyBuffer_Release()
- as such, your function could simply return the bytes/bytearray PyObject *, and not fill the char * pointer
- adding tests for both the bytes and unicode cases would be nice