gogopython

package module
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 6, 2024 License: Apache-2.0 Imports: 10 Imported by: 2

README

gogopython

Python, but in Go.

CGO_ENABLED=0 go build

Heads up: this currently requires Python 3.12. No if's, and's or but's.

Using

gogopython merely exposes the Python C-Api, so you'll need to use it just like you would without Go. For an example of spinning up a sub-interpreter, see the example program in cmd/main.go.

Library Detection

The biggest pain is finding the Python dynamic library. On some Linux systems, it must be installed separately (Debian-based distros for sure).

sudo apt install libpython3.12

gogopython will try to find the library using distutils via the given Python binary. This may require installing setuptools via pip.

On macOS, it will use otool to

Quick command line test

Simply point the test program at your Python3 binary.

# Create and activate virtual environment.
python3 -m venv venv
. venv/bin/activate

# Install setuptools. This is used for library discovery.
pip install setuptools

# We no longer need the virtual environment enabled.
deactivate

# Run the test app.
go run example/example.go ./venv/bin/python3

Note: if on Linux, make sure you have setuptools installed.

Known Issues

  • Requires Python 3.12 as it uses sub-interpreters. Sorry, not sorry.

  • Linux requires a shim using the ffi Go module that uses purego to leverage libffi, so on Linux libffi must be available. This is all because some super old Python C API functions decide to return a struct on the stack and purego only supports that on macOS currently.

  • The Python api is super thread local storage oriented. Using it with Go is a small nightmare. Gratuitous use of runtime.LockOSThread() is required.

  • The helper function for finding the Python dynamic library won't work with Python installed via XCode as it's a funky dual-arch binary with some dynamic library funkiness and otool can't resolve the actual Python dylib location (if there even is one!)

Documentation

Overview

gogopython wraps a Python dynamic library with Go native functions, making embedding of Python in a native Go app relatively easy. (For some definition of easy)

It wraps common Python C API functions needed to manage interpreters and create/modify Python objects. Not all functions are wrapped. Not all features are wrappable in pure Go as in some cases they're C macros.

Since the #1 headache in using an embedding Python interpreter is finding the necessary library, Python home, and paths for packages, gogopython provides a few helper functions to try figuring this out for the user.

Note: Currently only Python 3.12 is supported.

Index

Constants

View Source 
const (
	PyMemAllocator_NotSet        = iota // Don't change allocator (use defaults).
	PyMemAllocator_Default              // Use defaults allocators.
	PyMemAllocator_Debug                // Default with debug hooks.
	PyMemAllocator_Malloc               // Use malloc(3).
	PyMemAllocator_MallocDebug          // Use malloc(3) with debug hooks.
	PyMemAllocator_PyMalloc             // Use Python's pymalloc.
	PyMemAllocator_PyMallocDebug        // Use Python's pymalloc with debug hooks.
)

Variables

View Source 
var (
	// Converts a Go string into a Python *wchar_t, optionally storing some
	// error information in the provided index (if non-nil).
	Py_DecodeLocale func(s string, index *int) WCharPtr
	// Converts a Python *wchar_t to a Go string, optionally storing some
	// error information in the provided index (if non-nil).
	Py_EncodeLocale func(p WCharPtr, index *int) string

	// Pre-initialize the provided Python interpreter config using "isolated"
	// defaults.
	PyPreConfig_InitIsolatedConfig func(*PyPreConfig)

	// Initialize the provided Python interpreter config using defaults.
	PyConfig_InitPythonConfig func(*PyConfig_3_12)
	// Initialize the provided Python interpreter config using "isolated"
	// defaults.
	PyConfig_InitIsolatedPythonConfig func(*PyConfig_3_12)
	PyConfig_Clear                    func(*PyConfig_3_12)

	// Tear down the global Python interpreter state.
	//
	// This can deadlock depending on the GIL state. It can also panic. Be
	// careful!
	Py_FinalizeEx func() int32

	// Tear down a sub-interpreter using the provided thread state.
	//
	// Can panic or deadlock. Be careful!.
	Py_EndInterpreter func(PyThreadStatePtr)

	// Reports whether we have the GIL. 1 if true, 0 if false.
	PyGILState_Check func() int32

	// Take a reference to the GIL. Caution: this is recursive.
	PyGILState_Ensure func() PyGILState

	// Release a reference to the GIL. Caution: this is recursive.
	PyGILState_Release func(PyGILState)

	PyEval_AcquireThread func(PyThreadStatePtr)
	PyEval_ReleaseThread func(PyThreadStatePtr)
	PyEval_SaveThread    func() PyThreadStatePtr
	PyEval_RestoreThread func(PyThreadStatePtr)

	PyThreadState_Get            func() PyThreadStatePtr
	PyThreadState_New            func(PyInterpreterStatePtr) PyThreadStatePtr
	PyThreadState_Swap           func(PyThreadStatePtr) PyThreadStatePtr
	PyThreadState_Clear          func(PyThreadStatePtr)
	PyThreadState_Delete         func(PyThreadStatePtr)
	PyThreadState_DeleteCurrent  func()
	PyThreadState_GetInterpreter func(PyThreadStatePtr) PyInterpreterStatePtr

	PyInterpreterState_Get    func() PyInterpreterStatePtr
	PyInterpreterState_GetID  func(PyInterpreterStatePtr) int64
	PyInterpreterState_Clear  func(PyInterpreterStatePtr)
	PyInterpreterState_Delete func(PyInterpreterStatePtr)

	// Run a given Python script in the current interpreter, returning an exit
	// code based on if there was a Python exception raised.
	PyRun_SimpleString func(str string) int32

	// Run a given Python script in the current interpreter using the given
	// StartToken mode and globals/locals dicts.
	//
	// Globals will be accessible like any global and the script can mutate the
	// globals mapping using the "globals" keyword in the script.
	//
	// Locals will contain any declared local values from the script and is a
	// simple way to "return" Python data.
	PyRun_String func(str string, start StartToken, globals, locals PyObjectPtr) PyObjectPtr

	// Simplified form of Py_CompileStringFlags.
	Py_CompileString func(str, filename string, start StartToken) PyCodeObjectPtr
	// Simplified form of Py_CompileStringExFlags with optimize set to UseInterpreterLevel.
	Py_CompileStringFlags func(str, filename string, start StartToken, flags *PyCompilerFlags) PyCodeObjectPtr

	// Parse and compile the Python script in str and return the code object. The filename is used
	// to populate the __file__ information for tracebacks and exception messages.
	//
	// Returns NullPyCodeObjectPtr on error.
	Py_CompileStringExFlags func(str, filename string, start StartToken, flags *PyCompilerFlags, optimize OptimizeLevel) PyCodeObjectPtr

	PyEval_EvalCode func(co PyCodeObjectPtr, globals, locals PyObjectPtr) PyObjectPtr

	PyModule_New          func(string) PyObjectPtr
	PyModule_AddObjectRef func(module PyObjectPtr, name string, item PyObjectPtr) int32

	PyBool_FromLong func(int64) PyObjectPtr

	PyLong_AsLong               func(PyObjectPtr) int64
	PyLong_AsLongAndOverflow    func(PyObjectPtr, *int64) int64
	PyLong_AsUnsignedLong       func(PyObjectPtr) uint64
	PyLong_FromLong             func(int64) PyObjectPtr
	PyLong_FromUnsignedLong     func(uint64) PyObjectPtr
	PyLong_FromLongLong         func(int64) PyObjectPtr
	PyLong_FromUnsignedLongLong func(uint64) PyObjectPtr

	PyFloat_AsDouble   func(PyObjectPtr) float64
	PyFloat_FromDouble func(float64) PyObjectPtr

	PyTuple_New     func(int64) PyObjectPtr
	PyTuple_SetItem func(tuple PyObjectPtr, pos int64, item PyObjectPtr) int32

	PyList_New     func(PyObjectPtr) int32
	PyList_Size    func(PyObjectPtr) int64
	PyList_GetItem func(PyObjectPtr, int64) PyObjectPtr
	PyList_SetItem func(list PyObjectPtr, index int, item PyObjectPtr) int32
	PyList_Append  func(list, item PyObjectPtr) int32
	PyList_Insert  func(list PyObjectPtr, index int, item PyObjectPtr) int32

	PyDict_New           func() PyObjectPtr
	PyDictProxy_New      func(mapping PyObjectPtr) PyObjectPtr
	PyDict_Clear         func(PyObjectPtr)
	PyDict_SetItem       func(dict, key, val PyObjectPtr) int32
	PyDict_SetItemString func(dict PyObjectPtr, key string, val PyObjectPtr) int64
	PyDict_GetItem       func(dict, key, val PyObjectPtr) PyObjectPtr
	PyDict_GetItemString func(dict PyObjectPtr, key string) PyObjectPtr

	PySet_New       func(iterable PyObjectPtr) PyObjectPtr
	PyFrozenSet_New func(iterable PyObjectPtr) PyObjectPtr
	PySet_Size      func(PyObjectPtr) int64
	PySet_Contains  func(set, key PyObjectPtr) int32
	PySet_Add       func(set, key PyObjectPtr) int32
	PySet_Discard   func(set, key PyObjectPtr) int32
	PySet_Pop       func(set, key PyObjectPtr) PyObjectPtr
	PySet_Clear     func(set PyObjectPtr) int32

	PyBytes_FromString            func(string) PyObjectPtr
	PyBytes_FromStringAndSize     func(*byte, int64) PyObjectPtr
	PyByteArray_FromStringAndSize func(*byte, int64) PyObjectPtr
	PyBytes_AsString              func(PyObjectPtr) *byte
	PyBytes_Size                  func(PyObjectPtr) int64

	PyUnicode_FromString       func(string) PyObjectPtr
	PyUnicode_AsWideCharString func(PyObjectPtr, *int) WCharPtr

	Py_DecRef func(PyObjectPtr)
	Py_IncRef func(PyObjectPtr)

	PyErr_Clear func()
	PyErr_Print func()

	PyMem_Free func(*byte)

	PyObject_Type   func(PyObjectPtr) PyTypeObjectPtr
	PyType_GetFlags func(PyTypeObjectPtr) uint64
)
View Source 
var (
	Py_PreInitialize            func(*PyPreConfig) PyStatus
	PyConfig_SetBytesString     func(*PyConfig_3_12, *WCharPtr, string) PyStatus
	Py_InitializeFromConfig     func(*PyConfig_3_12) PyStatus
	Py_NewInterpreterFromConfig func(state *PyThreadStatePtr, c *PyInterpreterConfig) PyStatus
)

Our problem children. These all return PyStatus, a struct. These need special handling to work on certain platforms like Linux due to how purego is currently written.

Functions

func FindPythonHomeAndPaths added in v0.5.0 

func FindPythonHomeAndPaths(exe string) (string, []string, error)

Use the provided Python executable to discovery the Python Home and path settings.

func Load_library 

func Load_library(exe string) error

Given the path to a Python binary (exe), attempt to load and wrap the appropriate dynamic library for embedding Python. Load_library uses Cmd from os/exec, so it follows Cmd's resolution semantics.

Currently assumes the provided binary is for Python 3.12.

func PyBytesToString 

func PyBytesToString(b WCharPtr) string

Extract a Go string from a Python *wchar_t.

On failure, returns either an empty string or panics!

Types

type GilType 

type GilType int32
const (
	DefaultGil GilType = 0 // On Python 3.12, defaults to SharedGil
	SharedGil  GilType = 1 // On Python 3.12 and newer, uses unified GIL.
	OwnGil     GilType = 2 // On Python 3.12 and newer, creates a unique GIL.
)

type OptimizeLevel added in v0.6.0 

type OptimizeLevel = int32
const (
	UseInterpreterLevel              OptimizeLevel = -1 // Uses whatever the interpreter was built with.
	NoOptimization                   OptimizeLevel = 0  // No optimization, __debug__ is True.
	RemoveDebugsAndAsserts           OptimizeLevel = 1  // __debug__ is False, no asserts.
	RemoveDebugsAssertsAndDocstrings OptimizeLevel = 2  // __debug__ is False, no asserts, no docstrings.
)

type PyCodeObjectPtr added in v0.6.0 

type PyCodeObjectPtr PyObjectPtr

Opaque pointer to an underlying Python code object. 

const NullPyCodeObjectPtr PyCodeObjectPtr = 0

Represents a NULL pointer to a Python code object.

type PyCompilerFlags added in v0.6.0 

type PyCompilerFlags struct {
	Flags          int32
	FeatureVersion int32
}

type PyConfig_3_12 

type PyConfig_3_12 struct {
	ConfigInit int32

	Isolated              int32
	UseEnvironment        int32
	DevMode               int32
	InstallSignalHandlers int32
	UseHashSeed           int32
	HashSeed              uint64
	FaultHandler          int32
	TraceMalloc           int32
	PerfProfiling         int32
	ImportTime            int32
	CodeDebugRanges       int32
	ShowRefCount          int32
	DumpRefs              int32
	DumpRefsFile          WCharPtr
	MallocStats           int32
	FilesystemEncoding    WCharPtr
	FilesystemErrors      WCharPtr
	PycachePrefix         WCharPtr
	ParseArgv             int32

	OrigArgv    pyWideStringList
	Argv        pyWideStringList
	XOptions    pyWideStringList
	WarnOptions pyWideStringList

	SiteImport          int32
	BytesWarning        int32
	WarnDefaultEncoding int32
	Inspect             int32
	Interactive         int32
	OptimizationLevel   int32
	ParserDebug         int32
	WriteBytecode       int32
	Verbose             int32
	Quiet               int32
	UserSiteDirectory   int32
	ConfigureCStdio     int32
	BufferedStdio       int32
	StdioEncodings      WCharPtr
	StdioErrors         WCharPtr
	// LegacyWindowsStdio  int32 // if windows
	CheckHashPycsMode WCharPtr
	UseFrozenModules  int32
	SafePath          int32
	IntMaxStrDigits   int32

	/* Path configuration inputs */
	PathConfigWarnings int32
	ProgramName        WCharPtr
	PythonPathEnv      WCharPtr
	Home               WCharPtr
	PlatLibDir         WCharPtr

	/* Path configuration outputs */
	ModuleSearchPathsSet int32
	ModuleSearchPaths    pyWideStringList
	StdlibDir            *byte
	Executable           *byte
	BaseExecutable       *byte
	Prefix               *byte
	BasePrefix           *byte
	ExecPrefix           *byte
	BaseExecPrefix       *byte

	/* Parameter only used by Py_Main */
	SkipSourceFirstLine int32
	RunCommand          *byte
	RunModule           *byte
	RunFilename         *byte

	/* Set by Py_Main */
	SysPath0 *byte

	/* Private Fields */
	InstallImportLib int32
	InitMain         int32
	IsPythonBuild    int32
}

Python Interpreter Configuration

This is a version-dependent structure, unfortunately. We need this because it's the stable way of configuring the Home and Path (PythonPathEnv).

Sadly this is also dependent on platform (Windows vs. not-Windows) and some compile time decisions for the Python implementation (e.g. debug, stats).

Ultimately, this should be made private and the configuration complexity hidden from the programmer.

type PyGILState 

type PyGILState int32

Opaque state of the GIL, used sort of as a cookie in the ensure/release function calls.

type PyInterpreterConfig 

type PyInterpreterConfig struct {
	// Whether to share the main interpreters object allocator state.
	//
	// If this is 0, you must set CheckMultiInterpExtensions to 1.
	// If this is 1, you must set Gil to OwnGil.
	UseMainObMalloc int32

	// Whether to allow using Python's os.fork funcion.
	// Note: this doesn't block exec syscalls and subprocess module will still work.
	AllowFork int32

	// Whether to allow using Python's os.exec* functions.
	// Note: this doesn't block exec syscalls and subprocess module will still work.
	AllowExec int32

	// Whether to allow creating Python threads using the threading module.
	AllowThreads int32

	// Whether to allow creating Python daemon threads.
	AllowDaemonThreads int32

	// If 1, require multi-phase (non-legacy) extension modules. Must be 1 if you
	// enable UseMainObMalloc.
	CheckMultiInterpExtensions int32

	// The GIL mode for this sub-interpreter.
	Gil GilType
}

Configuration for a sub-interpreter. All int32 values are really booleans, so 0 = false, 1 = true. (Non-zero may also = true. Not sure!)

type PyInterpreterStatePtr 

type PyInterpreterStatePtr uintptr

Opaque pointer to a Python InterpreterState. 

const NullInterpreterState PyInterpreterStatePtr = 0

NULL version of a Python InterpreterState.

type PyMemAllocator added in v0.5.0 

type PyMemAllocator = int32

type PyObjectPtr 

type PyObjectPtr uintptr

Opaque pointer to an underlying PyObject instance. 

const NullPyObjectPtr PyObjectPtr = 0

Represents a NULL pointer to a Python PyObject

type PyPreConfig 

type PyPreConfig struct {
	ConfigInit        int32
	ParseArgv         int32
	Isolated          int32
	UseEnvironment    int32
	ConfigureLocale   int32
	CoerceCLocale     int32
	CoerceCLocaleWarn int32
	// LegacyWindowsFSEncoding // only on Windows
	Utf8Mode  int32
	DevMode   int32
	Allocator PyMemAllocator
}

type PyStatus 

type PyStatus struct {
	Type     int32
	Func     WCharPtr
	ErrMsg   WCharPtr
	ExitCode int32
}

Return status of some specific Python C API calls.

This is the biggest headache of this whole thing. A few functions return this struct directly instead of either via a pointer or by reference in the function args. It creates a nightware to deal with the various ABI logic for how structs get returned that don't fit into a cpu register width.

If someone has a time machine, please go back and tell Guido not to do this. Please.

type PyThreadStatePtr 

type PyThreadStatePtr uintptr

Opaque pointer to a Python ThreadState. 

const NullThreadState PyThreadStatePtr = 0

NULL version of a Python ThreadState.

type PyTypeObjectPtr added in v0.3.0 

type PyTypeObjectPtr PyObjectPtr

Opaque pointer to an underlying PyTypeObject instance. 

const NullPyTypeObjectPtr PyTypeObjectPtr = 0

Represents a NULL pointer to a Python PyTypeObject

type PythonLibraryPtr 

type PythonLibraryPtr = uintptr

Opaque pointer to a Python dynamic library state in purego.

type StartToken 

type StartToken = int32

Confusingly named, but used to dictate to the Python interpretser & compiler how to interpret the provided Python script in string form. 

const (
	PySingleInput   StartToken = 256 // Used for single statements.
	PyFileInput     StartToken = 257 // Used for modules (many statements).
	PyEvalInput     StartToken = 258 // Used for expressions(?).
	PyFuncTypeInput StartToken = 345 // No idea what this is!
)

type Type added in v0.3.0 

type Type uint64

PyObject types based on inspecting the tpflags of a PyTypeObject 

const (
	Long    Type = (1 << 24)  // Python long.
	List    Type = (1 << 25)  // Python list.
	Tuple   Type = (1 << 26)  // Python tuple.
	Bytes   Type = (1 << 27)  // Python bytes (not bytearray).
	String  Type = (1 << 28)  // Python unicode string.
	Dict    Type = (1 << 29)  // Python dictionary.
	None    Type = 0          // The Python "None" type.
	Float   Type = 1          // Python float.
	Set     Type = 2          // Python set.
	Unknown Type = 0xffffffff // We have no idea what the type is...
)

func Py_BaseType added in v0.3.0 

func Py_BaseType(obj PyObjectPtr) Type

Identify the Python base type from a Python *PyObject.

This uses a heuristic based on inspecting some internal object flags as most of the Python C API for type inspection is written in macros.

See https://docs.python.org/3/c-api/type.html#c.PyType_GetFlags if curious about the flags.

func (Type) String added in v0.7.0 

func (t Type) String() string

type WCharPtr 

type WCharPtr *byte

Opaque pointer to a Python wchar_t string.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.