Combining C and Python functions in a module

0 votes
asked Jun 18, 2009 by james-hopkin

I have a C extension module, to which I would like to add some Python utility functions. Is there a recommended way of doing this?

For example:

import my_module

my_module.super_fast_written_in_C()
my_module.written_in_Python__easy_to_maintain()

I'm primarily interested in Python 2.x.

3 Answers

0 votes
answered Jun 18, 2009 by mike-hordecki

Prefix your native extension with an underscore. Then, in Python, create a wrapper module that imports that native extension and adds some other non-native routines on top of that.

0 votes
answered Jun 18, 2009 by john-machin

The usual way of doing this is: mymod.py contains the utility functions written in Python, and imports the goodies in the _mymod module which is written in C and is imported from _mymod.so or _mymod.pyd. For example, look at .../Lib/csv.py in your Python distribution.

0 votes
answered Jun 19, 2009 by alex-martelli

The existing answers describe the method most often used: it has the potential advantage of allowing pure-Python (or other-language) implementations on platforms in which the compiled C extension is not available (including Jython and IronPython).

In a few cases, however, it may not be worth splitting the module into a C layer and a Python layer just to provide a few extras that are more sensibly written in Python than in C. For example, gmpy (lines 7113 ff at this time), in order to enable pickling of instances of gmpy's type, uses:

copy_reg_module = PyImport_ImportModule("copy_reg");
if (copy_reg_module) {
    char* enable_pickle =
        "def mpz_reducer(an_mpz): return (gmpy.mpz, (an_mpz.binary(), 256))\n"
        "def mpq_reducer(an_mpq): return (gmpy.mpq, (an_mpq.binary(), 256))\n"
        "def mpf_reducer(an_mpf): return (gmpy.mpf, (an_mpf.binary(), 0, 256))\n"
        "copy_reg.pickle(type(gmpy.mpz(0)), mpz_reducer)\n"
        "copy_reg.pickle(type(gmpy.mpq(0)), mpq_reducer)\n"
        "copy_reg.pickle(type(gmpy.mpf(0)), mpf_reducer)\n"
    ;
    PyObject* namespace = PyDict_New();
    PyObject* result = NULL;
    if (options.debug)
        fprintf(stderr, "gmpy_module imported copy_reg OK\n");
    PyDict_SetItemString(namespace, "copy_reg", copy_reg_module);
    PyDict_SetItemString(namespace, "gmpy", gmpy_module);
    PyDict_SetItemString(namespace, "type", (PyObject*)&PyType_Type);
    result = PyRun_String(enable_pickle, Py_file_input,
                          namespace, namespace);

If you want those few extra functions to "stick around" in your module (not necessary in this example case), you would of course use your module object as built by Py_InitModule3 (or whatever other method) and its PyModule_GetDict rather than a transient dictionary as the namespace in which to PyRun_String. And of course there are more sophisticated approaches than to PyRun_String the def and class statements you need, but, for simple enough cases, this simple approach may in fact be sufficient.

Welcome to Q&A, where you can ask questions and receive answers from other members of the community.
Website Online Counter

...