The Python/C API

226 Pages·2016·0.89 MB·English

The Python/C API
Release 3.5.1
Guido van Rossum
and the Python development team
February24,2016
PythonSoftwareFoundation
Email: [e-mail]

CONTENTS
1 Introduction 3
1.1 IncludeFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Objects,TypesandReferenceCounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 EmbeddingPython . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.5 DebuggingBuilds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 StableApplicationBinaryInterface 11
3 TheVeryHighLevelLayer 13
4 ReferenceCounting 19
5 ExceptionHandling 21
5.1 Printingandclearing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.2 Raisingexceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.3 Issuingwarnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
5.4 Queryingtheerrorindicator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.5 SignalHandling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
5.6 ExceptionClasses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5.7 ExceptionObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5.8 UnicodeExceptionObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5.9 RecursionControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.10 StandardExceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
6 Utilities 33
6.1 OperatingSystemUtilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
6.2 SystemFunctions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
6.3 ProcessControl. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6.4 ImportingModules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6.5 Datamarshallingsupport. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.6 Parsingargumentsandbuildingvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.7 Stringconversionandformatting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.8 Reflection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.9 Codecregistryandsupportfunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
7 AbstractObjectsLayer 53
7.1 ObjectProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.2 NumberProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
7.3 SequenceProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
7.4 MappingProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
i
7.5 IteratorProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
7.6 BufferProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
7.7 OldBufferProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
8 ConcreteObjectsLayer 71
8.1 FundamentalObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
8.2 NumericObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
8.3 SequenceObjects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
8.4 ContainerObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
8.5 FunctionObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
8.6 OtherObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
9 Initialization,Finalization,andThreads 125
9.1 Initializingandfinalizingtheinterpreter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
9.2 Process-wideparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
9.3 ThreadStateandtheGlobalInterpreterLock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
9.4 Sub-interpretersupport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
9.5 AsynchronousNotifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
9.6 ProfilingandTracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
9.7 AdvancedDebuggerSupport. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
10 MemoryManagement 139
10.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
10.2 RawMemoryInterface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
10.3 MemoryInterface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
10.4 CustomizeMemoryAllocators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
10.5 CustomizePyObjectArenaAllocator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
10.6 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
11 ObjectImplementationSupport 145
11.1 AllocatingObjectsontheHeap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
11.2 CommonObjectStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
11.3 TypeObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
11.4 NumberObjectStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
11.5 MappingObjectStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
11.6 SequenceObjectStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
11.7 BufferObjectStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
11.8 AsyncObjectStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
11.9 SupportingCyclicGarbageCollection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
12 APIandABIVersioning 171
A Glossary 173
B Aboutthesedocuments 185
B.1 ContributorstothePythonDocumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
C HistoryandLicense 187
C.1 Historyofthesoftware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
C.2 TermsandconditionsforaccessingorotherwiseusingPython . . . . . . . . . . . . . . . . . . . . . 188
C.3 LicensesandAcknowledgementsforIncorporatedSoftware . . . . . . . . . . . . . . . . . . . . . . 191
D Copyright 205
Index 207
ii
ThePython/CAPI,Release3.5.1
This manual documents the API used by C and C++ programmers who want to write extension modules or embed
Python. Itisacompaniontoextending-index,whichdescribesthegeneralprinciplesofextensionwritingbutdoesnot
documenttheAPIfunctionsindetail.
CONTENTS 1
ThePython/CAPI,Release3.5.1
2 CONTENTS
CHAPTER
ONE
INTRODUCTION
TheApplicationProgrammer’sInterfacetoPythongivesCandC++programmersaccesstothePythoninterpreterat
a variety of levels. The API is equally usable from C++, but for brevity it is generally referred to as the Python/C
API.TherearetwofundamentallydifferentreasonsforusingthePython/CAPI.Thefirstreasonistowriteextension
modules for specific purposes; these are C modules that extend the Python interpreter. This is probably the most
commonuse. ThesecondreasonistousePythonasacomponentinalargerapplication; thistechniqueisgenerally
referredtoasembeddingPythoninanapplication.
Writinganextensionmoduleisarelativelywell-understoodprocess,wherea“cookbook”approachworkswell.There
areseveraltoolsthatautomatetheprocesstosomeextent. WhilepeoplehaveembeddedPythoninotherapplications
sinceitsearlyexistence,theprocessofembeddingPythonislessstraightforwardthanwritinganextension.
Many API functions are useful independent of whether you’re embedding or extending Python; moreover, most ap-
plicationsthatembedPythonwillneedtoprovideacustomextensionaswell,soit’sprobablyagoodideatobecome
familiarwithwritinganextensionbeforeattemptingtoembedPythoninarealapplication.
1.1 Include Files
Allfunction,typeandmacrodefinitionsneededtousethePython/CAPIareincludedinyourcodebythefollowing
line:
#include "Python.h"
This implies inclusion of the following standard headers: <stdio.h>, <string.h>, <errno.h>,
<limits.h>,<assert.h>and<stdlib.h>(ifavailable).
Note: SincePythonmaydefinesomepre-processordefinitionswhichaffectthestandardheadersonsomesystems,
youmustincludePython.hbeforeanystandardheadersareincluded.
All user visible names defined by Python.h (except those defined by the included standard headers) have one of the
prefixesPyor_Py. Namesbeginningwith_PyareforinternalusebythePythonimplementationandshouldnotbe
usedbyextensionwriters. Structuremembernamesdonothaveareservedprefix.
Important:usercodeshouldneverdefinenamesthatbeginwithPyor_Py.Thisconfusesthereader,andjeopardizes
theportabilityoftheusercodetofuturePythonversions,whichmaydefineadditionalnamesbeginningwithoneof
theseprefixes.
The header files are typically installed with Python. On Unix, these are located in the directories
prefix/include/pythonversion/ and exec_prefix/include/pythonversion/, where prefix
and exec_prefix are defined by the corresponding parameters to Python’s configure script and version is
sys.version[:3]. OnWindows,theheadersareinstalledinprefix/include,whereprefixistheinstal-
lationdirectoryspecifiedtotheinstaller.
3
ThePython/CAPI,Release3.5.1
Toincludetheheaders,placebothdirectories(ifdifferent)onyourcompiler’ssearchpathforincludes. Donotplace
theparentdirectoriesonthesearchpathandthenuse#include <pythonX.Y/Python.h>; thiswillbreakon
multi-platform builds since the platform independent headers under prefix include the platform specific headers
fromexec_prefix.
C++usersshouldnotethatthoughtheAPIisdefinedentirelyusingC,theheaderfilesdoproperlydeclaretheentry
pointstobeextern "C",sothereisnoneedtodoanythingspecialtousetheAPIfromC++.
1.2 Objects, Types and Reference Counts
MostPython/CAPIfunctionshaveoneormoreargumentsaswellasareturnvalueoftypePyObject*. Thistypeis
apointertoanopaquedatatyperepresentinganarbitraryPythonobject. SinceallPythonobjecttypesaretreatedthe
samewaybythePythonlanguageinmostsituations(e.g.,assignments,scoperules,andargumentpassing),itisonly
fittingthattheyshouldberepresentedbyasingleCtype.AlmostallPythonobjectsliveontheheap:youneverdeclare
anautomaticorstaticvariableoftypePyObject,onlypointervariablesoftypePyObject*canbedeclared. The
soleexceptionarethetypeobjects;sincethesemustneverbedeallocated,theyaretypicallystaticPyTypeObject
objects.
AllPythonobjects(evenPythonintegers)haveatypeandareferencecount. Anobject’stypedetermineswhatkindof
objectitis(e.g.,aninteger,alist,orauser-definedfunction;therearemanymoreasexplainedintypes). Foreachof
thewell-knowntypesthereisamacrotocheckwhetheranobjectisofthattype;forinstance,PyList_Check(a)
istrueif(andonlyif)theobjectpointedtobyaisaPythonlist.
1.2.1 Reference Counts
Thereferencecountisimportantbecausetoday’scomputershaveafinite(andoftenseverelylimited)memorysize;it
countshowmanydifferentplacestherearethathaveareferencetoanobject. Suchaplacecouldbeanotherobject,or
aglobal(orstatic)Cvariable,oralocalvariableinsomeCfunction. Whenanobject’sreferencecountbecomeszero,
theobjectisdeallocated. Ifitcontainsreferencestootherobjects,theirreferencecountisdecremented. Thoseother
objectsmaybedeallocatedinturn,ifthisdecrementmakestheirreferencecountbecomezero,andsoon. (There’san
obviousproblemwithobjectsthatreferenceeachotherhere;fornow,thesolutionis“don’tdothat.”)
Referencecountsarealwaysmanipulatedexplicitly.ThenormalwayistousethemacroPy_INCREF()toincrement
an object’s reference count by one, and Py_DECREF() to decrement it by one. The Py_DECREF() macro is
considerablymorecomplexthantheincrefone,sinceitmustcheckwhetherthereferencecountbecomeszeroandthen
causetheobject’sdeallocatortobecalled.Thedeallocatorisafunctionpointercontainedintheobject’stypestructure.
Thetype-specificdeallocatortakescareofdecrementingthereferencecountsforotherobjectscontainedintheobject
ifthisisacompoundobjecttype,suchasalist,aswellasperforminganyadditionalfinalizationthat’sneeded.There’s
nochancethatthereferencecountcanoverflow;atleastasmanybitsareusedtoholdthereferencecountasthereare
distinctmemorylocationsinvirtualmemory(assumingsizeof(Py_ssize_t) >= sizeof(void*)). Thus,
thereferencecountincrementisasimpleoperation.
Itisnotnecessarytoincrementanobject’sreferencecountforeverylocalvariablethatcontainsapointertoanobject.
In theory, the object’s reference count goes up by one when the variable is made to point to it and it goes down by
onewhenthevariablegoesoutofscope. However,thesetwocanceleachotherout,soattheendthereferencecount
hasn’tchanged. Theonlyrealreasontousethereferencecountistopreventtheobjectfrombeingdeallocatedaslong
asourvariableispointingtoit. Ifweknowthatthereisatleastoneotherreferencetotheobjectthatlivesatleastas
longasourvariable,thereisnoneedtoincrementthereferencecounttemporarily. Animportantsituationwherethis
arisesisinobjectsthatarepassedasargumentstoCfunctionsinanextensionmodulethatarecalledfromPython;the
callmechanismguaranteestoholdareferencetoeveryargumentforthedurationofthecall.
However, a common pitfall is to extract an object from a list and hold on to it for a while without incrementing its
referencecount. Someotheroperationmightconceivablyremovetheobjectfromthelist,decrementingitsreference
count and possible deallocating it. The real danger is that innocent-looking operations may invoke arbitrary Python
4 Chapter1. Introduction
ThePython/CAPI,Release3.5.1
codewhichcoulddothis;thereisacodepathwhichallowscontroltoflowbacktotheuserfromaPy_DECREF(),
soalmostanyoperationispotentiallydangerous.
A safe approach is to always use the generic operations (functions whose name begins with PyObject_,
PyNumber_, PySequence_ or PyMapping_). These operations always increment the reference count of the
objecttheyreturn. ThisleavesthecallerwiththeresponsibilitytocallPy_DECREF()whentheyaredonewiththe
result;thissoonbecomessecondnature.
ReferenceCountDetails
ThereferencecountbehavioroffunctionsinthePython/CAPIisbestexplainedintermsofownershipofreferences.
Ownership pertains to references, never to objects (objects are not owned: they are always shared). “Owning a
reference”meansbeingresponsibleforcallingPy_DECREFonitwhenthereferenceisnolongerneeded. Ownership
canalsobetransferred,meaningthatthecodethatreceivesownershipofthereferencethenbecomesresponsiblefor
eventuallydecref’ingitbycallingPy_DECREF()orPy_XDECREF()whenit’snolongerneeded—orpassingon
thisresponsibility(usuallytoitscaller). Whenafunctionpassesownershipofareferenceontoitscaller,thecalleris
saidtoreceiveanewreference. Whennoownershipistransferred,thecallerissaidtoborrowthereference. Nothing
needstobedoneforaborrowedreference.
Conversely,whenacallingfunctionpassesinareferencetoanobject,therearetwopossibilities: thefunctionsteals
areferencetotheobject,oritdoesnot. Stealingareferencemeansthatwhenyoupassareferencetoafunction,that
functionassumesthatitnowownsthatreference,andyouarenotresponsibleforitanylonger.
Fewfunctionsstealreferences;thetwonotableexceptionsarePyList_SetItem()andPyTuple_SetItem(),
which steal a reference to the item (but not to the tuple or list into which the item is put!). These functions were
designedtostealareferencebecauseofacommonidiomforpopulatingatupleorlistwithnewlycreatedobjects;for
example, thecodetocreatethetuple (1, 2, "three") couldlooklikethis(forgetting abouterrorhandlingfor
themoment;abetterwaytocodethisisshownbelow):
PyObject *t;
t = PyTuple_New(3);
PyTuple_SetItem(t, 0, PyLong_FromLong(1L));
PyTuple_SetItem(t, 1, PyLong_FromLong(2L));
PyTuple_SetItem(t, 2, PyUnicode_FromString("three"));
Here, PyLong_FromLong() returns a new reference which is immediately stolen by PyTuple_SetItem().
Whenyouwanttokeepusinganobjectalthoughthereferencetoitwillbestolen,usePy_INCREF()tograbanother
referencebeforecallingthereference-stealingfunction.
Incidentally, PyTuple_SetItem() is the only way to set tuple items; PySequence_SetItem() and
PyObject_SetItem() refuse to do this since tuples are an immutable data type. You should only use
PyTuple_SetItem()fortuplesthatyouarecreatingyourself.
EquivalentcodeforpopulatingalistcanbewrittenusingPyList_New()andPyList_SetItem().
However, in practice, you will rarely use these ways of creating and populating a tuple or list. There’s a generic
function, Py_BuildValue(), that can create most common objects from C values, directed by a format string.
For example, the above two blocks of code could be replaced by the following (which also takes care of the error
checking):
PyObject *tuple, *list;
tuple = Py_BuildValue("(iis)", 1, 2, "three");
list = Py_BuildValue("[iis]", 1, 2, "three");
It is much more common to use PyObject_SetItem() and friends with items whose references you are only
borrowing,likeargumentsthatwerepassedintothefunctionyouarewriting. Inthatcase,theirbehaviourregarding
1.2. Objects,TypesandReferenceCounts 5
ThePython/CAPI,Release3.5.1
referencecountsismuchsaner,sinceyoudon’thavetoincrementareferencecountsoyoucangiveareferenceaway
(“haveitbestolen”).Forexample,thisfunctionsetsallitemsofalist(actually,anymutablesequence)toagivenitem:
int
set_all(PyObject *target, PyObject *item)
{
Py_ssize_t i, n;
n = PyObject_Length(target);
if (n < 0)
return -1;
for (i = 0; i < n; i++) {
PyObject *index = PyLong_FromSsize_t(i);
if (!index)
return -1;
if (PyObject_SetItem(target, index, item) < 0) {
Py_DECREF(index);
return -1;
}
Py_DECREF(index);
}
return 0;
}
The situation is slightly different for function return values. While passing a reference to most functions does not
changeyourownershipresponsibilitiesforthatreference,manyfunctionsthatreturnareferencetoanobjectgiveyou
ownership of the reference. The reason is simple: in many cases, the returned object is created on the fly, and the
referenceyougetistheonlyreferencetotheobject. Therefore,thegenericfunctionsthatreturnobjectreferences,like
PyObject_GetItem()andPySequence_GetItem(),alwaysreturnanewreference(thecallerbecomesthe
ownerofthereference).
Itisimportanttorealizethatwhetheryouownareferencereturnedbyafunctiondependsonwhichfunctionyoucall
only—theplumage(thetypeoftheobjectpassedasanargumenttothefunction)doesn’tenterintoit! Thus,ifyou
extractanitemfromalistusingPyList_GetItem(),youdon’townthereference—butifyouobtainthesame
itemfromthesamelistusingPySequence_GetItem()(whichhappenstotakeexactlythesamearguments),you
doownareferencetothereturnedobject.
Hereisanexampleofhowyoucouldwriteafunctionthatcomputesthesumoftheitemsinalistofintegers; once
usingPyList_GetItem(),andonceusingPySequence_GetItem().
long
sum_list(PyObject *list)
{
Py_ssize_t i, n;
long total = 0, value;
PyObject *item;
n = PyList_Size(list);
if (n < 0)
return -1; /* Not a list */
for (i = 0; i < n; i++) {
item = PyList_GetItem(list, i); /* Can't fail */
if (!PyLong_Check(item)) continue; /* Skip non-integers */
value = PyLong_AsLong(item);
if (value == -1 && PyErr_Occurred())
/* Integer too big to fit in a C long, bail out */
6 Chapter1. Introduction