Nikita The Spider * Python IPC - Modules shm and shm_wrapper

## IPC with Python - System V Shared Memory and Semaphores

This describes the module `shm` (written by Vladimir Marangozov) that gives access to System V shared memory and semaphores on *nix systems as well the module `shm_wrapper` (written by me) which is a companion module that offers more Pythonic access. Windows users, you're out of luck here; these modules only work on platforms that support System V shared objects. Most *nixes do (including OS X) but Windows does not.

You can download shm version 1.2.2 which contains the module `shm_wrapper`, the module `shm`, installation instructions and sample code. You might also want to read about some known bugs.

## The Modules

There's two modules available here. `shm` is essential. It is a single C language file that compiles into a Python module. The module's functions and features map fairly closely to the system calls like `shmctl`, but with nicer names.

The module `shm_wrapper` offers easier-to-use, higher-lever access to `shm`'s features. For example, `shm` memory objects support reading the object's permissions through the `perm` attribute but you must set them by calling `setperm()`. By contrast, `shm_wrapper` exposes a simple gettable/settable attribute called `permissions`.

`shm_wrapper` also allows you to ignore the messy details surrounding keys and ids for the objects. Since the fact that `ftok()` is broken on most (all?) modern systems makes keys not very useful, there's no reason for you to spend time messing with them if you can avoid doing so.

### Which Should You Use?

If you're an old (or young) Sys V hacker, you might prefer `shm`'s "closer to the metal" feel. Otherwise, use `shm_wrapper`.

## `shm_wrapper`

This module provides classes that act as handles to shared memory and semaphores as well as functions to create and destroy each. Memory segments and semaphores are distinguished from one another by a guaranteed-unique key that this module generates automatically when the object is created.

### `shm_wrapper` Functions

create_memory(size, permissions = 0666, InitCharacter = ' ')
  Creates a shared memory segment and returns a SharedMemoryHandle instance (described below). Each byte of the memory will be initialized to InitCharacter. If you pass a multibyte character for that parameter, the results are undefined. You can destroy the memory either by calling `remove_memory()` or by calling the `.remove()` method on a handle to said memory.
remove_memory(key)
  Removes (releases) the shared memory identified by `key`. Raises `KeyError` if no shared memory has that key.
create_semaphore(InitialValue = 1, permissions = 0666)
  Creates a semaphore and returns a SemaphoreHandle instance (described below). You can destroy the semaphore either by calling `remove_semaphore()` or by calling the `.remove()` method on a handle to said semaphore.
remove_semaphore(key)
  Destroys the semaphore identified by `key`. Raises `KeyError` if no semaphore has that key.

### `shm_wrapper` Class - SharedMemoryHandle

This is a handle to a piece of shared memory that allows you to read and write to the memory and manipulate its attributes. Important methods and attributes are below. Not all of the attributes are documented here; refer to the module itself for information on some of the more obscure ones.

read(NumberOfBytes = 0, offset = 0)
  Reads the specified number of bytes from position offset and returns a string.
write(s, offset = 0)
  Writes the string s to position offset.
remove()
  Removes (releases) the shared memory. If any process tries to access it after calling `.remove()`, the results are system-dependent (but will probably be unpleasant).
key (read only)
  The integer key that identifies this segment. By passing the key to other processes, the process that created the shared memory gives them the ability to find it easily.
size (read only)
  The size of the segment in bytes.
permissions
  The permissions on this memory.

### `shm_wrapper` Class - SemaphoreHandle

This is a handle to a semaphore that allows you to wait on the semaphore and also read and write to its attributes. Important methods and attributes are below. Not all of the attributes are documented here; refer to the module itself for information on some of the more obscure ones.

P()
  Blocks while the semaphore is zero and, once unblocked, decrements it. Stands for prolaag or probeer te verlagen (try to decrease).
V()
  Increments the semaphore. Stands for verhoog (increase).
Z()
  Blocks until zee zemaphore is zero.
remove()
  Removes (destroys) the semaphore. If any process tries to access it after calling `.remove()`, the results are system-dependent (but will probably be unpleasant).
key (read only)
  The integer key that identifies this semaphore. By passing the key to other processes, the process that created the semaphore gives them the ability to find it easily.
value
  The semaphore's value.
permissions
  The permissions on this semaphore.
blocking
  Turns blocking mode on or off. In non-blocking mode, `P()` and `Z()` will merely raise `shm.error` if the semaphore is unavailable.
undo
  Turns the SEM_UNDO flag on or off, but I believe support for SEM_UNDO is OS dependent.
WaitingForZero (read only)
  The number of processes waiting for this semaphore to hit zero (i.e. waiting in a call to `.Z()`).
WaitingForNonZero (read only)
  The number of processes waiting for this semaphore to hit non-zero (i.e. waiting in a call to `.P()`).

## `shm`

Vladimir Marangozov's `shm` module is the core of the onion, so to speak. It is a single C language file that compiles into a Python module. The module's functions and features map fairly closely to the system calls like `shmctl`, but with nicer names. The module has the following features:

### `shm` Functions

create_memory(Key, Size, [Perm=0666])
  Creates a new shared memory segment and returns an shm.Memory object (described below). Fails if the key is not unique.
create_semaphore(Key, [Value=1, Perm=0666])
  Creates a new semaphore and returns an shm.Semaphore object (described below). Fails if the key is not unique.
ftok(string Path, int ProjId)
  Calls the system's `ftok()` which is supposed to map each filename to a unique integer but is probably broken on your operating system.
getshmid(Key)
  Maps a memory key to an integer id. Raises KeyError if the key doesn't exist.
getsemid(Key)
  Maps a semaphore key to an integer id. Raises KeyError if the key doesn't exist.
memory(Shmid)
  Returns an shm.Memory handle to a shared memory segment if one exists with the id `Shmid`, otherwise returns a memory object with the `addr` attribute set to 0. I haven't yet figured out what purpose this serves.
memory_haskey(Key)
  True if a shared memory segment with the given key exists.
remove_memory(Shmid)
  Destroys (removes from the system) the shared memory segment identified by `Shmid`.
remove_semaphore(Shmid)
  Destroys (removes from the system) the semaphore identified by `Shmid`.
semaphore(Semid)
  Returns an shm.Semaphore handle to a semaphore if one exists with the id `Semid`, otherwise returns a semaphore object that points to ???. I haven't yet figured out what purpose this serves.
semaphore_haskey(Key)
  True if a semaphore with the given key exists.

### `shm` Errors

The `shm` module defines its own `shm.error`.

### `shm` Class - Memory

A Memory object is a handle to a chunk of shared memory created by `shm.create_memory()`. Memory objects have these attributes and methods:

attach([addr = 0, how = 0])
  Attaches to the memory segment. See your system's man page for `shmat` for valid parameter values
detach()
  Detaches from the memory segment
read(NumberOfBytesToRead, [offset = 0]), returns a string
  Reads bytes from the memory segment.
write(s, [offset = 0])
  Writes the string s to the memory segment
setperm(perm)
  Sets the permissions
setgid(gid)
  Sets the gid
setuid(uid)
  Sets the uid
shmid (read only)
  The segment's id
key (read only)
  The segment key or IPC_PRIVATE (0)
size (read only)
  The segment's size in bytes
attached (read only)
  A Boolean that reports whether or not the memory has been attached by a call to `.attach()`
nattch (read only)
  The number of processes currently attached to this segment
perm (read only)
  The segment's permissions
addr (read only)
  Attachment address in the process address space
cgid (read only)
  The gid of the creator
cpid (read only)
  The pid of the creator
cuid (read only)
  The uid of the creator
gid (read only)
  The gid of the owner
uid (read only)
  The uid of the owner
lpid (read only)
  The pid of the last process to touch(?) this object

### `shm` Class - Semaphore

A Semaphore object is a handle to a semaphore created by `shm.create_semaphore()`. Semaphore objects have these attributes and methods:

P()
  Blocks while .val == 0; then decrements before returning. Stands for prolaag or probeer te verlagen (try to decrease).
V()
  Increments .val. Stands for verhoog (increase).
Z()
  Blocks until .val == 0.
setblocking(block)
  Turns blocking on or off.
setundo(undo)
  Turns the undo flag on or off. (See your system's man page notes about `semop` and `SEM_UNDO`.)
setval(Value)
  Sets the value.
setperm(perm)
  Sets the permissions
setgid(gid)
  Sets the gid
setuid(uid)
  Sets the uid
semid (read only)
  The sempahore's id
key (read only)
  The sempahore key or IPC_PRIVATE (0)
blocking (read only)
  A Boolean that reports whether or not blocking mode is on or off.
cgid (read only)
  The gid of the creator
cuid (read only)
  The uid of the creator
gid (read only)
  The gid of the owner
uid (read only)
  The uid of the owner
lpid (read only)
  The pid of the last process to touch(?) this object
ncnt (read only)
  The number of processes waiting for this semaphore's value to become > 0.
perm (read only)
  The semaphore's permissions
val (read only)
  The value of the semaphore's counter.
zcnt (read only)
  The number of processes waiting for this semaphore's value to become = 0.

### Version History

- Current – 1.2.2 (11 May 2008) – Changed the __del__() method in shm_wrapper's SharedMemoryHandle object to avoid referring to the shm module since that module might not be available during deletion. Thanks to René Neumann for the bug report and patch.
- 1.2.1 (11 Apr 2008) – No code change. I forgot to update the version string in version 1.2's setup.py, so that version installed itself claiming to be 1.1.4 even though it wasn't. That's now corrected. I also added the missing "INSTALL" file. Thanks to Jeff Schwab for letting me know it was missing.
- 1.2 (27 Jan 2008) – Now thread-friendly! Patch provided by James Teh (thanks!). Also fixed some int/long/ulong sloppiness pointed out by kAbY that could cause crashes on 64-bit platforms.
- 1.1.4 (10 Jan 2008) –
  - Changed setup.py to use Python 2.2-friendly syntax.
  - Undid the "fix" I'd applied in 1.1.3 and replaced it with code that properly detects which flavor the ipc_perm struct your system has. Thanks to Aaron Lindsey and Bill Codington.
- 1.1.3 (23 November 2007) – Changed #define of key to _key instead of __key for OS X Leopard. Thanks to Bill Hart.
- 1.1.2 (18 November 2007) – `shm_wrapper` now installed by default since it is the recommended interface for using this code. Fixed an implicit cast of size_t to unsigned int in the repr function that was exposed by compiling against Python 2.5. Also updated setup.py to be a bit nicer.
- 1.1.1 (5 March 2007) – Updated setup.py with compile options that work for Linux users.
- 1.1 (8 February 2007) – Rereleased `shm` after it had languished a bit in recent years. This version contains a number of fixes and changes as well as the new companion module `shm_wrapper`.
  - Created `shm_wrapper` to give a more consistent and Pythonic interface.
  - Fixed a memory leak in `PyShm_memory()` and `PyShm_semaphore()` spotted by Jon Corbet.
  - Fixed a memory leak in `PyShm_create_memory()` and `PyShm_create_semaphore()` related to inlining `Py_BuildValue()`.
  - Fixed a typo that caused `xxx_setuid()` to set the group id instead.
  - Changed `PyMem_DEL` to `PyObject_Del` in the `_dealloc` functions for Python 2.5. Thanks to Jason Tackaberry/Pierre Ossman.
  - Fixed `sprintf()` in `PyShmMemory_repr()` to use the correct format.
  - Added the ability to read the blocking mode of a semaphore.
  - Changed the function prototypes from K&R style so that compiling with `-Wstrict-prototypes` is now clean.
  - Added an #ifdef for happy compiles under FreeBSD.
  - Added a setup.py file.
- 1.0 (sometime in the 1990s) - Original version by Vladimir Marangozov.

### Interesting Tools

Many systems (although not some versions of OS X) come with `ipcs` and `ipcrm`. The former shows existing shared memory, semaphores and message queues on your system and the latter allows you to remove them.

### SHM and Threads

As of version 1.2, `shm` should be safe to use in threaded applications. (Previous versions were not).

### Sample Code

The tarball includes code demonstrating the use of both `shm_wrapper` and `shm`. The demo code comes in the form of two complementary applications (two apps for each demo = a total of four apps). The apps are called Mrs. Premise and Mrs. Conclusion (as in, "Four hours to bury a cat?") and they converse with one another through shared memory. Run Mrs. Premise in one terminal and Mrs. Conclusion in another.

The conversation starts with Mrs. Premise creating the shared memory and then seeding it with a random string. Mrs. Conclusion then calculates the md5 hash of this string and writes that back into the shared memory. Mrs. Premise calculates the md5 hash of that string and writes it to the shared memory, and so it goes back and forth for as many iterations as are specified. Using md5 hashes allows Mrs. Premise and Mrs. Conclusion to verify that the other process correctly read what was written because it makes the response predictable. This is important for detecting memory corruption (see below).

You can specify whether or not you want the flow of the conversation to be controlled with a semaphore. (This option as well as the number of iterations are specified in `DemoConstants.py`.) If you opt not to use the semaphore, memory corruption (i.e. a "simultaneous" write by both processes to the shared memory) will probably occur if you run enough iterations. On my test systems -- a G4 Powerbook and a PIII running FreeBSD -- memory corruption always happened in less than 5000 iterations. If you use the semaphore, memory corruption will not occur. When either process detects corruption, it raises an `AssertionError`.

In addition to demonstrating the use of `shm_wrapper` and `shm`, this demo also illustrates why one needs to be careful with shared memory programming. Consider that even without semaphores these two processes writing to the same bit of memory as fast as they can still require thousands of iterations to step on one another. Now imagine a similarly infrequent bug, but instead of one that's caused by two deliberately careless demo programs, imagine one buried in thousands of lines of your code that causes Some Random Event every two weeks or so. You don't want to have to track down a bug like that. Speaking of bugs...

### Known Bugs

Bugs? My code never has bugs! However, there is a suboptimal anomoly...

If I'm correctly interpreting what I see when I run `top`, the newest version of `shm` still leaks memory during the create/destroy cycle of both Memory and Semaphore objects. This happens despite plugging several existing memory leaks and despite the fact that Python's garbage collector doesn't report anything amiss. The leak is tiny (about 12 or 13 bytes per create/destroy cycle) and it is released when the Python process ends, so this will only be a problem if you have a long-running process that creates and destroys lots of these objects. Note that the leaks occur only when creating (or destroying) new objects. Getting a handle to an existing object doesn't cause a problem.

I created a simple program called MemoryLeakDemo.py that demonstrates the problem. Start it in one window with `top` running in another window and you can watch the python process eat memory.

### About `ftok` – Use It At Your Own Peril

Most sample code that you see involving the use of System V semaphores/shared memory recommends `ftok()` to generate an integer key that's guaranteed to be unique on that machine. The main convenience of this is that processes can get the key (and thus a handle to the shared memory or semaphore) simply by using a previously agreed-upon filename. However, most modern implementations of `ftok` don't guarantee that it returns a unique key, which means it creates a key that may or may not work. If it doesn't work you have to fall back on a reliable alternative method of key generation, so you might as well just use the alternative in the first place.

The operating systems affected by this include OS X, Open/Net/FreeBSD, and Linux. (See the BUGS or NOTES section of the referenced man pages.)

In my experience, `ftok` frequently returned duplicate keys for different files in the same directory on my G4 Powerbook. Rather than probing and documenting `ftok`'s limitations, I decided to just avoid it entirely and rely on Python's random number generator to provide keys for me. It's not as convenient as using a previously agreed-upon filename as a key, but buggy implementations of `ftok` don't (reliably) provide that either. One alternative is to have the creating process generate a random key and write that to a previously agreed-upon file.

### Potential Compile Problems

Please let me know if you find a platform for which shm doesn't compile as-is.

On some systems, you might get this error when compiling:

    shmmodule.c:186: error: redefinition of `union semun'

If you see this compile error, then you need to add your platform in setup.py to the if statement just above where HAVE_UNION_SEMUN is added.

Also, I suspect that BSD users (other than FreeBSD-ers) might see something like this:

    shmmodule.c:1375: error: `PAGE_SIZE' undeclared (first use in this function)

You probably need to `#include <machine/param.h>`. That's already done for FreeBSD thanks to an `#ifdef`.

Last but not least, version 1.1.4 introduced some fancy code in setup.py. The need for this comes from the ipc_perm struct in `ipc.h`. It contains a member that's called `key`, `_key` or `__key` depending on your system. Setup.py should now autodetect which version you have, but the autodetection code is new and hasn't been well exercised (or exorcised). If if doesn't work for you and you're in a rush to get shm working, you can hack shmmodule.c yourself. Find the definition of the `ipc_perm` structure in `/include/sys/ipc.h` (`/include/bits/ipc.h` on some systems) and note which version of key it uses in its definition. Then in shmmodule.c, replace the code instances of IPC_PERM_KEY_NAME with the variable name from ipc.h.