Vmaps - Python Arrays on mmap()

Version 1.1
Jan 22, 2002



    Normally you'd expect to see something about where to ask questions, report bugs, etc here; but first: some begging!

    This module was written partly to fill the author's need, the usual genesis of open projects. The versatile form of it before you, documented and released, far surpasses the original need; in attempt to bring joy to the persons whose questions "is there something like this?" going back yea unto the latter 90's are all there is to be found, searching for "python mmap atomic" and similar keywords. Those who need this, need it fairly badly; and some effort has been expended to make Vmaps useful (if not necessarily optimal) for everything the author can imagine.

    The reason for this extra effort wasn't altruism, rather a cold blooded plan to raise MONEY for the non-profit Snafu Center for Cognitive Science, which will be feeding the author as soon as it has any income. Most of the select group of users who have been itching to turn python loose on big problems on big machines should be able to afford a donation of what this software is worth to them. (Those who, like the author, are managing to work with larger systems by gosh and golly and "good lord is that the power bill?" may defer donations until they have some income.)

    Supporters will have preference when it comes to support in the more traditional sense. The author can be reached via email: dragon @ and may usually be found lurking in #lair on .


    This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

    This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.


    This module has been developed with Python 2.2 on:

    • Linux 2.2 / Glibc2 2.13 (redhat 6.2) - 2x Pentium Pro 512k @233MHz
    • Solaris 8 (SunOS 5.8 sun4d) SPARCcenter-2000 8x TI SuperSparc @85Mhz

    minimal yet successful testing has been done on:

    • FreeBSD 4.4-STABLE 1x AMD Duron @700Mhz

    Anything else; good luck, reports of failures and successes welcome: see the Support section.


    The latest version of the distribution should be available from:

    The files contained in the distribution are:

    • Vmaps.html - This documentation.
    • Vmapsmodule.c - The Vmaps module itself.
    • gpl.txt - a copy of the GPL, the license under which this software is released.
    • - An example program. (see
    • - The Disutils interface / installation program.

    The standard Python Disutils are used; so compiling and installing this module into your interpreter is minimally a matter of running:

      python install
    You probably want the optional code which this simple method won't get you; read on. Impatient types who just ran that command should skip ahead to the Tutorial now.

System Dependencies and Build Options

    The important options are to the build_ext command, where you may define a preprocessor symbol to get the atomic swap routines and thereby header locking. The module should build and run just fine without those routines; the header lock functionality (see Headers) will not be included, nor will the atswap() routines.

    For Intel:

      python  build_ext  -D ARCH_IA32  install

    For Sparc:

      python  build_ext  -D ARCH_SPARC -l rt  install

    The "-l rt" is needed to get the sched_yield() call on Solaris, does no harm on Linux/Intel, and should not be used on FreeBSD.

Other Compile Options

    When returning sequences, either lists or tuples may be generated. Lists are used by default, tuples should be perceptibly faster in use, depending on the use. To change what type of sequence is returned; see the NEWRSEQ and SETRSEQ defines near the top of the Vmapsmodule.c file, and also the final parameter to the CTOPY_ARRAY macros.

    The following defines are also available as module constants:


    ( 64 )

    The size to which the optional Vmap header (see VM_HEADER) is rounded off. Odd-size offsets can cause core dumps and segfaults when accessing 8byte floats on the SPARC, for example; and improperly aligned data access is astoundingly slow everywhere it seems.




Known Bugs

    This documentation while believed to be correct, isn't complete, and could be more clear. There are too many "TODO"s left in the Tutorial. The code contains no docstrings and few comments.

    This is relentlessly 32bit code. Expect horrible failures trying to use this module on a 64 bit system, if it even compiles. No attempt to make large file support work has been made.

    Switching a Vmap from VM_HEADER mode and back will cause problems; don't do that. The process of initializing a Vmap header for the first time could be smoothed out, too. Generally the two modes of operation are intended for different uses.

    There is less error checking than there needs to be in some places; and what errors are returned are not always sensible or informative.


    • Jan 12 2002 - Version 1.0
    • Jan 22 2002 - 1.1; added colget(), added missing errorchecking to atswap() (now it will tell you about out of bound element indices instead of possibly segfaulting).

Future Directions

    There need to be a lot more methods to operate on the data "in place".

    Assignments of arrays and other objects could be optimized, possibly by using the buffer interface to those objects.

    Optionalizing the error checking might be good for some minor speed improvements; develop with the error checking enabled and turn it off for real runs.

    More systems need to be supported, for donations of large memory SMP machines based on YOUR favorite CPU towards that end please see the Support section.

Vmaps Module Reference

    The module defines one function, which creates Vmap objects, and several integer constants to be used with those objects.

newmap ( fileno, size, [start, [mm_flags, [mm_prot, [vm_flags, [vm_type, [elsize]]]]]] )

    Creates a new Vmap instance. Accepts keyword arguments, plus these variables can be changed after the Vmap has been created (see Attributes). The Vmap is not necessarily ready to be accessed after creation, unless the VM_AUTOPEN flag is used, the Vmap's open() must be called first.

    fileno is the system file handle of a file opened with appropriate permissions. size is the number of bytes of the file to make addressable as memory, and may be smaller than the size of the file. If size is larger than the length of the file, accesses past the end of the file will probably fail. See mmap() documentation.

    start (default 0) is the offset into the file at which to begin the mapping. No alignment is necessary for this number. mm_flags (default DFL_MM_FLAGS) controls how mmap() governs the memory used for the mapping, whether swap space is reserved, etc: see Flags. mm_prot (default DFL_MM_PROT similarly sets the access permissions for the mapped region, whether data may be read or written; see mmap() Protection modes, below.

    vm_flags (default DFL_VM_FLAGS) is the initial FLAGS to give the Vmap object. See Flags, and the setflag(), clearflag(), and getflag() methods. vm_type (default Char) is the initial TYPE of the Vmap, and elsize (default 1) operates as with the astype() method.



    The size in bytes of a memory page on this system. The offset parameter of the raw_msync(), raw_madvise(), raw_mlock(), and raw_munlock() methods needs to be a multiple of this number; and those operations actually operate on whole pages. See also the elpage() method, for quickly determining what page an array element is on.

mmap() Protection modes


    Allows data to be read from the memory area.


    Allows data to be written to the mapped memory area.


    Allows execution of code in the mapped memory area. I don't think its possible to make use of this from python.

mmap() Flags


    All processes using this area will share the same memory; data written by one process is visible instantly to all other processes.


    Makes the mapping "Copy on Write"; each process has a private view of the data.


    Used with a file descriptor of -1, and the MAP_PRIVATE flag to acquire memory not tied to a file.


    Tells the OS not to reserve a swap page for every memory page mapped. See your system's references.

msync() Flags


    Sync mapped memory to (or from) disk, do not return from msync call until its done.


    Mark memory to be sync'd ASAP, return from call immediately.


    System dependant: The Solaris man page says:

    MS_INVALIDATE invalidates all cached copies of data in memory, so that further references to the pages will be obtained by the system from their backing storage locations. This operation should be used by applications that require a memory object to be in a known state.

    But Linux man pages say:

    MS_INVALIDATE asks to invalidate other mappings of the same file (so that they can be updated with the fresh values just written).

    This may mean the same thing.

madvise() Flags


    Normal memory management is to be performed.


    Advise the system that data in this region will be accessed in a random fasion. Minimal data will be read per access.


    Tells the system that the data is likely to be accessed only once, so it should try to free resources as soon as it can after that access.


    Inform the system that the data will be needed soon, so that it may begin reading as soon as possible.


    Inform the system that the data is no longer needed and that it may start freeing resources.


    Tell the system that the data is due to be overwritten. The system may actually dispose of the data, returning 0 filled pages on subsequent requests.

Vmap Access Types


    a single string.


    an array of strings, each SIZE bytes long. SIZE can be modified at any time, which changes the number of elements in the array without changing the number of bytes in the mmap's area. Access past the end of mmap'd address space should therefore be possible to do. See also the VM_DOFILL flag.



    an array of 4 byte long integers.


    an array of 8 byte C "long long" integers, as Python long integers.


    an array of 8 byte C doubles as Python Floats.


    an array consisting of SIZE 4 byte integers per element. IE, if SIZE ==3, each element is a sequence of 3 4 byte integers, using 12 bytes total per element.


    As above, based on 8 byte C long long integers as Python long ints.


    As above, based on 8 byte C doubles as Python long floats.

Vmap Flags


    A Vmap object does not call its own open() on instantation; it must normally be opened explixitly first. If this flag is set, however, open() is called auto-magically on access to the Vmap, and open need never be called at all.


    If this flag is not set, the Vmap will call its own close() after an access is finished, before returning to Python. You probably want to clear this flag only if VM_AUTOPEN is set; which combination gives completely transparent mmap()-ing / munmap()-ing on accesses only.


    When set this flag causes an madvise call to be done for the whole mapped area when it is opened (explicitly with the open() method or implicitly for the VM_AUTOPEN flag). The flag given to madvise is the Vmap instance's mm_advflags attribute. Any error return from the madvise() call is ignored and will not cause the open() to fail.


    If set, msync() with flags MS_SYNC will be called whenver the Vmap object is closed.


    If set, msync() with flags MS_ASYNC will be called whenver the Vmap object is closed. If neither this flag nor VM_SYNCLOSE are set, no msync() call is done by the Vmap object before munmap() is called, and thus data could be lost.


    If set, the timestamps returned by the times() method are updated on accesses as appropriate. If not set, those timestamps are not touched and the associated overhead of keeping them is dispensed with.


    For the Char and FixChar access type, controls whether element and slice deletions, and "short" assignments will clear data (to the value specified with the fillwith attribute).


    For the FixLong access type, determines whether data is stored little endian (flag set) or big endian (unset).


    For the FixLong access type, if set the data is treated as signed integers; otherwise they are unsigned.


    Set for Vmaps that have headers. Data access will be offset into the mmap()'d area by the size of the header, which is read from the header; which obviously leads to potential problems initializing a new Vmap... for which case and others, see next flag.


    When set, do locking of the header. Unset, header access is unlocked and not safe if the data is shared. Depends on the atomic swap primitive (see Compile Options); if that isn't present, locks always succeed and header access is not safe for shared Vmaps. See also Header.


    If the VM_HEADER flag is set, and the header data in the mmap()'d area appears corrupted or a lock fails; if this flag is set the access attempt that caused the header to be read will fail with an exception, otherwise it will try to succeed in the possible absence of necessary data. In other words you almost always want this if you have a header.


    If set, the Vmap instance's access type is reset from the (shared) header on access, otherwise we access the mmap'd area with whatever access type the Vmap instance is using. The astype() method updates the TYPE and SIZE stored in the header if this flag is set.


    If set, the COUNT in the header is used as the "length" of the array, instead of the number of items in the array (bytes mapped / bytes per element). Use the count_get(), count_add(), and count_sub() methods to manipulate the COUNT.

Vmap Object Reference

    The Vmap object is a process local cache of the variables necessary to call mmap(), and access the resulting memory as an array of Python objects.

    The behavoir of a Vmap depends on the TYPE (see Types) of data it is set to return, and its flags (see Flags). Generally speaking it is a sequence.

    Vmap objects do not do concantation or repeating. They respond properly to len, regular item and slice addressing:

      x = aVmap[N],
      aVmap[N] = x,
      seq = aVmap[LO:HI],
      aVmap[LO:HI] = seq

    ... the 'in' operator: (except for FixFloat type)

      if x in aVmap:

    ... and item and slice deletion: (clears the element(s)).

      del aVmap[N],
      del aVmap[LO:HI]

    The following is necessarily somewhat generalized, read every sentence with an implicit "except for the exceptions". See the Tutorial for more detailed explainations of those exceptions.

    When assigning values to items or slices in a Vmap, the type of the Python value given must be appropriate for the Vmap's access type (ie, when the Vmap is being an array of integers (type Int), feed it python integers; when its being floats feed it python floats). Values of the wrong type in item assignment will be coerced by python where posisble, and raise an error otherwise. In sequence assignments, particularly in assigning an "item" of a 2 dimensional Vmap access type, a python value of incorrect type can cause the element to be silently set to 0, without an exception being raised.

    Sequence assignments can be fed anything convertable to a tuple. Normal Python access methods are used on incoming sequences, so for example array module arrays may be assigned to Vmap slices without special effort.

    Sequence returns are python lists of values of appropriate types (floats etc.) This data is a new, process local, unshared copy of the data in the mmap()'d memory. Assignments likewise are copying process local memory into the mmap()'d area. Either lists or tuples are returned throughout; which sequence type (list or tuple) can be changed at compile time with a minor edit of the source code.

    It needs noting that using len on a Vmap with the VM_HEADER flag set locks the header while size and COUNT data is read.

    Vmap objects have the "buffer interface" emulated from the stock Python mmap module, but no testing has been done of that. Its expected to work, barring the cases where it doesn't (example: a Vmap with flag VM_AUTOPEN set and VM_STAYOPEN cleared will not actually be closed after use by the buffer interface routines, as there's no way to know when the user is done using them).


    Read Only


    1 if the Vmap has been and is still open()ed and ready for access, 0 otherwise.


    The Vmap instance's Flags as a python integer.


    The Type of the Vmap instance, as py integer.


    The length of the Vmap; reported from local memory which in the case of a Vmap with a header and COUNT ( flags VM_HEADER | VM_HDRCOUNT ), may no longer be up to date.



    The size of the Vmap header in bytes, 0 if the VM_HEADER flag is not set, overheadbytes + bytes requested at the initializing open() call, rounded up to the nearest multiple of HEADALIGN bytes.


    The size of the internally maintained Vmap header, in bytes.

    Read Always / Write If Closed



            The parameters given to the newmap() call. May be written to if the Vmap is not open.

            Always Read / Write



            The byte (as python int) used to fill out FixChar elements when the VM_DOFILL flag is set.


            When the Vmap has a header (VM_HEADER set) and the VM_HDRLOCK flag is set, accesses to data contained in the header will wait for this many iterations. See Header.


            When iterating waiting for the header lock, if this attribute is non-zero; sched_yield() will be called after every unsuccessful iteration.

          Universal Methods

            Fundamental Methods

          open ( [init, [bytes]] )

            calls mmap(). If the VM_HEADER flag is set, reads the Vmap header and adjusts the Vmap instance's view of the data accordingly (VM_HDRTYPE, VM_HDRLEN, etc).

            If the init parameter is nonzero, the Vmap header will be initialized (ignoring any possible lock), using this instance's then current type and element size, and with the count set to 0.

            If the bytes parameter is nonzero, the Vmap header will be expanded to include the requested number of bytes of the mmap'ed data area. This space is not cleared, merely reserved; the pre-existing contents are still there. The space allocated will probably be a bit more than requested; the header is rounded out to an even alignment.

            Calling open on an already open Vmap re-reads the header and adjusts the instance's variables as needed.

          close ( )

          astype ( [type, [size]] )

            Reports and optionally changes this Vmap instance's access type. If size is supplied for an access type of fixed length elements (1d ints etc), it will be ignored. For the FixChar and FixLong access types, size is the number of bytes per element; for the numeric types it is the number of complete numbers per element... size == 3 for a 2d Float array results in elements of 24bytes, 3 items of 8 bytes each.

            If the VM_HEADER flag is set when this method is called, the type and element size information stored in the shared Vmap header is updated as well. The header lock will be held during that operation.

            Vmap Instance Information / Manipulation

          elsize ( )

            Returns the number of bytes a single array element occupies.

          elpage ( ndx )

            Returns the page number of array element ndx. This page number multiplied by the system PAGESIZE results in byte offsets suitable for feeding to the raw_madvise() and etc. methods.

          getflag ( flag )

            Return 1 if the Vmap instance has flag set; 0 otherwise.

          setflag ( flag )

            Set flag in this Vmap instance.

          clearflag ( flag )

            Clear flag in this Vmap instance.

          times ( [reset] )

            Returns a 5 tuple (read, write, sync, open, close) of the access timestamps kept when the VM_KEEPTS flag is set. The timestamps are all floats, as returned by time.time.

            If reset is non-zero; all the timestamps are reset to 0 after their existing values are retreived.

          schyield ( )

            Calls sched_yield(). No parameters and no errors.

            Data Manipulation

          find ( n, [lo, [hi]] )

            Search for data matching n (which will be coerced if possible by python rules for number types), optionally starting the search at element lo and terminating it at element hi, if those parameters are supplied. Returns -1 if an element matching n was not found, or the integer index into the array at which the data was found on success.

            When searching the 2d array types (Int2d, Long2d, Float2d), all items within an element (all columns in the row) are tested, and any which match return the element index; the particular column offset in the element which matched is not saved.

            NOTE: Not actually implemented for the FixLong type.

          sort ( [start, [count]] )

            Calls qsort() to re-order te elements of the array in place. 2D types are sorted by the value of first item (first column) only, but elements are kept intact. FixChar and FixLong types don't support sort, and sorting a Char Vmap, while functional, seems like something one would rarely need.

            The sort can be limited to a section of the data by providing an start, the element offset at which to begin (defaults to 0), and optionally count, how many elements to sort (defaults to "the rest").

          copyfrom ( source, [bytes, [istart, [sstart]]] )

            Use memmove() to quickly copy parts of Vmaps to each other (or themselves). source is the Vmap object to copy from; bytes (defaults to the smaller of this or source vmaps' size less starting offsets) is the number of bytes to copy, istart is the offset into this vmap at which to start writing data, sstart is the offset in the source vmap to start copying from.

            Both Vmaps are handled according to their flags then in effect.

          raw_string ( [offset, [size]] )

            Retreives a copy of data from the mmap'd area, starting at the optional offset (default 0) as a Python string of size bytes if given, or the size of the entire Vmap if not. This does not make any accounting for the Vmap header if there happens to be one.

            Raw System Call Interfaces

          raw_msync ( [flags, [offset, [size]]] )

            calls msync(), with optional flags (default MS_SYNC), and optional offset and size which default to 0, and the size of the Vmap.

            See PAGESIZE for restrictions on the offset and size parameters.

          raw_madvise ( [flags, [offset, [size]]] )

            Calls madvise(), with optional flags (defaults to the mm_advflags attribute), and optional offset and size which default to 0, and the size of the Vmap. See madvise() Flags for legal values for the flags parameter.

            The presence of the madvise() call on your system is implied by the existance of the MADV_NORMAL and other madvise() flags in the module. If those constants are not present, this method will probably raise an error. madvise() calls made per the VM_ADVOPEN flag will fail too; there the error is carefully ignored.

            See PAGESIZE for restrictions on the offset and size parameters.

          raw_mlock ( [offset, [size]] )

            (Only root can use this) Call mlock() on the region specified by offset and size which default to 0 and Vmap size.

            See PAGESIZE for restrictions on the offset and size parameters.

          raw_munlock ( [offset, [size]] )

            (Only root can use this) Call munlock() on the region specified by offset and size which default to 0 and Vmap size.

            See PAGESIZE for restrictions on the offset and size parameters.

            Header Methods

            See the Headers section.

          swapheader ( )

            Byteswaps the Vmap header (just the internally maintained data, the user header area is not touched).

            No lock is attempted.

          getheader ( )

            Returns as a string the contents of the "user space" in the Vmap header. These are the bytes reserved with optional parameter to open() when initializing the header. There may be more bytes than requested at initialization; the header is rounded out to keep array data aligned.

            The header lock is acquired and held while the data is copied.

          setheader ( astring )

            Sets the contents of the "user header" (allocated when the header was initialized). If astring is larger than the space allocated when the header was initialized, only the beginning of the string is copied. Any spare space in the user header is left untouched, no filling or clearing is done.

            The header lock is acquired and held while the data is copied.

          count_add ( toadd )

            Adds toadd to the COUNT kept in the Vmap header, and returns the old value. COUNT may not exceed the number of elements in the array as calculated using the Vmap instance's access type and size variables.

            The header lock is acquired and held for this operation.

          count_sub ( tosub )

            Subtracts tosub from the COUNT kept in the Vmap header, and returns the old value. COUNT will not be less than zero.

            The header lock is acquired and held for this operation.

          count_get ( toadd )

            Returns the COUNT kept in the Vmap header.

            The header lock is acquired and held for this operation.

          Type Specific Methods

          atswap ( el, new )

            AVAILABLE: Int

            Atomic swap of the value in array element el with new. Returns the value previously in el. Uses the arch dependant inline assembly language primitive to ensure actual serialization on SMP systems.

          atswap ( el, col, new )

            AVAILABLE: Int2d

            (2d version) Atomic swap of the value in array element el, col with new. Returns the value previously in el, col.

          byteswap ( [start, [count]] )

            AVAILABLE: Int, Long, Float, Int2d, Long2d, Float2d

            If the Vmap instance's access type is a 4byte or 8byte numeric, byteswap the data area of the Vmap. The optional parameters have the same meaning as in sort().

          resize ( size )

            AVAILABLE: FixChar, FixLong

            Resets the size of elements in a Vmap to size bytes each. If the VM_HEADER flag is set; the header will be updated (see astype()).

          setrange ( v, [lo, [hi]] )

            AVAILABLE: Int, Long, Float

            Set a range of array elements all to the same value v. If lo or hi are omitted they default to 0 and the size of the array, repectively.

          sumrange ( [lo, [hi]] )

            AVAILABLE: Int, Long, Float

            Return the sum of a range of elements. If lo or hi are omitted they default to 0 and the size of the array, repectively.

          cntbndrange ( [vlo, [vhi, [lo, [hi]]]] )

            AVAILABLE: Int, Long, Float

            "Count in Bounds, of range" ... Return the count of elements greater than vlo and less than vhi. These default to -1, and 1. If lo or hi are omitted they default to 0 and the size of the array, repectively.

          minmax ( [imin, [imax, [lo, [hi]]]] )

            AVAILABLE: Int, Long, Float

            Returns a two item sequence (minOfs, MaxOfs), containing the element index of the minimum and maximum values found in the specified portion of the array. imain and imax are the "initial" maximum and minimum values given to the search; if no element is smaller than imin or larger than imax the corresponding returned value will be -1.

          colxchg ( cola, colb, [lo, [hi]] )

            AVAILABLE: Int2d, Long2d, Float2d

            For each array element (row), exchanges the data at element column cola with the data at element column colb. Optional lo and hi parameters are the beginning and ending element offsets on which to operate, they default to 0 and len( array ) respectively.

          colget ( el, col )

          System Calls

            The specifics of these vary from system to system; see your system's documentation for the details that apply to you.


            man pages: Linux, Solaris,

            Creates "on demand" memory from a file (or "nowhere", see MAP_ANON); data is read from the file transparently at need, usually faster than is possible traditional file IO.



            man pages: Linux, Solaris,

            Marks mmap()'d memory to be written back to its associated file, or alternately flushes memory in favor of data on disk.


            man pages: Linux, Solaris,

            Gives the system advice on how a mmap()'d region should be handled for most efficient operation. Not functional in Linux 2.2; returns an error.


            man pages: Linux, Solaris,

            Copies (possibly overlapping) memory areas.


            man pages: Solaris,

            Relinquishes a timeslice, allowing another process on the system to use this CPU.


            man pages: Linux, Solaris,

            Disables swapping for an area of memory.


          Vmap Headers

            Vmap objects have internal support for a header area at the beginning of the mmap()'d area. The data stored there enables the TYPE, SIZE, and COUNT of the Vmap to be both persistient and shared amongst all processes using the mmap()'d area.

            If the VM_HDRFAIL flag is set, operations that require access to the header will fail if the header isn't initialized, or is locked by another process (assuming header locking has been enabled with the VM_HDRLOCK flag). The the VM_HDRFAIL flag is not set, failed header access doesn't raise an error, and operations fall back to the Vmap instance's memory of what the header looked like last time it was accessed.

            The behavoir of the locking is modified by the hlckspins and hlckyield attributes. The entire header is protected by a single mutex; whenever that lock is needed the Vmap will try hlckspins iterations before deciding the lock has failed. If the hlckyield attribute is non-zero, sched_yield() will be called on each iteration.

            Vmaps which have headers access that header before almost any Python operation, to get the proper length to error check with and so on. If the VM_HDRLOCK flag is set, this could affect performance. The lock is never held very long (as in, under 100 assembler instructions), but a process may be interrupted by the operating system "whenever" which can stuff things up.


            When the header is first initialized with the open() method, space can be requested for random user data, which is stored between the Vmap internal header and the array data. Even if no such space was requested there are likely to be a few bytes available from the HEADALIGN roundoff. This data can be set using the setheader() method, and retreived with the getheader() method. Access to this data is serialized according to the VM_HDRLOCK flag.



              If the Vmap instance's VM_HDRTYPE flag is set; the astype() method is automatically called on accesses to keep the Vmap instance's view of the data consistient with the other users of that data.


              If the Vmap instance's VM_HDRLEN flag is set, it tries to behave like a list. The len() of a Vmap with the VM_HDRLEN flag set is not the numbr of elements that can be fit into the mmap()'d area as usual, but this shared COUNT number. The count_get(), count_add(), and count_sub() methods manipulate this number, using locked access if the VM_HDRLOCK flag is set.

              The COUNT may not drop below 0, nor may it be raised above the number of elements possible to the mmap()'d area's size, figured using the access type and element sizes then in effect.

            Shared memory and SMP

              NOTE: A full explaination of the subtleties and best practices for parallel programming on shared memory systems is far beyond the scope of this document, and your author's capabilities. Read the following as a breif introduction to the field written by a newcomer, therefore.

              Sharing memory between processes on a single processor system is easy. No process will be modifying data concurrently with another, because only one is running at any given time. On multi-processor systems however, there will be concurrent processes, and there has to be some accomodation made to keep those processes from overwriting the same data at the same time. This requires hardware support.

              Read Stevens "Unix Network Programming: vol 2, Inter Process Communication". It goes into great detail and provides implementations of the POSIX semaphore, etc operations; but there's always a part missing: the mutexes. The implementaions given always go to OS services for that, for excellent reasons; the hardware support is different on every architechture. Using the nice abstract interfaces imposes limits, though.

              If you compiled the system specific code when you installed the Vmaps module, you have access to the Intel and Sparc hardware support needed for real, garunteed serialized access of shared memory. The atswap() method (atomic swap) retrieves an array element, and stores a new value into that element, as a single operation. If 2 CPUs do that swap at exactly the same moment, one or the other will get the data the first swapped in; the hardware ensures some sort of ordering as opposed to simultaneous access.

              Given this, building a mutex is easy:

                while 1:
                   dt = amap.atswap(0,-1)   # try lock
                   if dt !=-1: break        # we get it?
                   amap.schyield()          # no, let someone else have CPU
                # end while
                #  .... operate on locked data ....
                amap.atswap(0,0)            # unlock

              This code fragment illustrates using element 0 of a Int type Vmap as a mutex. A value of -1 in this element means "locked", any other value means "unlocked" (here 0). If another process is holding the lock, the swap here will return -1, and the schyield() call will be made to allow another process to use CPU, since this one will just be spinning, waiting for this lock. Swapping -1 with -1 obviously doesn't cause problems.

              If our swap returns something other than -1, we have acquired the mutex, locked the data, and other processes will be waiting for us to store a non -1 value back to signify that we are done( "unlock"). This example shows using another atswap() call; it could be done with normal access as well:

                #  .... operate on locked data ....
                amap[0] = 0                 # unlock

              Which way is better depends on what exactly you are doing. The atswap() call may be a bit quicker in most cases.

   (Piddly Purposeless Parallel Python)

              The distribution includes an example program This is a simple demonstration of multiple processes modifying shared memory. The program is organized for making minor changes and seeing the effect on run time; all the interesting variables are right at the top of the file.

              First, it creates a Int2d, shared Vmap to work with. The it spawns RunJobs children, and watches those do the actual operations. Once they have finished, it summarizes the data to make sure that all the increment operations the children were supposed to perform actually happened.

              The children immediately after being fork()ed begin to generate the random array indices upon which they will be operating. When RunJobs is larger than the number of CPUs in the system they will finish this task sequententially. Children that finish early sleep waiting for everyone to be ready to operate.

              Once all the child processes have their random numbers, all of them begin calling the Operate() function on random array elements. This function uses column 0 of an array element to lock the row, increments a random column, and unlocks the row. If the lock operation doesn't succeed immediately, a count of iterations through the "wait for lock" loop is incremented (Spins).

              When a child process has done its share of the total iterations specified (TotalIter), it prints out is "Spun" counts, and loops waiting for the other children to finish. A "spun 234 on 200" means that 234 total lock wait loops (schyield() calls) were made as a result of operations on 200 distinct Operate() calls.

              The child processes do not exit immediately, to make it easier to test the effects of msync() by the parent before and after the children close their inherited view of the shared Vmap. Changing the synchronization conditions should be easy.

            Vmap Tutorial

              Because of the multifacted nature of Vmap objects, they may seem more complex than they actually are. This demonstration and walk through should serve to detail some of the complexities and clarify the mysteries.

              Firstly, fire up your Python interpreter, and import the Vmaps module:

                Python 2.2 (#1, Dec 25 2001, 05:56:47) 
                >>> import Vmaps
                >>> v=Vmaps

              ... the "v" is to save typing "Vmaps" every time we refernce the module from here on (and guess what your humble author was doing Christmas morning).

            Annoymous Mappings

              Lets play with a quick annonymous mapping first. This is memory not backed by a file to which it can be msync()'d. The -1 file descriptor number (see fileno attribute) is conventionally used for annonymous mappings. The default flags to the full suite of parameters to the newmap() call will do for now.

                >>> amap = v.newmap(-1,8192)

            Basic Access

              ... this Vmap (named amap) is now ready for access. The default Type is Char, which is quite similar to a normal Python string:

                >>> len(amap)
                >>> amap[0]
                >>> amap[3]='A'
                >>> amap[:5]
                >>> amap[1:10] = ' '*9 
                >>> amap[:12]
                '\x00         \x00\x00'
                >>> ' ' in amap
                >>> amap.find('A')  # it was there but it isnt now
                >>> amap.find(' ')

              Fairly straightforward Python. Being an annonymous mapping, the initial contents are all zero bytes. The builtin Python in operator and the find() method are implemented using the same underlying search routine.

                >>> amap.close()
                >>> amap[0]
                Traceback (most recent call last):
                  File "<stdin>", line 1, in ?
                IOError: Vmap closed
                >>> amap[:12]

              This demonstrates two features. First, we can close() the Vmap, which in the case of this anonymous mapping erases the data we have stored so far. Second, we can re-open() the Vmap, rather than having to instantate a new one.

              We can use the builtin Python del keyword (putting some pattern in first):

                >>> amap[:] = '.' * 10
                >>> amap[:12]
                >>> del amap[4]
                >>> del amap[7:9]
                >>> amap[:12]

              Notice the slice assignment of 10 bytes to a slice of len( amap ). The Char and FixChar types are more forgiving of this treatment than the numeric types, as shall be demonstrated.

            FixChar Type

              The FixChar type is an array of fixed length strings. This demonstrates much easier than it explains, but first we must change the Vmap's access type:

                >>> amap.astype(v.FixChar,23) # returns the type code

              This uses astype() to tell our Vmap instance amap to access data as type FixChar; which it returns as confirmation that we gave it a proper type code. The Type codes are integers.

              Now when we access the Vmap, it returns 23 byte strings for each element:

                >>> amap[0]
                >>> len(amap[0])
                >>> amap[1]='xxxxxxxxxxxxx'
                >>> amap[0:2]  # returns a list of 2 23 byte strings
                ['....\x00..\x00\x00.\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'xxxxxxxxxxxxx\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00']
                >>> len(amap)  # 8192 / 23 =

              The len (number of elements) has changed too. Notice the pre-exisitng data is still there, our view of it has changed. The string we assigned to element 1 was not a full 23 bytes. What happenes to the rest of the space in that circumstance can be controlled with the VM_DOFILL flag (which is set by default), and the fillwith attribute.

                >>> amap.fillwith=67
                >>> del amap[1]  # see above, it was partly filled with 'x'
                >>> amap[0] = 'mmmm'
                >>> amap[:2]
                >>> amap[:3] # (spaces added to element 3 for html)
                '\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00']
                >>> amap.clearflag(v.VM_DOFILL)
                >>> amap.fillwith=0
                >>> amap[1]='xxxx' # without VM_DOFILL this time
                >>> amap[:2]

              The clearflag() method does just that. The fillwith attribute is copied to all bytes to be cleared; the VM_DOFILL flag determines whether "leftover" space and deletions cause data to be cleared or if not set, left alone.

            Numeric and 2D Access Types

              Now to show off some of the other access types:

                >>> amap.astype(v.Int)  
                >>> len(amap)  # 8192 / 4 bytes per integer
                >>> amap[0]    # 'mmmm' when cast to an integer =
                >>> amap[100:110]  # we haven't touched these yet
                [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
                >>> amap.setrange(1)  # this can be limited to just a portion of the array
                >>> amap[100:110]
                [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
                >>> amap[-4:]
                [1, 1, 1, 1]

              Again, our view of the existing data has changed, and ASCII interpreted as binary integers is usually not pretty. Using the setrange() method we can quickly clear the whole array to sensible values for 4byte integers.

                >>> amap.astype(v.Float)
                >>> amap[2] # Float is 8 byte C double 
                >>> amap.setrange(1,1,10)  
                >>> amap[:12]
                [2.121995791459338e-314, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.121995791459338e-314, 2.121995791459338e-314]

              Floating point (Float). Whee!

                >>> amap.astype(v.Float2d,3) # items, not bytes
                >>> len(amap) # 8192 / 24 (3 floats per element @8 bytes)
                >>> amap[:4]  # each element is a list of 3 floats
                [[2.121995791459338e-314, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 2.121995791459338e-314, 2.121995791459338e-314]]

              This is the data as a two dimensional array. Each element (row) has 3 columns of individual 8 byte double floating point numbers.

            FixLong Type and Unforgiving Acceptance

              Finally, the FixLong type deserves a demonstration, and the promised pickiness:

                >>> amap.astype(v.FixLong, 14) # bytes (the items here are bytes)
                >>> len(amap) # 8192 / 14 =
                >>> amap[:3]  # at this point we're psuedo-random :)
                [20282409608374036906816896499712L, 4872769679114799555279886951120896L, 74352564683758538135984603136L]
                >>> amap.setflag(v.VM_LLASG)  # how about if those are signed?
                >>> amap[:3]
                [20282409608374036906816896499712L, -319527179420028073250609378099200L, 74352564683758538135984603136L]
                >>> amap[0:4] = -1L    # nah
                Traceback (most recent call last):
                  File "<stdin>", line 1, in ?
                ValueError: Vmap assignment expected sequence

              Oops... Its all messy anyway. FixLong hasn't got setrange(), so:

                >>> amap.close()
                >>> amap.astype()  # no parameters just reports the current type
                >>> len(amap)      # the size is recalled as well
                >>> amap[0:4] = [-1L] * 4  # this works
                >>> amap[0:5]      
                [-1L, -1L, -1L, -1L, 0L]
                >>> amap[0:4] = [-1L] * 6  # this doesn't
                Traceback (most recent call last):
                  File "<stdin>", line 1, in ?
                IndexError: Vmap slice assignment is wrong size

              Oops again. The numeric types are a less forgiving about the sequences they accept and the values contained therein than are the Char and FixChar types:

                >>> amap[:5]   # that failure didn't change data, others will
                [-1L, -1L, -1L, -1L, 0L]
                >>> amap[0:4] = [ -1L, -1L, 1L, 8 ]  # this int wont be coerced
                Traceback (most recent call last):
                  File "<stdin>", line 1, in ?
                ValueError: Vmap assignment expected long
                >>> amap[0:4] = [-1L,-1L,1L,8L]    # there we go
                >>> amap[:5]
                [-1L, -1L, 1L, 8L, 0L]

            Stupid Bignum Tricks

              Now for a couple of probably non-useful tricks (recall that these are 14 byte integers):

                >>> amap.clearflag(v.VM_LLASG) # returns the Vmap's flags as integer
                >>> amap[:5]  # as unsigned, was: [-1L, -1L, 1L, 8L, 0L]
                [5192296858534827628530496329220095L, 5192296858534827628530496329220095L, 1L, 8L, 0L]
                >>> amap.setflag(v.VM_LLALE) # least significant == most significant now
                >>> amap[:5]  # -1 is all bits on, so endianess isnt important...
                [5192296858534827628530496329220095L, 5192296858534827628530496329220095L, 20282409603651670423947251286016L, 162259276829213363391578010288128L, 0L]

              Gotta love them Python bignums. FixLong Vmaps as well as FixChar have a resize() method, so just for giggles:

                >>> amap.resize(8000)  
                >>> len(amap) # 8192 / 8000 =
                >>> amap[0]# broken for html
                2271371013423771532966636899650014224536397371723 1670476922125503827279038503193467041246456334782 656935715744462339438782354166906879L
                >>> amap.clearflag(v.VM_LLALE) # we've dirtied the first 28 bytes...
                >>> amap[0] # if those are the most significant bytes (big endian)...
                831232460999333652239585333103 ...[clipped]... 1943132749824L
                >>> len(str(amap[0])) # thats a very large number

            Using Files

              So far Vmaps have done nothing that the Numeric arrays can't do... Almost. The memory for the annonymous mapping used in previous examples is actually freed when the Vmap is closed, as opposed to Numeric arrays' storage which is kept by the program for possible later use. On modern systems with swapping and virtual memory, this is usually not a big issue.

              To use a file to store a Vmap, give it a real file handle, attached to a file open for read and write, and large enough to contain the wntire mmap()'d area. Attempting to access data past the end of the file will cause interesting and system dependant failures.

              The first thing to do is create a file that is large enough, and has some known contents:

                >>> f = open('xxxx','w+')
                >>> f.write(chr(2) * 16384)

              Next we'll create a Vmap (named bmap) attached to that file:

                >>> bmap = v.newmap(f.fileno(), 16384, 0, v.MAP_SHARED)# 0 is offset into file
                >>> # the Vmap is the default Char type
                >>> bmap[:5]

              Our Vmap contains the file contents.

                >>> bmap[:10] = '.' * 9
                >>> bmap[:12]
                >>> bmap.raw_msync(v.MS_SYNC)

              The raw_msync() method allows us to request the data in memory be written to disk. The system may have written your data back to the file, or not, depending on circumstances; msync() makes sure it gets written out. Solaris man page states:

              Normal system activity can cause pages to be written to disk. Therefore, there are no guarantees that msync() is the only control over when pages are or are not written to disk.

              TODO: ...finish this, PAGESIZE, elpage()

            Other Features

            Using Headers

              Vmap objects can use a header at the beginning of the mmap()'d area to carry information about the shape of the data in the rest of the Vmap, and optionally the COUNT of elements in the Vmap that are "active". More on that in a moment. First, lets create a new Vmap, with a header, using the VM_HEADER flag.

              TODO: Type/size persistance

              TODO: Count

              TODO: User header

            Source code and documentation Copyright © 2002 by Mark Lamb
            Linux (r) is a registered trademark of Linus Torvalds
            Solaris (r) is a registered trademark of Sun Microsystems
            Pentium and Pentium Pro (r) are trademarks of Intel
            SPARC and SPARCcenter (r) are trademarks of SPARC International
            ... "of" is probably trademarked by someone; that and all the others are their owners'.