Performance of System V Style Shared Memory Support in Python 3.8

In version 3.8, Python supports System V style shared memory. This support allows creation of memory segments that can be shared between Python processes and, consequently, help avoid (de)serialization costs when sharing data between processes.

Image for post
Image for post

Evaluation

To evaluate the performance gains from shared memory, I ran the following simple test — create a list of integers and double each integer in the list in parallel by chunking the list and processing each chunk in parallel.

Test Code

Result and Observation

The below runtime data was averaged across six test runs.

Without shared memory
data 99 ints : 0.004216 secs per iteration
data 999 ints : 0.004034 secs per iteration
data 9,999 ints : 0.004522 secs per iteration
data 99,999 ints : 0.012110 secs per iteration
With shared memory
data 99 ints : 0.005805 secs per iteration
data 999 ints : 0.015656 secs per iteration
data 9,999 ints : 0.718858 secs per iteration
data 99,999 ints : 67.926538 secs per iteration

Clearly, the performance based on shared memory is worse (4x, 179x, 6792x) that the performance based on non-shared memory.

Even with bool, float, and string data values (and corresponding minor tweaks to worker function), the test results in similar observation.

Questions

Besides the obvious question — Is this evaluation flawed?, the observation raises three questions.

  1. If so, this evaluation suggests ShareableList will not improve performance when used with vanilla lists of primitive types. However, ShareableList can only contain int, float, bool, string, bytes, and None values. So, under what circumstances will ShareableList improve performance?
  2. Also, in general, under what circumstances will the new shared memory support improve the performance?

Summary

The new shared memory feature (ShareableList in particular) in Python 3.8 is unlikely to improve performance off-the-shelf. A thorough evaluation of the circumstances under which this feature will improve performance is needed to help developers decide to use this feature.

Note

Since this feature was included in Python, I am assuming there was a backing performance evaluation that the feature would improve performance. However, the above result suggests otherwise. So, if you think the above evaluation is flawed, then please do leave a comment with the reasons why you think the evaluation is flawed. This will help me fix it!

Written by

Programming, experimenting, writing | Past: SWE, Researcher, Professor | Present: SWE

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store