Matching memory access patterns and data placement for NUMA systems
Authors:
Open Access
- 31 March 2012
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 230-241
Abstract
Many recent multicore multiprocessors are based on a nonuniform memory architecture (NUMA). A mismatch between the data access patterns of programs and the mapping of data to memory incurs a high overhead, as remote accesses have higher latency and lower throughput than local accesses. This paper reports on a limit study that shows that many scientific loop-parallel programs include multiple, mutually incompatible data access patterns, therefore these programs encounter a high fraction of costly remote memory accesses. Matching the data distribution of a program to the individual data access patterns is possible, however it is difficult to find a data distribution that matches all access patterns. Directives as included in, e.g., OpenMP provide a way to distribute the computation, but the induced data partitioning does not take into account the placement of data into the processors' memory. To alleviate this problem we describe a small set of language-level primitives for memory allocation and loop scheduling. Using the primitives together with simple program-level transformations eliminates mutually incompatible access patterns from OpenMP-style parallel programs. This result represents an improvement of up to 3.3X over the default setup, and the programs obtain a speedup of up to 33.6X over single-core execution (19X on average) on a 4-processor 32-core machine.Keywords
This publication has 14 references indexed in Scilit:
- Feedback-directed page placement for ccNUMA via hardware-generated memory tracesJournal of Parallel and Distributed Computing, 2010
- Memphis: Finding and fixing NUMA-related performance problems on multi-core platforms2010
- What can performance counters do for memory subsystem analysis?2008
- Thread clustering2007
- Programming for parallelism and locality with hierarchically tiled arrays2006
- Using Hardware Counters to Automatically Improve Memory Performance2005
- Generalized multipartitioning of multi-dimensional arrays for parallelizing line-sweep computationsJournal of Parallel and Distributed Computing, 2003
- Data distribution support on distributed shared memory multiprocessors1997
- Operating system support for improving data locality on CC-NUMA compute servers1996
- Locality and Loop Scheduling on NUMA Multiprocessors1993