NUMA-aware Memory Allocation

Alex Levitski

Technical Writer

This tutorial explains how to leverage the NUMA-aware memory allocation within the GridGain Unified Real-Time Data Platform. NUMA stands for "Non-Uniform Memory Access." It is a computer memory design used in multiprocessors, where the memory access time depends on the memory location relative to the processor. Using NUMA-aware allocation, applications running on GridGain can achieve higher performance as they optimize memory access patterns. Specific performance benefits depend on your environment, application load, and system resources.

Assumptions

You are using a Linux system, like amd64 Linux (NUMA only works on Linux).
You are using a HotSpot Java implementation.
Your hardware supports NUMA. If it doesn’t, NUMA-aware memory allocation will not improve performance.

Step 1: Install the Required Packages

Make sure that your Linux system has libnuma and numactl installed. These packages are necessary for managing and configuring NUMA on your system. Use your system’s package manager to install them:

sudo apt-get update
sudo apt-get install libnuma-dev numactl

sudo yum install numactl libnuma-devel

Step 2: Configure NUMA Nodes

While libnuma comes with a default configuration that works seamlessly with GridGain, you may need to tailor the NUMA nodes to your specific requirements for optimal performance. Use numactl to modify the configuration.

To view your system’s NUMA configuration:
```
numactl --hardware
```
To set a specific NUMA policy for an application, use numactl with the relevant options. For example, to bind a process to a specific node:
```
numactl --cpunodebind=0 --membind=0 your_command
```
Adjust node numbers based on your hardware configuration and requirements.

Step 3: Enable the NUMA Allocator in GridGain

Enable the ignite-numa-allocator module that enables GridGain data regions to allocate memory using NUMA:

Navigate to your GridGain installation directory.
Move the ignite-numa-allocator module from the libs/optional directory to libs.
```
mv libs/optional/ignite-numa-allocator libs/
```

Step 4: Configure NUMA for Data Regions

To enable NUMA-aware allocation for a specific data region, adjust the memoryAllocator property in that region’s DataRegionConfiguration in the GridGain configuration file (e.g., ignite.xml):

Open your GridGain configuration file in a text editor.
Locate the DataRegionConfiguration section.
Set the memoryAllocator property to one of the NUMA allocation strategies (see strategy descriptions below).

Simple Allocation Strategy

This strategy is the best for attaching a data region to a specific NUMA node.

On All NUMA Nodes

<property name="dataStorageConfiguration">
    <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
        <property name="defaultDataRegionConfiguration">
            <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                <property name="name" value="Default_Region"/>
                ...
                <property name="memoryAllocator">
                    <bean class="org.apache.ignite.mem.NumaAllocator">
                        <constructor-arg>
                            <bean class="org.apache.ignite.mem.SimpleNumaAllocationStrategy"/>
                        </constructor-arg>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</property>

On a Specific Node

<property name="dataStorageConfiguration">
    ...
    <property name="memoryAllocator">
        <bean class="org.apache.ignite.mem.NumaAllocator">
            <constructor-arg>
                <bean class="org.apache.ignite.mem.SimpleNumaAllocationStrategy">
                    <constructor-arg name="node" value="0"/>
                </bean>
            </constructor-arg>
        </bean>
    </property>
    ...
</property>

Interleaved Allocation Strategy

This strategy is optimal for distributing a data region across multiple NUMA nodes.

On all NUMA Nodes

<property name="dataStorageConfiguration">
    ...
    <property name="memoryAllocator">
        <bean class="org.apache.ignite.mem.NumaAllocator">
            <constructor-arg>
                <bean class="org.apache.ignite.mem.InterleavedNumaAllocationStrategy"/>
            </constructor-arg>
        </bean>
    </property>
    ...
</property>

On Specified NUMA Nodes

<property name="dataStorageConfiguration">
    ...
    <property name="memoryAllocator">
        <bean class="org.apache.ignite.mem.NumaAllocator">
            <constructor-arg>
                <bean class="org.apache.ignite.mem.InterleavedNumaAllocationStrategy">
                    <constructor-arg name="nodes">
                        <array>
                            <value>0</value>
                            <value>1</value>
                        </array>
                    </constructor-arg>
                </bean>
            </constructor-arg>
        </bean>
    </property>
    ...
</property>

Local Node Allocation Strategy

This strategy is used for allocating on the local NUMA node of the process.

<property name="dataStorageConfiguration">
    ...
    <property name="memoryAllocator">
        <bean class="org.apache.ignite.mem.NumaAllocator">
            <constructor-arg>
                <bean class="org.apache.ignite.mem.LocalNumaAllocationStrategy"/>
            </constructor-arg>
        </bean>
    </property>
    ...
</property>

Step 5: Enable NUMA-aware Allocation in JVM

To ensure the JVM itself is optimized for NUMA, use the -XX:+UseNUMA JVM option by setting the JVM_OPTS environment variable:

export JVM_OPTS="-XX:+UseNUMA"

Step 6: Start a GridGain Node and Load Data

Start your GridGain node with the modified configuration:
```
bin/ignite.sh path/to/your/ignite.xml
```
Using either a thick or a thin client, put some data into the data region you’ve configured for NUMA allocation to trigger the allocation.

Step 7: Verify NUMA Allocation

To confirm that NUMA allocation is functioning correctly, inspect the numa_maps file for the GridGain node’s process:

Find your GridGain node’s process ID (PID).

Check the numa_maps file:

grep libnuma_alloc /proc/YOUR_GRIDGAIN_NODE_PID/numa_maps

A non-empty list of pages allocated with NUMA means that your data region is now utilizing NUMA-aware memory allocation, which should enhance the performance of your GridGain deployments on NUMA-enabled systems.

© 2024 GridGain Systems, Inc. All Rights Reserved. Privacy Policy | Legal Notices. GridGain® is a registered trademark of GridGain Systems, Inc.
Apache, Apache Ignite, the Apache feather and the Apache Ignite logo are either registered trademarks or trademarks of The Apache Software Foundation.

Last updated on May 07, 2024