Selected work
08 / 08Open source · Kubernetes GPU isolation2026

CASE STUDY · 2026

HAMi #2125: SSH Memory Limit Reproduction.

Trace why SSH launched processes lose their GPU memory limit.

Public reproduction · No GPU required

A GPU free reproduction showing how an SSH login can load HAMi’s hook but create a separate, unbounded shared memory cache after its configuration environment is rebuilt.

OVERVIEW

What it is
and who it helps.

This reproduction models the configuration handoff behind HAMi issue #2125 without requiring a GPU. The harness compares the container process tree with `su -` and a real SSH login: the normal path shares a cache with a 3000m limit, while the login path opens a different cache whose missing limit is interpreted as unlimited.

#2125Upstream issue reproduced
4 casesContainer, nested, su, SSH
No GPUConfiguration handoff only
THE CHALLENGE

The dynamic hook can still be loaded through `/etc/ld.so.preload` after an SSH login, which makes the failure look like a broken hook. The project traces the more subtle case: `sshd` reconstructs the process environment, the shared cache path disappears, and initialization can create and seed a second shared region before file based overrides are loaded.

HOW I APPROACHED IT

From the first
interaction to the edge cases.

01

Build a GPU free comparison harness

Created a minimal Docker and shell based reproduction that compares the container process tree, nested children, `su -`, and a real SSH login while holding cgroups and namespaces constant.

02

Compare the actual shared regions

Recorded the normal processes opening `/tmp/container.cache` with a 3000m limit, while `su -` and SSH opened `/tmp/cudevshr.cache` with a zero limit, which represents unlimited memory.

03

Trace initialization order

Followed the missing `CUDA_DEVICE_MEMORY_SHARED_CACHE` value through the fallback path, fresh cache creation, and first process initialization. The existing consistency check does not run because the login process believes it is the first initializer.

04

Document the evidence and its limits

Published the reproduction scripts and source references. The repository describes the cache path fallback as the likely cause, while clearly noting that this source level reading has not yet been confirmed on physical GPU hardware.

OUTCOME

A small reproduction for a hidden boundary.

The public harness narrows the failure to the difference between a hook file that survives an SSH login and environment based configuration that does not. Its source level root cause analysis is presented as a likely explanation, not a confirmed on hardware conclusion.

BUILT WITH

Linux processesOpenSSHCDynamic linkerShared memoryDocker

UP NEXT · 01

Sorted.

Read case study