Two unrelated globals silently landed on the same cache line — and the profiler pointed at the wrong instruction.

1. Background: DORADD in one minute

DORADD (PPoPP'25) is a deterministic parallel runtime we designed for fast log replay, state machine replication, etc. It executes a log of requests in a streaming manner (DORADD is the very first one to achieve low latency at microsecond scale, while all prior work fundamentally relies on batching). To also maintain high throughput, DORADD proposes a novel system architecture, leveraging a dispatcher pipeline with each core dedicated to a subtask of dispatching (prefetching, object lookup, scheduling), and a worker pool for request execution.

rpc_handler → prefetcher → indexer → spawner → 8 worker cores
   (core 0)     (core 1)    (core 2)  (core 3)    (cores 4-11)

The spawner is the interesting one, which is in charge of resolving task dependencies due to the determinism requirement. For every transaction it creates a behaviour from verona-rt — which waits until all objects the transaction touches are free, then runs on a worker. Spawner is the bottleneck of the dispatcher pipeline: on our Ice Lake Xeon (Gold 5318N), it can at maximum achieve 5.1 Mrps on uniform YCSB benchmark (10 objects per transaction).

Two globals appear later in this story. All you need to know now: they belong to different subsystems and are used by completely different threads.

• mem_array_idx (behaviourcore.h): the cursor of the static slab the spawner allocates behaviours from. Written by the spawner several times per transaction. No other thread touches it.
• global_epoch (epoch.h): verona-rt's memory-reclamation epoch, one 8-byte atomic in .bss. Read by every worker constantly — on every dequeue from the scheduler's work queue.

2. The problem, and the TL;DR

We upgraded the testbed to Ubuntu 24.04, where /usr/bin/c++ is g++-13, and re-ran the paper's YCSB experiment:

Same flags (-O3 -march=native), same machine, same input, ~30% gone. No warning, no other symptom. It looks exactly like a compiler code-generation regression. The takeaways up front:

1. The regression was not codegen. The machine code of the hot function, Spawner::run(), is byte-for-byte identical in a fast build and a slow build (728 instructions; only relocation addresses differ).
2. The root cause is false sharing between mem_array_idx (spawner-written, per transaction) and global_epoch (worker-read, continuously). A libstdc++ change in GCC 13 — removing the per-translation-unit std::__ioinit iostream-init object — shifted .bss by 32 bytes and dropped both variables onto the same 64-byte cache line.
3. Per-instruction cycle attribution shows where the pipeline waits, not why. Precise sampling piled 28% of spawner cycles on one innocent 16-byte load.
4. Addresses predict the performance. Read the two variables' addresses with nm (the binutils tool that lists each symbol and its address in a binary) and check whether they fall on the same 64-byte line. This check classified all 15 binaries we ever built (four compiler majors, many flag experiments) as fast or slow, with no exception. perf c2c then confirmed the contention directly.
5. The fix is a single alignas(64). With it, g++-13 hits 5.05 Mrps — the compiler choice stops mattering entirely.

3. Diagnosis

Basic triage first. The bottleneck stayed the spawner in both builds, so the gap is spawner cost: ~660 vs ~960 cycles per transaction. The usual suspects came back clean: frequency, hyperthreading, the request generator, libatomic call counts, inlining and code size, dTLB misses, AVX-512 downclocking — all measured, all equal.

Precise profiling (PEBS) then produced one spectacular red herring. A single 16-byte load in the behaviour-construction path carried 4% of spawner cycles under g++-11 vs 28% under g++-13 — on its face, most of the missing ~300 cycles on one instruction. Controlled experiments cleared it: the load is equally cheap in both builds (perf mem with unbiased sampling), and removing those vector copies entirely made g++-13 slower. The samples piled on that instruction because the pipeline stalled there, not because the instruction was slow — attribution, not causation. We return to this lesson in §5.

With the instruction stream cleared, what else differs between a g++-11 and a g++-13 build? The standard library headers. That suggested the decisive control experiment: hold the compiler fixed, swap only the headers.

g++-12  -isystem /usr/include/c++/12 ...   →  4.94 Mrps   (call it hdr12)
g++-12  -isystem /usr/include/c++/13 ...   →  3.50 Mrps   (call it hdr13)

Same compiler, full regression. Now diff the two binaries' hot function — and find no diff: Spawner::run() is 728 identical instructions in both; every diff line is a relocation address. The code is the same. The addresses are not. In nm output (addresses are file offsets; u marks C++17 inline variables), everything in the slow build's .bss sits 32 bytes lower:

fast (hdr12):                          slow (hdr13):
1f350 b std::__ioinit                  (gone — removed in libstdc++ 13)
...                                    ...
1f464 u BehaviourCore::mem_array_idx   1f444 u BehaviourCore::mem_array_idx
1f468 u BehaviourCore::mem_array       1f448 u BehaviourCore::mem_array
...                                    ...
1f490 u GlobalEpoch::global_epoch      1f470 u GlobalEpoch::global_epoch

The 32-byte shift traces to something wonderfully mundane: GCC 13's libstdc++ stopped emitting the per-translation-unit std::__ioinit object (iostream initialization moved to init_priority, a link-time init-ordering attribute), which deleted 8 bytes plus alignment ripple from every TU that includes <iostream>.

A 32-byte shift matters because of where the 64-byte line boundaries fall:

fast (hdr12)                          slow (hdr13)
┌ line 0x1f440 ──────────────┐        ┌ line 0x1f440 ──────────────┐
│ +0x24  mem_array_idx       │        │ +0x04  mem_array_idx       │
└────────────────────────────┘        │ +0x30  global_epoch        │
┌ line 0x1f480 ──────────────┐        └────────────────────────────┘
│ +0x10  global_epoch        │                 same line!
└────────────────────────────┘

No thread ever shares these two variables by name. Now they share a line, and the cache coherence protocol works on whole lines:

     spawner core                          8 worker cores
  stores mem_array_idx                   load global_epoch
  several times per txn                  on every dequeue
          │                                     │
          ▼                                     ▼
     ┌────────────────────────────────────────────────┐
     │ cache line 0x1f440 (slow build)                │
     │   +0x04 mem_array_idx      +0x30 global_epoch  │
     └────────────────────────────────────────────────┘

  Each spawner store invalidates the line in all worker caches.
  Each worker load pulls the line back from the spawner cache.
  The line ping-pongs on every transaction — and the spawner, the
  bottleneck core, pays the missing ~300 cycles per transaction.

Two pieces of confirmation nailed it:

The rule holds on all 15 binaries, with no exception. During this investigation we had built 15 binaries: four compiler majors, many flag experiments, the header shims above. For each one, take the two addresses from nm and divide by 64 to get each variable's cache line. One rule classifies every binary correctly: same line → slow (~3.5 Mrps), different lines → fast (~5 Mrps). The addresses predict throughput before the binary ever runs. The rule even explained a false lead: two builds with changed instruction-scheduler flags had looked ~5% faster, but the flags had only nudged .bss addresses — layout again, not code.

perf c2c catches it in the act. It is a perf mode built for exactly this problem (c2c means cache-to-cache). It samples loads and stores in hardware, groups the samples by 64-byte cache line, and ranks lines by HITM count. A HITM ("hit modified") is a load that finds its line in Modified state in another core's cache: some other core wrote the line moments ago, and the coherence protocol must ship the line across cores to serve the load. A few HITMs are normal; one line collecting them at a high rate means two cores fight over that line. When the writes and the reads also land on different offsets within the line, it is false sharing by definition.

On the slow build, our line is the #1 user-space HITM line in the whole system, holding 10.3% of all HITMs:

Cacheline 0x....1440   (runtime address = PIE load base + file offset 0x1f440)
  offset +0x04  stores   Spawner<YCSBTransaction>::run()      ← mem_array_idx
  offset +0x30  loads    verona::rt::MPMCQ<...>  avg 238 cyc  ← global_epoch

How to read this: for each hot line, the report breaks accesses down by byte offset inside the line, and names the function behind each access. Offset +0x04 receives stores from Spawner::run() — that is mem_array_idx (file offset 0x1f444 = line base + 4). Offset +0x30 receives loads from the workers' queue code — that is global_epoch (0x1f470 = base + 0x30). Those loads average 238 cycles, the typical cost of pulling a line out of another core's cache, several times a local cache hit. One screen of output names both variables, both code paths, the write/read pattern, and the cross-core latency — the complete false-sharing case.

4. The fix, and results

The fix isolates the spawner-hot cursor on its own cache line (behaviourcore.h):

struct alignas(64) MemArrayState
{
  char*    arr;
  uint32_t idx;
  char     pad[52];
};
static inline MemArrayState mem_state{};
static inline char*&    mem_array     = mem_state.arr;
static inline uint32_t& mem_array_idx = mem_state.idx;

We first tried the other side: wrapping global_epoch in the same kind of alignas(64) padded struct. That fix fails (3.74 Mrps), for a subtle reason. global_epoch is a function-local static, so the compiler emits a hidden second symbol for it: a guard variable — one byte that records "already initialized", checked on every call to global_epoch() for thread-safe initialization. Workers therefore read the guard as often as the epoch itself. The wrapper moves the epoch to its own line, but the guard is a separate .bss symbol and stayed on the hot line next to mem_array_idx — so the false sharing continued.

mem_array_idx has no hidden companions. Once the cursor sits alone on its own line, no worker-read data can share a line with it, wherever the epoch and its guard land. Hence the rule: isolate the variable you own — the side where you control every symbol.

With the patch, g++-11 through g++-14 are equivalent. There is no compiler regression to report — and there never was one.

5. Conclusion

• Data layout is an invisible input to your benchmark. Two unrelated globals, from different subsystems, ended up as neighbors through pure link order — and an 8-byte change in a header moved a performance cliff. Any contended or hot-written global deserves explicit alignas(64) isolation; relying on layout luck means any rebuild can flip you over the edge. (Watch hidden neighbors: static-local guard variables travel with their variable.)
• Treat profiler attribution as a hypothesis, not a verdict. PEBS handed us a hot instruction, and a plausible microarchitectural story around it cost a day before controlled experiments killed it. The differential experiment (same compiler, swapped headers, identical hot code) is what actually localized the cause.
• perf c2c is great. Once false sharing is on the suspect list, it doesn't just detect it — it names the line, the offsets, both code paths, and the latency.

Setup for reproduction: Xeon Gold 5318N (Ice Lake), 1 NUMA node, Ubuntu 24.04, YCSB uniform no-contention, taskset -c 4-11 ./ycsb -n 8 <log> -i exp:180 (taskset confines the 8 workers to cores 4–11; the four pipeline stages pin themselves to cores 0–3 internally; exp:180 = exponential request inter-arrivals with a 180 ns mean, the saturating load), -O3 -march=native, performance governor, THP enabled. Burst profiling: perf record -C 1 (core 1 = spawner) -e cycles:pp -F 50000, then bin samples by time and keep bins not dominated by the spawner's spin-loop offsets.