STVLA: Spatial-aware and Temporal-memory for Vision-Language-Action Models

Abstract

Robotic manipulation often requires both spatial scene understanding and task history. However, many vision-language-action (VLA) policies rely primarily on current observations, without explicitly integrating spatial structure and task history. Appearance-based memories can be sensitive to occlusion and viewpoint changes, while spatial representations of the current scene alone do not preserve how the scene has changed during task execution. We introduce STVLA, a framework that incorporates features from a pretrained visual geometry foundation model into temporal memory alongside visual representations. This allows the policy to recall past scene structure as context for current action decisions. A joint retrieval mechanism favors historical information that is relevant to and complements the current observation. Across evaluations on LIBERO, LIBERO-Mem, MIKASA-Robo, and SimplerEnv Bridge, STVLA achieves the highest average success rate among the compared methods on each benchmark. On real-world block stacking, object organization, and phone packaging, it improves success over π_0.5 by 58.33, 30.00, and 35.00 percentage points, respectively. These results support integrating scene geometry into temporal memory to improve manipulation requiring both spatial precision and task history.

STVLA overview: spatial representations distilled from a frozen visual geometry model, a bounded visual-spatial memory, four simulated benchmarks, and three real-world tasks — **Overview.** STVLA combines compact spatial representations distilled from a frozen visual geometry model with a bounded visual–spatial memory, so the policy can retrieve and fuse task-relevant historical evidence during action generation. It attains the highest average success among the compared methods on each simulated benchmark, and improves real-world success over π_0.5 by 58.33, 30.00, and 35.00 percentage points on block stacking, object organization, and phone packaging.

AGIBOT G2

Real-World Demonstrations

Three tasks on the AGIBOT G2, each observed by one fixed head-view camera and a wrist camera on each arm. Every clip is one successful execution.

Block Stacking

100.0% success

Place a blue block stably on a red block at varying target heights.

Object Organization

85.0% success

Press the button with the right arm, store the block, place the box in the drawer, and press the button again with the left arm.

Phone Packaging

45.0% success

Pack the phone case, manual, cable, and charger, including a bimanual charger handover and final placement of the case.

Real-World Results

STVLA reaches 76.7% average success and is the best result on every task. The gains over π_0.5 are 58.3, 30.0, and 35.0 percentage points. On block stacking, all 60 / 60 evaluated trials succeed, 11.7 percentage points above MemoryVLA and 41.7 percentage points above Spatial Forcing.

Success rate (%). Best per column in bold.
Method	Block stacking	Object organization	Phone packaging	Average
π_0.5	41.7	55.0	10.0	35.6
Spatial Forcing	58.3	40.0	10.0	36.1
MemoryVLA	88.3	35.0	15.0	46.1
STVLA (ours)	100.0	85.0	45.0	76.7

Block stacking: 20 trials at each of three target heights. Object organization and phone packaging: 20 trials per method. A block-stacking trial succeeds only if the blue block remains stable after release.

Four benchmarks

Simulation Experiments

On each benchmark, STVLA attains the highest average success among the compared methods. One benchmark is shown at a time, and each clip is one successful rollout.

LIBERO

STVLA reaches 97.8% average success across the five suites, compared with 96.6% for π_0.5 and 96.5% for MemoryVLA. Success is higher than π_0.5 on four suites. The largest differences are on Goal (97.6% vs. 94.6%) and LIBERO-10 (97.6% vs. 95.4%).

Success rate (%). Best per column in bold.
Method	Spatial	Object	Goal	LIBERO-10	LIBERO-90	Average
Diffusion Policy	78.3	92.5	68.3	50.5	–	72.4
Octo	78.9	85.7	84.6	51.1	–	75.1
UniAct	77.0	87.0	77.0	70.0	73.0	76.8
SpatialVLA	88.2	89.9	78.6	55.5	46.2	71.7
TraceVLA	84.6	85.2	75.1	54.1	–	74.8
OpenVLA	84.7	88.4	79.2	53.7	73.5	75.9
CoT-VLA	87.5	91.6	87.6	69.0	–	81.1
π₀-FAST^*	96.4	96.8	88.6	60.2	83.1	85.0
CronusVLA	90.1	94.7	91.3	68.7	–	86.2
CogACT	97.2	98.0	90.2	88.8	92.1	93.2
π₀^*	96.8	98.8	95.8	85.2	–	94.2
MemoryVLA	98.4	98.4	96.4	93.4	95.6	96.5
π_0.5^*†	97.4	99.2	94.6	95.4	96.4	96.6
STVLA (ours)^*†	98.8	98.2	97.6	97.6	96.8	97.8

^* Additional proprioception and wrist-camera inputs. ^† Trained and evaluated by us, with 50 trials per task. Other rows are quoted from MemoryVLA with the original evaluation protocols. Published averages cover five suites when LIBERO-90 is available and four otherwise; they are not directly comparable across all rows. The 1.2-point gain is relative to our π_0.5 baseline over the same five suites.

LIBERO-Mem

History-dependent manipulation across ten tasks. STVLA reaches 41.0% average success, compared with 38.0% for π_0.5. The largest gains are on the repeated-placement tasks B × 5 and B × 7, each 30 percentage points above π_0.5.

Bt × 3 had no successful trial, so that task is omitted from the gallery.

Success rate (%). Best per column in bold.
Method	B → P	Bt → P	B × 3	B × 5	B × 7	Swap2	Rot3	Cs → Bskt	Cs + E	Average
π₀	50.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	5.0
SlotVLA (h = 8)	50.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	5.0
MemoryVLA	55.0	0.0	35.0	5.0	0.0	0.0	0.0	0.0	0.0	9.5
π_0.5^†	95.0	5.0	80.0	40.0	10.0	10.0	5.0	55.0	80.0	38.0
STVLA (ours)^†	95.0	0.0	90.0	70.0	40.0	5.0	5.0	25.0	80.0	41.0

B: bowl; Bt: bottle; P: plate; Cs: cream cheese; Bskt: basket; E: empty basket. × n denotes n repetitions. Swap2 and Rot3 denote swapping two bowls and rotating three bowls. ^† Our models, evaluated with 20 trials per task; other results are reported baselines.

MIKASA-Robo

On five memory-intensive tasks, STVLA reaches 45.2% average success, 12.8 percentage points above π_0.5 and 4.0 percentage points above MemoryVLA. Success is higher than π_0.5 on all five tasks.

Success rate (%). Best per column in bold.
Method	ShellGame Touch	Intercept Medium	Remember Color3	Remember Color5	Remember Color9	Average
SpatialVLA	23.0	27.0	27.0	17.0	11.0	21.0
OpenVLA-OFT	47.0	14.0	59.0	16.0	6.0	28.4
π₀	33.0	42.0	35.0	22.0	15.0	29.4
π_0.5^†	78.0	22.0	28.0	22.0	12.0	32.4
MemoryVLA	88.0	24.0	44.0	30.0	20.0	41.2
STVLA (ours)^†	86.0	38.0	46.0	32.0	24.0	45.2

^† Trained and evaluated by us, with 100 trials per task. Other results are quoted from MemoryVLA.

SimplerEnv Bridge

On four WidowX tasks, STVLA reaches 74.0% average success, compared with 71.9% for MemoryVLA and 66.7% for π_0.5. On Stack Cube, success is 54.2% versus 37.5% for MemoryVLA, a gain of 16.7 percentage points.

Success rate (%). Best per column in bold.
Method	Spoon on towel	Carrot on plate	Stack cube	Eggplant in basket	Average
OpenVLA	4.2	0.0	0.0	12.5	4.2
SpatialVLA	16.7	25.0	29.2	100.0	42.7
π_0.5^†	83.3	37.5	45.8	100.0	66.7
MemoryVLA	75.0	75.0	37.5	100.0	71.9
STVLA (ours)^†	79.2	62.5	54.2	100.0	74.0

^† Trained and evaluated by us, with 24 trials per task. Other results are taken from the works cited in the manuscript.

The architecture

Method Overview

STVLA keeps scene geometry and appearance in one bounded memory,
then retrieves history that is relevant to and complements the current observation.

STVLA architecture: memory retrieval and gate fusion on top, the vision-language action backbone in the middle, and spatial tokenization at the bottom — **Architecture.** Current visual and spatial tokens query a bounded memory bank through a relevance and non-redundancy score and score-biased cross-attention; a gate then fuses the retrieved features with the current representations (top). The fused tokens condition the vision–language backbone and the flow-matching action expert (middle). Spatial tokenization compresses current-frame features from frozen VGGT-Ω through pooling cross-attention and a variational bottleneck; the reconstruction branch is used only during training (bottom). Flame and snowflake icons mark trainable and frozen modules.

Spatial tokenization. Frozen VGGT-Ω uses preceding frames as context and passes only the current-frame features to a trainable compressor. Learned-query pooling and a variational bottleneck produce 256 spatial tokens. At inference, the posterior mean is used in place of sampling.

Visual–spatial memory. A bank of at most 16 entries stores paired visual and spatial tokens from the same episode. A parameter-free score combines feature similarity with normalized feature difference, and biases two cross-attention layers toward history that is relevant to and complements the current observation. With no history, the current tokens pass through unchanged. The default bias scale is β = 0.5.

Gated action conditioning. A learned element-wise gate fuses the retrieved features with the current tokens. Visual and spatial groups then enter separate prefix segments, both accessible to the action expert. After retrieval, the current tokens are appended to the bank, and the bank resets at each episode boundary.

Block stacking

Ablation Studies

Real-world block stacking, averaged over three target heights with 20 trials each.
Within each study, all other settings are held fixed. Defaults are β = 0.5 and K = 4.

Spatial Information and Memory

Removing memory lowers success from 100.0% to 95.0%, and removing spatial information lowers it to 96.7%. The two variants fail differently: without memory, execution hesitates more; without spatial information, the failures are empty grasps and placement errors.

Success rate (%). Best per column in bold.
Variant	Success (%)
π_0.5	41.7
Without memory	95.0
Without spatial	96.7
STVLA (ours)	100.0

Without memory, spatial tokens are retained and the memory bank is removed. Without spatial information, memory is retained over visual tokens only.

Retrieval Bias Strength

Success varies non-monotonically with the bias scale. The highest rate is 100.0% at β = 0.5, compared with 80.0% when the score-induced bias is disabled. A stronger bias does not by itself raise success.

Success rate (%) across retrieval-bias settings.
Bias scale β	0	0.05	0.3	0.5	0.8	1.0
Success (%)	80.0	71.7	71.7	100.0	71.7	81.7

Temperature is fixed at 1. Setting the bias to zero retains memory retrieval but disables the score-induced bias.

Temporal Context for Geometry

The response to the number of VGGT-Ω input frames is non-monotonic. The highest success rate is 100% at K = 4. Additional frames give the geometry model more context for the current scene, but they also increase inference latency, so more frames do not necessarily improve closed-loop performance.

Success rate (%) across input-frame settings.
Input frames K	1	2	3	4	5
Success (%)	88.3	93.3	85.0	100.0	86.7

The current frame is always included. Sampling scheme and memory capacity are held fixed.

BibTeX

@misc{stvla2027,
  title={STVLA: Spatial-aware and Temporal-memory
         for Vision-Language-Action Models},
  author={Anonymous},
  year={2027},
  note={Manuscript under review}
}