ActionCodec2

A Streaming Action Tokenizer

Tokenize robot actions one control step at a time, with no neural network and a guaranteed error bound at every step. One pretrained instance serves 24 embodiments.

quickstart.py
import numpy as np
from transformers import AutoProcessor

codec = AutoProcessor.from_pretrained(
    "anonymous/ActionCodec2-1st-order",
    trust_remote_code=True,
)

# Synthetic episode for a copy-paste check. Replace with your own (T, 7) data.
actions = np.zeros((12, 7), dtype=np.float32)
actions[:, 6] = 1.0  # open gripper

tokens = codec.encode(actions, fps=15)
decoded_actions = codec.decode(tokens, fps=15)

Part I

Method

  1. 01Why streamingEmit tokens every control step, not every chunk.
  2. 02CoveringCover each step's reachable motion with a fixed grid, so every step stays within ε.
  3. 03Set-BPEMerge tokens that occur together, losslessly, so the stream stays short.

01Why streaming

Chunkwise locks; streaming reacts

Almost every action tokenizer encodes a whole chunk of motion, one to two seconds, as a unit: nothing runs until the chunk is complete, and then it runs open-loop. A streaming tokenizer gives each control step its own tokens, so the robot acts as each token lands.

CHUNK VS STREAMING

What a chunk tokenizer gives up

chunk tokenizerstreaming tokenizer

02Covering

Tokenization as a covering problem

Each step's tokens must be accurate on their own, even on motion never seen in training. The rule: the reconstruction stays within ε of the true action at every step.

1

A fixed grid, not a learned codebook, so unseen actions stay accurate.

WHERE THE ERROR GOES

A learned codebook vs a fixed grid

training datatokens per action: fewer → morean action never seen in training

2

One step, one axis. Only the range reachable in one step needs cells.

TRANSLATION

One step of streaming, along one axis

true actiontokenized actionallowed error ±ε

3

The same step in rotation, with cells covering the ball of reachable turns.

ORIENTATION · SO(3)

The same step, in rotation space

true orientationtokenized orientationallowed error ±ε

4

How few tokens that takes: for a real arm, 800 bins per axis become 17.

IN NUMBERS · 3-D TRANSLATION

How many bins does a robot arm need?

per axis · arm range 4 m · 5 mm resolution · 15 actions per second · top speed 1.2 m/s

03Set-BPE

Set-BPE: order-free lossless packing

Set-BPE makes common motions cheap by merging tokens as sets: across axes within a step, then across nearby steps. It is lossless, so the ε bound is untouched.

REAL DATA · PRETRAINED ACTIONCODEC2

One second of a real robot arm, into tokens and back

real motionrebuilt from tokensexecutedone moveone step, several axesseveral steps

8.0–11.9%fewer tokens than order-dependent BPE, on held-out data

Part II

Experiments

  1. 04BenchmarksOne VLA, only the tokenizer swapped: three simulation benchmarks and a real robot.
  2. 05Reinforcement learningPPO on a harder LIBERO-Object, from about 5% success.
  3. 06AgentsA VLM agent drives a robot in tokens and builds its own macros.

04Benchmarks

State of the art among action tokenizers

One VLA (G0.5), one training recipe; only the action tokenizer changes.

1

Four benchmarks: ActionCodec2 variants take the top places on all four. Axes are zoomed to where the methods differ.

baselines ACodec2 ACodec2+ ACodec2II ACodec2+II solid = pretrained (+) · hatched = fit to the benchmark corpus · II = second order
2

On the real robot: zero-shot DROID-Franka rollouts, side by side.

Snack storing and serving. Put the yellow candies and the blue candy on the pink plate, and the cola on the green plate.
Nested-cup arrangement. Put the blue cup into the pink cup, the two into the yellow cup, then the whole stack onto the green plate.

05Reinforcement learning

Reinforcement learning pays off further

Each policy is then fine-tuned with PPO on a harder LIBERO-Object, where objects and basket are re-placed every episode. From about 5% success, the ActionCodec2 variants climb fastest to 83–93%; the others level off at 71–77%.

Tokenizer
PPO step

Rolling mean over PPO steps, band ± one rolling std. Click the plot to jump to step 0, 100 or 200.

25 rollouts from one checkpoint, objects re-placed each time; a green check marks a success. At step 0 every policy still drifts left, a habit from standard LIBERO.

06Agents

ActionCodec2 as an agent's interface

Each token names a physical motion (x−1o: 1 cm left, gripper open), so a VLM agent can write tokens directly and save sequences it trusts as macros.

THE LOOP

Look up, act, reflect, grow the library

1

Tokens beat poses; macros beat both. As the macro library grows, decisions per submission fall from 17 to 8.3.

Three control interfaces for one VLM agent on RMBench. Language commands take about 45 decisions and 1,000 thinking tokens per submission; ActionCodec2 tokens with a retained macro library fall from 17 to 8.3 decisions and from 1,231 to 235 thinking tokens over four rounds while the library grows to 13 macros. On a new task, reusing the 13 macros takes 16 decisions instead of 63 and 2,853 thinking tokens instead of 4,534.
(a) One decision under each interface (illustrative). (b) Four chained rounds of RMBench blocks-try per interface (GPT-6 Astra, low effort). (c) A new block-rearrange task, from scratch vs with the 13 macros (Claude Opus 5.5, low effort).
2

What the agent learned: 13 macros, each named and described in its own words.

THE MACRO LIBRARY

Pick any of the 13 macros

head camerawrist camera

duration
times used
tokens
0 s
3

The macros carry over. On a new task: 63 decisions and a failure from scratch, 16 and a success with the macros.

REUSE ON A NEW TASK

The same new task, run twice

Move the cube between the mats onto the empty mat, press the button once, then move the other cube into the gap.

head cameraleft wristright wrist

One segment per decision; gold segments call a macro.

Thinking

Output

right arm
left arm
4

A long-horizon case study: 20 episodes of Multi-Drawer Search on a bimanual robot. Pick a milestone to replay it.

MULTI-DRAWER SEARCH · 20 EPISODES

Nine first-time milestones, replayed

Search two cabinets for a measuring cup and tongs; place each in the red bin, cup first.

Thinking

Macros

07Try it

One universal instance

  1. 01

    20khours

    Fitted on 24 embodiments

    Fitted once on 105M end-effector and 204M joint frames.

  2. 02

    4,096tokens

    Universal

    Out of distribution it may lose compression, never accuracy.

  3. 03

    0trained weights

    NN-free

    Closed-form covering primitives and exact greedy Set-BPE. No network, no training.

  4. 04

    8.8tokens/s

    Compact

    For 15 Hz end-effector motion (10.8 for joints). Set-BPE packs them losslessly, with 11.9% fewer tokens than order-dependent BPE.

  5. 05

    ≤5mm

    Bounded tracking error

    And 0.02 rad, at every step of a rollout of any length.

  6. 06

    1step at a time

    Streaming

    Tokens come out one control step at a time, so the policy reacts to every new observation instead of waiting for a chunk to finish.

  7. 07

    x−1o

    Physical meaning

    Every token names a motion: x−1o moves 1 cm left with the gripper open, so an agent can write tokens directly.

  8. 08

    drop-in

    Plug and play

    Use it as the action vocabulary of any autoregressive VLA. The decoder is stateless and has no weights to load.

  9. 09

    Try it

    On Hugging Face

    Hugging Face · coming soon