CorrectNav Inference

loading

Runtime

Rollout contract

reset
prompt
→
capture initial RGB
→
step
1 image
→
predict 6
execute ≤4
execute action 1
→
capture RGB 1
→
repeat per action
→
next step
send images only

Recommended: execute all four returned actions before replanning. The server also accepts early replanning after 1–4 actions. Capture one observation immediately after every non-Stop primitive action and send only those images in the next step. The image count identifies the executed prefix. Do not capture after Stop.

Minimal Python client

import asyncio, base64, io, json
import websockets
from PIL import Image

def jpeg_b64(rgb):
    buf = io.BytesIO()
    Image.fromarray(rgb).save(buf, "JPEG", quality=90)
    return base64.b64encode(buf.getvalue()).decode()

async def rollout(env, instruction):
    async with websockets.connect("ws://127.0.0.1:5901", max_size=64<<20) as ws:
        # 1. Reset server-side episode history.
        await ws.send(json.dumps({"type": "reset", "prompt": instruction}))
        reset = json.loads(await ws.recv())
        inference_id = reset["inference_id"]

        # 2. Capture the initial observation before executing any action.
        pending_images = [jpeg_b64(env.current_rgb())]
        while True:
            # 3. Send observations collected since the previous inference.
            await ws.send(json.dumps({
                "type": "step",
                "inference_id": inference_id,
                "images": pending_images,
            }))
            result = json.loads(await ws.recv())
            if result["status"] not in ("ok", "invalid_output"):
                raise RuntimeError(result)
            inference_id = result["inference_id"]

            # 4. Execute exactly four actions in normal operation. A rollout
            #    may be shorter only when the model emits Stop.
            actions = result["actions"]
            stop = [0.0, 0.0, 0.0]
            if len(actions) != 4 and stop not in actions:
                raise RuntimeError(f"expected 4 actions, got {len(actions)}")
            pending_images = []
            for action in actions[:4]:
                if action == stop:
                    env.stop()
                    return
                dx, dy, dyaw = action
                env.execute(dx=dx, dy=dy, dyaw=dyaw)  # meters, meters, radians
                pending_images.append(jpeg_b64(env.current_rgb()))  # capture immediately

            # 5. The next loop sends 1-4 images. The server infers the executed
            #    actions from the same-length prefix of its previous response.

asyncio.run(rollout(env, "Walk past the sofa and enter the kitchen."))

Sessions

Select a session