POMDPPlanners.planners.vectorized_planners.vopp package

Vectorized Online POMDP Planner (VOPP / PORPP).

This package hosts the VOPP planner and its episode runner, whose whole search is expressed as batched tensor operations over the flat-tensor belief tree in POMDPPlanners.core.tree.vectorized_belief_tree.

Provides:

class POMDPPlanners.planners.vectorized_planners.vopp.VOPPEpisodeRunner(planner, model, *, num_belief_particles=1000, max_steps=50, world_transition=None, world_observation=None)[source]

Bases: object

Runs closed-loop POMDP episodes with a VOPPPlanner.

The runner owns the interaction loop but no planning policy of its own: the planner decides actions, the vectorized model supplies the dynamics and the belief filter, and an optional pair of world hooks overrides the ground-truth transition / observation when a real simulator is the world.

Parameters:
num_belief_particles

Size of the particle belief carried between steps.

max_steps

Maximum number of actions per episode.

Example

>>> import torch
>>> from POMDPPlanners.environments.light_dark_pomdp.continuous_light_dark_pomdp import (
...     ContinuousLightDarkPOMDP,
... )
>>> from POMDPPlanners.environments.light_dark_pomdp.continuous_light_dark_vectorized_model import (
...     ContinuousLightDarkVectorizedModel,
... )
>>> from POMDPPlanners.planners.vectorized_planners import (
...     VOPPEpisodeRunner,
...     VOPPPlanner,
... )
>>> _ = torch.manual_seed(0)
>>> env = ContinuousLightDarkPOMDP(discount_factor=0.95, is_obstacle_hit_terminal=False)
>>> model = ContinuousLightDarkVectorizedModel(env, device=torch.device("cpu"))
>>> planner = VOPPPlanner(
...     model, num_actions=model.num_actions, num_particles=128,
...     max_depth=6, num_planning_iterations=8,
... )
>>> runner = VOPPEpisodeRunner(planner, model, num_belief_particles=256, max_steps=20)
>>> initial = torch.tensor([[0.0, 5.0]])
>>> result = runner.run_episode(initial)
>>> result.num_steps >= 1
True
run_episode(initial_state, initial_particles=None)[source]

Run one closed-loop episode and return its recorded trajectory.

Parameters:

  • initial_state (Tensor) – [1, ds] (or [ds]) ground-truth start state.

  • initial_particles (Optional[Tensor]) – Optional [num_particles, ds] initial belief; defaults to the start state replicated across the particle set.

Return type:

VOPPEpisodeResult

Returns:

A VOPPEpisodeResult holding the states, beliefs, actions, rewards, planning times, and root visit counts of the episode.

Raises:

ValueError – If initial_state does not describe a single state.

class POMDPPlanners.planners.vectorized_planners.vopp.VOPPPlanner(model, num_actions, *, temperature=2.0, num_particles=1000, max_depth=10, discount_factor=0.95, num_planning_iterations=100, value_heuristic=None, belief_capacity=1024, action_capacity=1024, value_dtype=torch.float32)[source]

Bases: object

Fully vectorized online POMDP planner (VOPP / PORPP).

The planner owns a VectorizedBeliefTree and repeatedly expands and backs it up entirely with batched tensor operations. A single call to plan() samples particles from the supplied root belief, runs the configured number of forward-search / preference-backup iterations, and returns the greedy root action.

Parameters:
device

Device every tensor lives on (taken from the model).

num_actions

Size of the fixed representative action set.

Example

See the module-level docstring for a runnable example.

plan(root_particles)[source]

Plan from a particle-set root belief and return the greedy action.

The tree is cleared, then num_planning_iterations forward-search / preference-backup passes refine the root action preferences. The action with the highest root preference is returned.

Parameters:

root_particles (Tensor) – [num_root_particles, ds] states representing the current belief; iterations resample num_particles of them with replacement.

Return type:

int

Returns:

The index of the greedy root action in [0, num_actions).

Raises:

ValueError – If root_particles is not a 2-D tensor on the planner’s device.

property tree: VectorizedBeliefTree

The belief tree built by the most recent plan() call.

tree_metrics()[source]

Root-tree analysis metrics for the most recent plan() call.

Returns the same TreeMetrics set the MCTS planners report (root action visit min / max / entropy, number of root actions, root visit count, max depth, leaf flag), making VOPP directly comparable to them.

Return type:

List[PolicyInfoVariable]

Returns:

A list of PolicyInfoVariable describing the current tree.

Submodules

POMDPPlanners.planners.vectorized_planners.vopp.vopp module

Vectorized Online POMDP Planner (VOPP / PORPP).

This module implements VOPPPlanner, a fully GPU-vectorized online POMDP solver following Hoerger, Sudrajat and Kurniawati, Vectorized Online POMDP Planning (arXiv:2510.27191). VOPP is the vectorized realization of Partially Observable Reference Policy Programming (PORPP): rather than interleaving numerical optimization with expectation estimation, it solves the value function analytically (a log-sum-exp over action preferences) and leaves only expectations to Monte Carlo.

The planner drives three collaborating pieces, none of which contains any planning-policy logic itself:

  • a VectorizedBeliefTree holding the tree topology, per-action visit/reward statistics, and two registered belief fields – preferences (the |A|-vector Psi of eq. 4) and value (the scalar belief value V);

  • a VectorizedGenerativeModel supplying the batched transition / observation / reward / terminal / key kernels; and

  • a root belief represented as a tensor of state particles.

Each planning iteration runs two fully vectorized passes over the tree:

  1. Forward search (Algorithm 2): from the root belief, sample n_p states, and repeatedly – for every live episode in parallel – sample an action from softmax(eta * Psi), push it through the generative model, accumulate the immediate reward on the action node, drop terminated episodes, and append the successor belief nodes. This descends one tree level per step until max_depth is exceeded, where a value heuristic scores the leaf states.

  2. Preference backup (Algorithm 3): from the deepest level up to the root, compute per-action Q = mean_reward + gamma * weighted_future_value and apply the PORPP preference update (eq. 5) Psi <- Psi - L_eta[Psi] + Q on every belief, recomputing each belief’s value V = L_eta[Psi] as it goes.

After the configured budget of iterations the planner returns the root action with the highest preference.

VOPP assumes a fixed, finite representative action set indexed 0 .. num_actions - 1; the model’s action_keys must return exactly that integer index so it can address a column of the dense preferences field.

References

Hoerger, M., Sudrajat, M., & Kurniawati, H. (2026). Vectorized Online POMDP Planning. arXiv:2510.27191. https://arxiv.org/abs/2510.27191

Example

Planning one step in Continuous Light-Dark:

>>> import torch
>>> from POMDPPlanners.environments.light_dark_pomdp.continuous_light_dark_pomdp import (
...     ContinuousLightDarkPOMDP,
... )
>>> from POMDPPlanners.environments.light_dark_pomdp.continuous_light_dark_vectorized_model import (
...     ContinuousLightDarkVectorizedModel,
... )
>>> from POMDPPlanners.planners.vectorized_planners import VOPPPlanner
>>> _ = torch.manual_seed(0)
>>> env = ContinuousLightDarkPOMDP(discount_factor=0.95, is_obstacle_hit_terminal=False)
>>> model = ContinuousLightDarkVectorizedModel(env, device=torch.device("cpu"))
>>> planner = VOPPPlanner(
...     model,
...     num_actions=model.num_actions,
...     num_particles=256,
...     max_depth=5,
...     num_planning_iterations=8,
...     discount_factor=0.95,
... )
>>> root_particles = torch.tensor([[2.0, 2.0]]).repeat(256, 1)
>>> action = planner.plan(root_particles)
>>> 0 <= action < model.num_actions
True
class POMDPPlanners.planners.vectorized_planners.vopp.vopp.VOPPPlanner(model, num_actions, *, temperature=2.0, num_particles=1000, max_depth=10, discount_factor=0.95, num_planning_iterations=100, value_heuristic=None, belief_capacity=1024, action_capacity=1024, value_dtype=torch.float32)[source]

Bases: object

Fully vectorized online POMDP planner (VOPP / PORPP).

The planner owns a VectorizedBeliefTree and repeatedly expands and backs it up entirely with batched tensor operations. A single call to plan() samples particles from the supplied root belief, runs the configured number of forward-search / preference-backup iterations, and returns the greedy root action.

Parameters:
device

Device every tensor lives on (taken from the model).

num_actions

Size of the fixed representative action set.

Example

See the module-level docstring for a runnable example.

plan(root_particles)[source]

Plan from a particle-set root belief and return the greedy action.

The tree is cleared, then num_planning_iterations forward-search / preference-backup passes refine the root action preferences. The action with the highest root preference is returned.

Parameters:

root_particles (Tensor) – [num_root_particles, ds] states representing the current belief; iterations resample num_particles of them with replacement.

Return type:

int

Returns:

The index of the greedy root action in [0, num_actions).

Raises:

ValueError – If root_particles is not a 2-D tensor on the planner’s device.

property tree: VectorizedBeliefTree

The belief tree built by the most recent plan() call.

tree_metrics()[source]

Root-tree analysis metrics for the most recent plan() call.

Returns the same TreeMetrics set the MCTS planners report (root action visit min / max / entropy, number of root actions, root visit count, max depth, leaf flag), making VOPP directly comparable to them.

Return type:

List[PolicyInfoVariable]

Returns:

A list of PolicyInfoVariable describing the current tree.

POMDPPlanners.planners.vectorized_planners.vopp.vopp_episode_runner module

Closed-loop episode driver for the vectorized planner (VOPP / PORPP).

VOPPPlanner only answers a single question – plan(root_particles) returns one greedy action index – so on its own it cannot run a POMDP episode: there is no ground-truth world stepping forward, no belief filter threading information between steps, and no record of the trajectory. This module adds exactly that thin layer.

VOPPEpisodeRunner drives a full closed loop entirely on-device:

  1. plan an action from the current particle belief with the wrapped VOPPPlanner (timed with a CUDA sync so the wall clock is real);

  2. step a ground-truth world state forward and draw an observation;

  3. run a sequential-importance-resampling (SIR) particle filter – propagate the belief particles through the model transition, weight them by the model observation likelihood, and resample – to obtain the next belief.

The world is, by default, the same VectorizedGenerativeModel the planner searches (a faithful “model-is-world” rollout, justified for Continuous Light-Dark by its native-parity test). A caller with a different ground-truth simulator – a real CARLA or Isaac world – injects it through the optional world_transition / world_observation hooks while the belief filter keeps using the vectorized model.

Every quantity the definition-of-done analysis needs is recorded per step: the true state, the belief particle cloud, the chosen action, the immediate reward, the planning wall-clock time, and the planner’s tree metrics.

class POMDPPlanners.planners.vectorized_planners.vopp.vopp_episode_runner.VOPPEpisodeResult(states=<factory>, beliefs=<factory>, action_indices=<factory>, rewards=<factory>, plan_times=<factory>, root_visit_counts=<factory>, reached_goal=False, num_steps=0)[source]

Bases: object

Recorded trajectory and per-step statistics of one VOPP episode.

Parameters:
states

[ds] true states, one per visited step plus the final state.

beliefs

Belief particle clouds [num_particles, ds], one per step.

action_indices

Greedy action index chosen at each step.

rewards

Immediate reward collected at each step.

plan_times

Wall-clock seconds spent inside each plan() call.

root_visit_counts

Planner root visit count (forward-search particle simulations) backing each step’s action.

reached_goal

Whether the world reached a terminal (goal) state.

num_steps

Number of executed actions.

action_indices: List[int]
beliefs: List[Tensor]
num_steps: int = 0
plan_times: List[float]
reached_goal: bool = False
rewards: List[float]
root_visit_counts: List[int]
states: List[Tensor]
property total_plan_time: float

Total wall-clock seconds spent planning across the episode.

property total_root_visits: int

Total forward-search particle simulations across the episode.

class POMDPPlanners.planners.vectorized_planners.vopp.vopp_episode_runner.VOPPEpisodeRunner(planner, model, *, num_belief_particles=1000, max_steps=50, world_transition=None, world_observation=None)[source]

Bases: object

Runs closed-loop POMDP episodes with a VOPPPlanner.

The runner owns the interaction loop but no planning policy of its own: the planner decides actions, the vectorized model supplies the dynamics and the belief filter, and an optional pair of world hooks overrides the ground-truth transition / observation when a real simulator is the world.

Parameters:
num_belief_particles

Size of the particle belief carried between steps.

max_steps

Maximum number of actions per episode.

Example

>>> import torch
>>> from POMDPPlanners.environments.light_dark_pomdp.continuous_light_dark_pomdp import (
...     ContinuousLightDarkPOMDP,
... )
>>> from POMDPPlanners.environments.light_dark_pomdp.continuous_light_dark_vectorized_model import (
...     ContinuousLightDarkVectorizedModel,
... )
>>> from POMDPPlanners.planners.vectorized_planners import (
...     VOPPEpisodeRunner,
...     VOPPPlanner,
... )
>>> _ = torch.manual_seed(0)
>>> env = ContinuousLightDarkPOMDP(discount_factor=0.95, is_obstacle_hit_terminal=False)
>>> model = ContinuousLightDarkVectorizedModel(env, device=torch.device("cpu"))
>>> planner = VOPPPlanner(
...     model, num_actions=model.num_actions, num_particles=128,
...     max_depth=6, num_planning_iterations=8,
... )
>>> runner = VOPPEpisodeRunner(planner, model, num_belief_particles=256, max_steps=20)
>>> initial = torch.tensor([[0.0, 5.0]])
>>> result = runner.run_episode(initial)
>>> result.num_steps >= 1
True
run_episode(initial_state, initial_particles=None)[source]

Run one closed-loop episode and return its recorded trajectory.

Parameters:

  • initial_state (Tensor) – [1, ds] (or [ds]) ground-truth start state.

  • initial_particles (Optional[Tensor]) – Optional [num_particles, ds] initial belief; defaults to the start state replicated across the particle set.

Return type:

VOPPEpisodeResult

Returns:

A VOPPEpisodeResult holding the states, beliefs, actions, rewards, planning times, and root visit counts of the episode.

Raises:

ValueError – If initial_state does not describe a single state.