05 · Product & medtech · Case study

Bloom, an operator interface for assistive robots

Extender project · ISIR, Sorbonne University / CNRS · 2026

In short

I designed, built and shipped Bloom, the interface wheelchair users rely on to drive an assistive robotic arm. In September 2026 it became the official Extender operator interface, replacing the previous tablet app.

A wheelchair user drives the Explorer arm, which holds a bottle. Icons show the possible inputs: muscle contraction, voice, brain activity, residual movement and gaze. — The problem in one picture: many possible inputs, one arm, one person who must stay in control. Illustration by ORTHOPUS.

140decision records, one for every architectural choice
960+automated tests, 620 frontend and 342 backend
17seed-app contracts checked on every release
2robots driven end to end in simulation by CI

Problem: Earlier interfaces coupled screen layout, input devices, ROS transport and one robot. Every study, new device or second robot meant new code, and safety was scattered between screen and robot.
Goal: One interface system researchers can reconfigure without a developer, operators can trust, and clinicians can put in front of a participant.
My role: Product framing, UX, architecture and delivery, then onboarding the team onto it.

Seven product decisions, and why

One control model, two surfacesA visual Builder and a kiosk Runtime render the same model, so the operator gets exactly what was reviewed.
Safety lives in the backendAllowlists, rate limits, validation, an audit log and one command owner per session, so no screen can bypass them.
A latched STOP everyone can pressReserved on every screen, it refuses every command source until released, even for a read-only supervisor.
Review before a screen reaches a userAn action-based checklist catches unreachable targets, controls too small to touch and a missing STOP. Findings block instead of warning.
ROS stays at the adapter boundaryThe frontend never imports ROS, so Bloom survived the control-stack rewrite and can serve robots beyond Extender.
Simulation first, CI drives the robotsEvery release runs end to end against both simulations, so hardware time goes to validation.
Ship the apps, not only the toolSeed apps a newcomer can open in ten minutes made the product real for the team before it was finished.

Animated tour of Bloom: composing a screen in the Builder, then driving the arm from the Runtime — Bloom in motion: a screen composed in the Builder, reviewed, then run as a kiosk.

Bloom Builder canvas with a screen being composed from widgets — Left: the Builder canvas. Right: a single-switch scanning profile for operators with very limited movement.

Bloom runtime in single-switch scanning mode, with one control highlighted at a time — Left: the Builder canvas. Right: a single-switch scanning profile for operators with very limited movement.

Users and constraintsWho Bloom is for, and what it had to live with

Operators: People with limited upper-limb mobility, driving the arm from a tablet, joystick, gamepad or a single assistive switch.
Researchers & clinicians: They run studies and need to compose a screen, review it and hand it to a participant safely.
Medical context: Risk-driven safety, traceable behaviour and an audit trail, not only a working demo.
Two robots: Orthopus Explorer and Kinova Gen3, with different limits: Explorer has no slip ring and hard joint limits.
Hardware & team: Lab tablets at 1024 × 600, three languages, and a control stack being rewritten at the same time.

What shippedBloom 0.2.0, MIT license, September 2026

Bloom operator interface showing the Explorer Manager Drive screen with its STOP rail

Input: Touch, keyboard and gamepad composed into one 6-DoF command, plus single-switch scanning profiles.
Apps: Explorer Manager and Kinova Manager: Drive, Positions, Robot feedback, Command sources, Joystick Lab and a Supervisor mirror.
Access: English, Spanish and French shells, guided operator practice, and a debug view with live joint states.
ROS 2: Adapters for the command manager, topics, services, discovery and a gripper camera stream.
Open: Release 0.2.0 under MIT, with tutorials, an operator safety guide and decision records.

What I would do nextHardware acceptance, small screens, safety zones

Hardware acceptance on the target tablets and both arms, with the assistive switches the scanning profiles were designed for.
The 1024 × 600 collapse layouts from the design handoff, then a native-speaker pass on Spanish and French.
Safety zones in the interface, once the controller exposes the Cartesian constraints the QP library already implements.
A short explainer of the hybrid control mode, so researchers, clinicians and operators describe it the same way.

The wider Extender workPlatform, control stack, research and coordination

Platform: Moved the workspace to Ubuntu 24.04 and ROS 2 Jazzy, and maintain the entry point, build guide and shared Python environment.
First interface: Built, then retired, the first tablet teleoperation path once Bloom covered it.
Control: Connected Bloom to the new three-layer control architecture, and drove visual servoing and AprilTag detection from it.
Demonstrator: Rebased the Petanque assistive demonstrator (PEPR Robotics) onto the new stack.
Onboarding: Wrote the organisation profile, tutorials and contribution rules for newcomers.
Research: Shared control and assistive manipulation, paper submissions (ICRA), coordination with clinicians and partners.