Physical interfaces for
virtual diagnostics.
A Meta Quest 3 application paired with a physical Arduino Uno and ESP32-based hardware control panel. Collaborative manipulation of volumetric MRI/CT medical imagery across networked headsets via Photon Fusion 2.
- Role
- XR Systems & Hardware Engineer
- Year
- 2026 - Ongoing
- Domain
- Medical Imaging · Industrial Simulation
- Platform
- Meta Quest 3 · Custom Hardware
- Core stack
- Unity · Photon Fusion 2 · Arduino
- Hardware
- Arduino Uno · ESP32
Tangible control in spatial computing.
Interacting with complex 3D medical datasets like MRI and CT scans in virtual space can be cumbersome using standard controllers. The challenge was to bridge the tactile precision of a physical control panel with the infinite bounds of spatial computing.
This project introduces an Arduino Uno and ESP32-based physical control panel that wirelessly communicates with a Meta Quest 3 headset. Sensors, dials, and physical inputs translate directly into the manipulation of volumetric medical imagery.
Through bidirectional real-time serial communication, users can collaboratively inspect, slice, and interact with the 3D data across networked headsets via Photon Fusion 2, enabling unprecedented remote surgical planning and educational simulations.
Hardware to software bridge.
Designing and assembling the physical control panel using Arduino Uno and ESP32 microcontrollers.
Establishing a low-latency bidirectional data flow between physical sensors and the Unity simulation via a wireless ESP32 bridge.
Implementing rendering techniques to display complex medical MRI/CT scan data as interactive 3D volumes in Unity.
Integrating Photon Fusion 2 to ensure all physical inputs and volumetric manipulations are instantly synchronized across multiple users.
Under the hood.
Custom embedded hardware
A bespoke control panel utilizing Arduino Uno and ESP32 handling multiple sensor inputs, providing a tactile interface that standard VR controllers cannot mimic.
Volumetric rendering
Medical MRI and CT scan data points are translated into fully rotatable, sliceable 3D volumes enabling real-time spatial inspection from any angle.
Bidirectional serial comms
An optimized data stream where physical panel states affect the virtual environment, while virtual simulation states can trigger physical feedback mechanisms.
Networked collaboration
Photon Fusion 2 architecture that seamlessly synchronizes the dense volumetric data states and hardware inputs across a shared multiplayer XR session.