Vision Tracking Gimbal Demo

Objective

This demo was designed to demonstrate real-time vision tracking and motion control using an EtherCAT-enabled system. The goal is to show how a Basler industrial camera and OpenCV can work in tandem with a high-performance CPU and RSI’s RMP EtherCAT Motion Controller to track moving objects with high precision. By running a 4 kHz control loop and responding instantly to live video input, the system benchmarks the latency and performance of the sensor-to-actuator loop.

Key objectives include:

• Demonstrate real-time response to visual input using a camera and gimbal system
• Validate and measure latency in the sensor-to-actuation loop
• Benchmark control performance on hardware such as the EtherCAT Mitsubishi MR-J5 servo drives and motors.
• Showcase deterministic motion control and I/O logic using RealTimeTasks in RMP
• Prove that developers can build real-time tracking systems without needing complex real-time programming skills

This application is ideal for engineers and developers searching for practical solutions in real-time robotic tracking, vision-guided motion control, and laser targeting systems.

‍

System Overview

The demo integrates a complete vision-guided motion control stack, combining industrial-grade hardware and software components:

• Processing: iPC400 Industrial PC running Linux with PREEMPT_RT enabled to ensure real-time deterministic performance and motion, and executing OpenCV for color tracking and vision processing.
• Motion Control: RSI’s RMP EtherCAT Motion Controller with the RapidCode API, enabling real-time deterministic motion and I/O control.
• Drives & Motors: Mitsubishi MR-J5 EtherCAT series servos for precise two-axis gimbal movement.
• Camera: Basler industrial camera with the Pylon SDK for 2-3 millisecond fast image acquisition.

‍

Implementation

This section outlines the key engineering components that enable real-time, vision-guided motion control using RSI’s RMP platform and supporting technologies.

Real-Time Operating System (iPC400 Industrial PC + Linux + PREEMPT_RT)

At the foundation of this system is the RSI iPC400 Industrial PC, powered by an Intel® Core™ i5 CPU with Time Coordinated Computing (TCC) support. This processor is specifically optimized for real-time latencies, enabling deterministic scheduling and ultra-low jitter performance. When combined with Linux and PREEMPT_RT, the iPC400 delivers a reliable real-time operating environment for executing vision processing and motion-control logic. By providing predictable response times and rock-solid timing precision, the iPC400 ensures that RSI’s platform, not just the cameras, is at the heart of achieving high-performance vision-guided motion control.

Real-Time Performance

PREEMPT_RT makes the Linux kernel fully preemptible, allowing high-priority motion control and vision-processing tasks to interrupt background processes without delay. This reduces interrupt latency to the microsecond range and ensures smooth and consistent execution of the demo’s 4 kHz control loop.

Vision acquisition, OpenCV-based tracking, and motion commands run as high-priority real-time threads, while non-critical services are isolated on other cores to minimize interference.

Kernel & System Tuning

To achieve maximum determinism, the system applies real-time tuning best practices:

• Disabling CPU frequency scaling and power-saving states for stable timing.
• Assigning critical tasks to dedicated CPU cores while isolating non-real-time workloads.
• Using real-time scheduling policies (FIFO / Round-Robin) to prioritize motion and vision threads.
• Minimizing unpredictable latency from network, storage, and I/O drivers.

Dual-Layer Determinism

Linux with PREEMPT_RT provides the OS-level guarantees, ensuring real-time responsiveness across the entire software stack. On top of this, RSI’s RMP RealTimeTasks deliver deterministic execution at the application level, locking vision updates and motion commands into a precise 4 kHz cycle. Together, this dual approach ensures low-latency, stable, and predictable control - a requirement for industrial-grade vision-guided automation.

Object Tracking (OpenCV)

The system utilizes YUV (YCbCr)-based object tracking, implemented in OpenCV, to detect a colored ball in real-time. Compared to RGB, the YUV color space separates luminance (Y) from the blue (U/Cb) and red (V/Cr) chrominance, making it more robust under variable lighting conditions. A binary mask isolates the target color, and the largest contour is used to estimate the ball’s position in the frame. This position is then mapped to motion control commands.

Image Acquisition (Basler + Pylon SDK)

A Basler acA640-300gc (GigE) industrial camera, interfaced via the Pylon SDK, captures full VGA (640×480) video at ~174 FPS (~6 ms). The camera's settings - including manual exposure and gain - are fine-tuned using a saved feature file, which ensures consistent image quality across runs. This manual configuration reduces jitter and improves frame timing, which is critical for maintaining tracking accuracy.

Motion Control (RMP Axis Objects)

Motion commands are executed using Axis objects from the RapidCode API, which interface directly with the Mitsubishi MR-J5 servo drives over EtherCAT. The system employs proportional velocity control with clamping to ensure responsive but stable movements of the gimbal. This prevents overshooting and keeps the laser smoothly locked onto the moving target.

Real-Time Execution (RealTimeTasks)

The core tracking and motion logic runs inside RealTimeTasks, a key feature of the RMP EtherCAT Motion Controller. This environment allows the system to execute both vision processing and motion commands at a deterministic 4 kHz control loop rate, without relying on an external real-time operating system. The task scheduler ensures tight control over execution order, timing, and resource use.

User Interface (Avalonia UI + gRPC, HTTP, & SSH)

A lightweight, cross-platform UI was built with Avalonia UI .NET, providing live system status, real-time tracking feedback, and remote control capabilities. The interface communicates with the RMP motion controller’s rapidserver over gRPC, the camera server over HTTP, and the host PC over SSH. This architecture enables modular, scalable, and platform-independent interaction between the UI and the embedded control application.

The user interface is divided into distinct regions to simplify interaction and monitoring. The top bar provides the application title and standard window controls. On the left panel, configuration cards allow setup of the key services required for the demo - such as the RapidServer, SSH, and camera server - along with a log window for connection and status messages. The right panel displays real-time status and mapping fields from the RMP controller, EtherCAT network, and active RealTimeTasks. The bottom bar contains action buttons for enabling motion, camera, and layout options. At the center, the main display area presents both the real-time render (tracking visualization) and the live camera feed, enabling users to observe detection accuracy and system response in real time.

Let’s focus on one of the key areas - the right panel (RMP RTTasks) tab. This section lists all RMP RealTimeTasks currently running or previously executed, providing visibility into how each task contributes to real-time motion control based on image tracking.

Real-Time Task Performance Overview Cards

Each card shows the performance of a RealTimeTask. The goal is for each task to run consistently and reliably within its configured cycle period.

For example, with the MoveMotors RealTimeTask, the task has been running continuously for over 2 days, more than 700 million cycles at a 4 kHz (250 us) period. Within each cycle, the user’s code, which updates the motor position based on the latest camera frame, executes in an average of just 0.0007 ms. Even at its longest, the recorded execution time was only 0.0481 ms, staying far below the configured 250 us cycle time and leaving extraordinary performance headroom.

Below are all the fields explained:

• Top-Right Progress Ring: Indicates whether the task is currently executing or not.
• Cycle: Displays the user-defined RTTask period, which represents the deterministic cycle time at which the task executes.
• Average: Displays the average time it takes for the user’s code/logic inside the task to execute.
• Max: Displays the longest execution for the user’s code/logic.
• Cycle, Average, & Max Bar: Provides a visual comparison:
  
  • Green = Cycle period.
  • Blue = Average execution time.
  • Red = Maximum execution time.
  • The goal is to see the Green bar (Cycle) to the right, meaning the Blue (Average) stays below it and the Red (Max) never exceeds it.
• Runtime: Shows the approximate total time the task has been running, derived from Cycle × Cycles.
• Cycles: Displays the total number of executions since the task started, based on its cycle period.
• Start Delta Max: Shows the maximum observed time between the start of one cycle and the start of the next.

‍

RMP Configuration Status

Another important section to highlight is the Configuration tab card, which displays the RMP Status. This view is valuable because it shows:

• The RMP motion controller counter
• The EtherCAT network counter
• All the global variables from the RealTimeTasks under the TaskManager

RTTasks Global Variables are especially important because they provide a way to retrieve information directly from the user’s code and logic inside each task. This makes it possible to surface and display meaningful data - such as detection results, system readiness, or error counts - in a clear and accessible way.

Demo System Diagram

Coming soon.

‍

Results

The demo was evaluated over several days of continuous operation to validate the performance and reliability of the real-time vision tracking and motion control system. Performance was measured at the level of individual RealTimeTask, each running at its own deterministic rate. The core motion-control tasks were executed at 250 us (4 kHz) intervals, while vision-processing tasks ran at the rate the Basler camera could return images/frames at around 6 ms per frame. This task-level scheduling confirmed that the system could sustain low-latency responsiveness without overloading compute resources.

Tracking accuracy remained consistently above 99.9%, with the camera staying aligned to the moving target during extended runs. Input from the Basler industrial camera was sustained at 174 fps at full VGA (640×480), ensuring stable visual input for the tracking pipeline. On the motion side, the system maintained deterministic servo loop execution at 4 kHz over EtherCAT, providing smooth and reliable motor responses.

The RealTimeTasks themselves played a central role in maintaining deterministic behavior across the pipeline. The Initialize task prepared the system state, configured the camera, and set up the multi-axis controller. DetectBall processed each captured frame to locate the ball’s center and radius, generating new motion targets when necessary at a 250 us cycle. MoveMotors consumed those targets and drove the MultiAxis system as close as possible to the commanded positions, all while respecting axis limits and cycling at a 250 us. Finally, OutputImage safely retrieved processed frames and wrote them to a temporary file for HTTP server access in a lower-priority task running at a 16.5 ms cycle. Together, these tasks provided a tightly synchronized workflow that linked vision and motion in a stable, repeatable manner.

Across continuous testing, the system operated without fault, underscoring the robustness of both the hardware and software stack. The combination of stable frame acquisition, accurate vision processing, and deterministic motion control confirmed the feasibility of integrating real-time vision-guided automation with minimal overhead.

‍

Video (See it in action)

Coming soon.

‍

Key Takeaways

This demo validated several important principles relevant to real-time vision and motion control integration in industrial automation:

Real-time Execution ─ Linux with PREEMPT_RT provides real-time and predictable timing

Running on Linux with PREEMPT_RT enabled ensured predictable scheduling, minimized jitter, and enabled the coordination of vision and motion threads under tight deadlines. Leveraging Linux with PREEMPT_RT was critical to maintaining deterministic performance from image acquisition through motion execution.

Motion Controller ─ RMP delivers deterministic high-frequency servo control

The RMP motion controller consistently executed servo loops at 4 kHz (250 us) over EtherCAT, maintaining stable, repeatable motion profiles. This confirmed the controller’s capability to support industrial applications that demand precise, deterministic, and high-speed motion control.

RMP RealTimeTasks ─ simplifies real-time development

Developers can achieve deterministic timing and coordination of vision + motion without needing to manage real-time threads, scheduling, or inter-process communication themselves..

Camera ─ real-time safe I/O is essential

Using the camera API in a wait-free manner ensures deterministic performance without introducing unpredictable delays.

Vision ─ modular tracking with powerful libraries

Since RTTasks can directly execute C++ code, developers can leverage libraries like OpenCV to build adaptable, reusable tracking pipelines for real-time robotic applications.

Demo App ─ Full-stack development is easier than ever

From image acquisition to UI control and visualization, the system shows how frameworks like .NET, Avalonia UI, gRPC, OpenCV, and AI can be combined to rapidly build scalable, production-ready automation applications with minimal overhead.

Real-World Applications

This demo represents more than just a technical exercise - it’s a practical foundation for real-time automation systems across various industries. The combination of deterministic motion control, high-speed vision processing, and modular software architecture enables a wide range of advanced applications:

Laser-Based Inspection and Alignment

In semiconductor manufacturing or PCB inspection, precise laser alignment is critical. The gimbal system demonstrated here can be adapted to automatically track and align lasers with micro-scale targets in real time.

Vision-Guided Robotics 

Industrial arms performing pick-and-place or assembly tasks benefit from synchronized vision and motion. The HSV tracking logic and control loop can be extended to multi-axis robotic platforms that need to react in milliseconds to changes in visual input.

Automated Testing and Calibration

Precision test systems for optical components, sensors, or mechanical devices often require motion profiles driven by visual cues. This demo showcases how such systems can be implemented using standard components and software tools.

Intelligent Material Handling

In warehouse automation or conveyor sorting, vision-guided gimbals can direct lasers or indicators toward dynamically identified items. With scalable UI and cross-platform support, this architecture can be deployed across edge devices in logistics environments.

R&D Prototyping and Interactive Demos

Research institutions and engineering teams can use this system as a platform for experimenting with real-time control concepts, vision-guided feedback loops, or custom actuation logic without needing a real-time OS or PLC framework.

‍

FAQ

How can I build a real-time vision tracking system with a gimbal and a camera?

You can use a Basler industrial camera, OpenCV for object tracking, and a 2-axis gimbal controlled via the RMP EtherCAT Motion Controller, which handles deterministic motion control at 4 kHz, enabling responsive and stable tracking.

How do I synchronize OpenCV tracking with servo motor control?

Synchronization between OpenCV tracking and servo motor control in this system is achieved through a tightly integrated, real-time architecture centered on the RMP EtherCAT Motion Controller's RealTimeTasks. The process begins with the Basler industrial camera capturing high-speed video frames via the Pylon SDK, which are then immediately processed by OpenCV for HSV-based object detection to determine the colored ball's precise position. This vision data is critical input for the RealTimeTasks, specifically the "DetectBall" task, which runs at a deterministic 250 us cycle and generates new motion targets based on the detected ball's location. Concurrently, the "MoveMotors" RealTimeTask, also operating at a 250 us cycle, consumes these targets and directly commands the Mitsubishi MR-J5 servo motors via the RapidCode API to adjust the 2-axis gimbal, thereby locking the laser onto the target. This seamless and high-frequency exchange of information, facilitated by PREEMPT_RT ensuring kernel-level real-time responsiveness and the RMP's deterministic RealTimeTasks, effectively minimizes latency in the sensor-to-actuator loop, guaranteeing precise and synchronized vision-guided motion control.

Can I use a Basler camera with EtherCAT motion control?

Yes. In this application, a Basler camera with the Pylon SDK is processed in real time to guide a gimbal via EtherCAT-controlled servo motors. The entire loop is coordinated using the RMP Motion Controller.

What’s the best motion controller for real-time vision-guided automation?

The RSI RMP Motion Controller is designed for applications like vision-guided robotics, laser alignment, tracking systems, and more. It provides deterministic behavior, supports 4 kHz update rates, and integrates easily with tools like OpenCV, Pylon SDK, gRPC, and many other modern popular libraries.

Is there a motion control platform that works with gRPC and cross-platform UI?

Yes. This system uses gRPC for communication and a cross-platform UI built in Avalonia UI .NET to interface with the RMP Motion Controller RapidCode API (C#). This makes it easy to deploy and monitor the system across different environments.

What’s a good example of a real-time motion control demo with vision input?

This demo combines a 2-axis gimbal, real-time OpenCV tracking, a Basler camera, and RSI’s RMP EtherCAT Motion Controller to dynamically follow a colored round target in real time.

‍

GitHub Repository

Coming soon.