Context Assistant

What I Built

Context Assistant is a Linux accessibility tool designed to identify the user interface element beneath the cursor and display its keyboard shortcut.

The project connects several layers of the Linux desktop stack—from low-level input events to GNOME Shell, D-Bus, and AT-SPI—to turn raw pointer movement into meaningful application context.

System Architecture

The first milestone deliberately focused on the smallest useful end-to-end path: pointer → application → UI element → shortcut → overlay.

Connecting to Linux Input

The input layer reads mouse and touchpad events directly from /dev/input rather than relying on high-level desktop callbacks. The event-driven pipeline uses poll() so the application can continuously monitor pointer movement without busy-waiting.

Bridging Input to Desktop Context

Raw input events do not contain useful semantic information such as which application is underneath the pointer. To bridge that gap, I used a GNOME Shell extension to obtain desktop context such as cursor coordinates, window information, application ID, and process ID.

The extension communicates this information to the C++ process through D-Bus, creating a bridge between the Linux desktop environment and the low-level application.

Finding the UI Element

Once the application context was known, AT-SPI provided the semantic accessibility layer needed to identify the interface element underneath the cursor.

Instead of treating the screen as a collection of pixels, the system uses accessibility information such as an element's role and name. This allows the assistant to reason about objects such as tabs, buttons, menus, and other application controls.

Debugging the System Boundaries

One of the most important challenges was understanding the boundaries between GNOME Shell and AT-SPI. For example, atspi_accessible_get_id() did not provide the application string identifier I initially expected. The process ID provided a much cleaner bridge between the two systems.

I also encountered coordinate-system mismatches caused by display scaling. Rather than assuming the AT-SPI hierarchy was broken, I isolated the problem by checking each coordinate system independently and comparing the values exposed by GNOME, Xrandr, and AT-SPI.

The D-Bus Bug

Another failure came from a D-Bus signature mismatch. The C++ application expected one message structure while the GNOME extension returned another.

The deeper lesson was that the source code I was editing was not necessarily the code being executed. The active GNOME extension was installed under the user's local GNOME extension directory, so I had to inspect the running artifact rather than only inspecting the repository copy.

From UI Context to Action

Once the system could identify an accessible element, the next step was mapping that semantic element to a known keyboard shortcut.

I intentionally kept shortcut mapping deterministic during the initial implementation rather than introducing AI before the underlying system pipeline was reliable.

The Final Interaction

The final stage is a lightweight GTK overlay that displays the shortcut without stealing focus from the application underneath. The overlay only appears when a known shortcut is available.

This keeps the interaction focused: move the pointer over an interface element, identify its semantic context, and surface the action the user needs.

Engineering Lessons

Context Assistant taught me how difficult systems become when multiple platform boundaries are involved. A failure could originate in Linux input events, GNOME Shell, D-Bus, AT-SPI, coordinate transformations, or the application's own logic.

The most effective debugging strategy was to isolate each boundary, verify its inputs and outputs, form a specific hypothesis, make one change, and test again.

The project ultimately connected a raw hardware event to meaningful desktop context: