I want to measure and analyze user movements and gestures in the UI in order to refine the application user experience. I had imagined that feature-tracking libraries (like EQATEC or Preemptive's Runtime Intelligence) would allow this. However, this does not appear to be the case.
Ideally, I'd like to be able to instrument a UI and then capture mouse and keyboard navigation gestures to display via a heat-map.
My searches have come up empty. Does anything OSS or commercial exist here?
On
Have a look at this application. It does not do what you want but it might be useful as a starting point for implementing the features you need: Perceptor: An artificially intelligent guided navigation system for WPF
On
Try http://iographica.com it creates lines where the mouse cursor moved, and circles where the mouse cursor stopped, the bigger the circle, the longer it was stopped there.
On
I know Snoop is more a tool for inspecting the visual tree of a WPF application, but it does also capture events (especially it captures these with information on which elements they happened, how they traveled in the visual tree and where they get handled). As it is open source you might want to extract that part about tracking events out and log this information to your needs.
On
Gma.UserActivityMonitor project and blindly convert it to .NET 4.0In HookManager.Callbacks.cs file make following changes.
using System.Diagnostics;Replace
s_MouseHookHandle = SetWindowsHookEx(
WH_MOUSE_LL,
s_MouseDelegate,
Marshal.GetHINSTANCE(
Assembly.GetExecutingAssembly().GetModules()[0]),
0);
With
using (Process curProcess = Process.GetCurrentProcess())
using (ProcessModule curModule = curProcess.MainModule)
{
s_MouseHookHandle = SetWindowsHookEx(
WH_MOUSE_LL,
s_MouseDelegate,
GetModuleHandle(curModule.ModuleName), 0);
}
Replace
s_KeyboardHookHandle = SetWindowsHookEx(
WH_KEYBOARD_LL,
s_KeyboardDelegate,
Marshal.GetHINSTANCE(
Assembly.GetExecutingAssembly().GetModules()[0]),
0);
With
using (Process curProcess = Process.GetCurrentProcess())
using (ProcessModule curModule = curProcess.MainModule)
{
s_KeyboardHookHandle = SetWindowsHookEx(
WH_KEYBOARD_LL,
s_KeyboardDelegate,
GetModuleHandle(curModule.ModuleName), 0);
}
In HookManager.Windows.cs add following two lines anywhere in HookManager class's definition.
[DllImport("kernel32.dll", CharSet = CharSet.Auto)]
public static extern IntPtr GetModuleHandle(string lpModuleName);
Now you should be able to build this and keep it aside. Now starts the WPF part.
Now Replace the MainWindow.xaml with following XAML make sure to check class name and project-name, I just created WpfApplication1 for testing.
<Window x:Class="WpfApplication1.MainWindow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
Title="MainWindow" Height="350" Width="525">
<Grid>
<Grid.RowDefinitions>
<RowDefinition Height="0.30*"/>
<RowDefinition Height="0.70*"/>
</Grid.RowDefinitions>
<StackPanel Grid.Row="0" Orientation="Horizontal">
<StackPanel Orientation="Vertical">
<CheckBox Name="MouseMove" Padding="5" Content="Mouse Move" Width="120" Height="30" Click="checkBoxOnMouseMove_CheckedChanged"/>
<CheckBox Name="MouseClick" Padding="5" Content="Mouse Click" Width="120" Height="30" Click="checkBoxOnMouseClick_CheckedChanged"/>
<CheckBox Name="MouseDown" Padding="5" Content="Mouse Down" Width="120" Height="30" Click="checkBoxOnMouseDown_CheckedChanged"/>
</StackPanel>
<StackPanel Orientation="Vertical">
<CheckBox Name="MouseUp" Padding="5" Content="Mouse Up" Width="120" Height="30" Click="checkBoxOnMouseUp_CheckedChanged"/>
<CheckBox Name="MouseDouble" Padding="5" Content="Mouse Double" Width="120" Height="30" Click="checkBoxMouseDoubleClick_CheckedChanged"/>
<CheckBox Name="MouseWheel" Padding="5" Content="Mouse Wheel" Width="120" Height="30" Click="checkBoxMouseWheel_CheckedChanged"/>
</StackPanel>
<StackPanel Orientation="Vertical">
<CheckBox Name="KeyDown" Padding="5" Content="Key Down" Width="120" Height="30" Click="checkBoxKeyDown_CheckedChanged"/>
<CheckBox Name="KeyPress" Padding="5" Content="Key Press" Width="120" Height="30" Click="checkBoxKeyPress_CheckedChanged"/>
<CheckBox Name="KeyUp" Padding="5" Content="Key Up" Width="120" Height="30" Click="checkBoxKeyUp_CheckedChanged"/>
</StackPanel>
<StackPanel Orientation="Vertical">
<TextBlock Name="labelMousePosition" Text="x={0:####}; y={1:####}"/>
<TextBlock Name="labelWheel" Text="Wheel={0:####}"/>
</StackPanel>
</StackPanel>
<TextBlock Name="textBoxLog" Text="START" VerticalAlignment="Stretch" HorizontalAlignment="Stretch" Grid.Row="1" ScrollViewer.VerticalScrollBarVisibility="Visible"/>
</Grid>
</Window>
Now add following code to MainWindow class defined MainWindow.xaml.cs file.
#region Check boxes to set or remove particular event handlers.
private void checkBoxOnMouseMove_CheckedChanged(object sender, EventArgs e)
{
if ((bool)MouseMove.IsChecked)
{
HookManager.MouseMove += HookManager_MouseMove;
}
else
{
HookManager.MouseMove -= HookManager_MouseMove;
}
}
private void checkBoxOnMouseClick_CheckedChanged(object sender, EventArgs e)
{
if ((bool)MouseClick.IsChecked)
{
HookManager.MouseClick += HookManager_MouseClick;
}
else
{
HookManager.MouseClick -= HookManager_MouseClick;
}
}
private void checkBoxOnMouseUp_CheckedChanged(object sender, EventArgs e)
{
if ((bool)MouseUp.IsChecked)
{
HookManager.MouseUp += HookManager_MouseUp;
}
else
{
HookManager.MouseUp -= HookManager_MouseUp;
}
}
private void checkBoxOnMouseDown_CheckedChanged(object sender, EventArgs e)
{
if ((bool)MouseDown.IsChecked)
{
HookManager.MouseDown += HookManager_MouseDown;
}
else
{
HookManager.MouseDown -= HookManager_MouseDown;
}
}
private void checkBoxMouseDoubleClick_CheckedChanged(object sender, EventArgs e)
{
if ((bool)this.MouseDouble.IsChecked)
{
HookManager.MouseDoubleClick += HookManager_MouseDoubleClick;
}
else
{
HookManager.MouseDoubleClick -= HookManager_MouseDoubleClick;
}
}
private void checkBoxMouseWheel_CheckedChanged(object sender, EventArgs e)
{
if ((bool)MouseWheel.IsChecked)
{
HookManager.MouseWheel += HookManager_MouseWheel;
}
else
{
HookManager.MouseWheel -= HookManager_MouseWheel;
}
}
private void checkBoxKeyDown_CheckedChanged(object sender, EventArgs e)
{
if ((bool)KeyDown.IsChecked)
{
HookManager.KeyDown += HookManager_KeyDown;
}
else
{
HookManager.KeyDown -= HookManager_KeyDown;
}
}
private void checkBoxKeyUp_CheckedChanged(object sender, EventArgs e)
{
if ((bool)KeyUp.IsChecked)
{
HookManager.KeyUp += HookManager_KeyUp;
}
else
{
HookManager.KeyUp -= HookManager_KeyUp;
}
}
private void checkBoxKeyPress_CheckedChanged(object sender, EventArgs e)
{
if ((bool)KeyPress.IsChecked)
{
HookManager.KeyPress += HookManager_KeyPress;
}
else
{
HookManager.KeyPress -= HookManager_KeyPress;
}
}
#endregion
//##################################################################
#region Event handlers of particular events. They will be activated when an appropriate check box is checked.
private void HookManager_KeyDown(object sender, System.Windows.Forms.KeyEventArgs e)
{
textBoxLog.Text += (string.Format("KeyDown - {0}\n", e.KeyCode));
}
private void HookManager_KeyUp(object sender, System.Windows.Forms.KeyEventArgs e)
{
textBoxLog.Text += (string.Format("KeyUp - {0}\n", e.KeyCode));
}
private void HookManager_KeyPress(object sender, System.Windows.Forms.KeyPressEventArgs e)
{
textBoxLog.Text += (string.Format("KeyPress - {0}\n", e.KeyChar));
}
private void HookManager_MouseMove(object sender, System.Windows.Forms.MouseEventArgs e)
{
labelMousePosition.Text = string.Format("x={0:0000}; y={1:0000}", e.X, e.Y);
}
private void HookManager_MouseClick(object sender, System.Windows.Forms.MouseEventArgs e)
{
textBoxLog.Text += (string.Format("MouseClick - {0}\n", e.Button));
}
private void HookManager_MouseUp(object sender, System.Windows.Forms.MouseEventArgs e)
{
textBoxLog.Text += (string.Format("MouseUp - {0}\n", e.Button));
}
private void HookManager_MouseDown(object sender, System.Windows.Forms.MouseEventArgs e)
{
textBoxLog.Text += (string.Format("MouseDown - {0}\n", e.Button));
}
private void HookManager_MouseDoubleClick(object sender, System.Windows.Forms.MouseEventArgs e)
{
textBoxLog.Text += (string.Format("MouseDoubleClick - {0}\n", e.Button));
}
private void HookManager_MouseWheel(object sender, System.Windows.Forms.MouseEventArgs e)
{
labelWheel.Text = string.Format("Wheel={0:000}", e.Delta);
}
#endregion
9.Build and run your WPF App, you might need to add using Gma.UserActivityMonitor; directive in MainWindow.
10.All the credit goes to George Mamaladze who originally built it, you might want to send a thank you note to him, also check the distribution license for this code from original author if you want to make money out of it.
11.I just made couple of minor changes to make it work with WPF. Thanks for following this long post, I'm not sure how to share this code, that is why providing instructions like this.
12.I'm seriously sucking at formatting the code well on SO, can someone leave a comment on how to format code, XML well. Also someone help me format snippets in this post. Thank you.
After trying a number of approaches, including the ones here as well as using UIAutomation Events and ETW for WPF, I've decided on a simple attachment of a handler to WPF events. This allows me to not only capture the event data, but also the UIElement which has the users attention, so it is much easier to trace the user action and intention. Without this, I'd need to capture a visual of the screen and make a visual determination of what is going on.
Here's a sample:
While it is not shown here, once I get the
UIElementI perform logging and can even then use theUIElement.DataContextto determine the state of the ViewModel which is driving the view so we can find patterns of usage during certain workflows and data-states as well as visual states. We can then get reports on this, as well as differentiate and compare our heat maps by paths through workflow and data values.