Using gestures within data handler function of flutter driver extension

2 min readMar 11, 2021

Sometimes you need to test event related to mouse movements within your E2E tests. Flutter driver does not provide yet native functionality for this activities but you can write your own within data handler function of the enableFlutterDriverExtension() function call.

First of all you will need to find element that points to the widget over which you want to perform a mouse move event. You can use collectAllElementsFrom function from flutter_test package for that.

final element = collectAllElementsFrom(
    WidgetsBinding.instance.renderViewElement,
    skipOffstage: false)
    .firstWhere((element) =>
    element.widget.key.toString().contains(yourSearchedValueKey));

Having the element you can get its render box which will give you access to the widget coordinates and size:

final RenderBox box = element.renderObject as RenderBox;
final centerOfTheWidgetOffset =
    box.localToGlobal(box.size.center(Offset.zero));

To create a movement action we will need to tell our function where to move pointer to. Lets pretend that we need to stay within the widget and move the pointer diagonally from center to its corner. The movement vector will point to this direction:

var moveOffset = Offset(box.size.width / 2, box.size.height / 2);

Now it is time to create something that is called a gesture in Flutter. Gesture is an action that you do with your input device (touchscreen, mouse, keyboards, etc). You will need packages flutter/widgets.dart,flutter/gestures.dart for that. Gestures are created using WidgetController, which is something that deals with actions related to widgets. But because we pretend to control widgets not during widget tests but within live application using flutter driver we need to use LiveWidgetController.

final wc =
    LiveWidgetController(WidgetsFlutterBinding.ensureInitialized());
final gesture = await wc.createGesture(kind: PointerDeviceKind.mouse); // this gesture will be simulating mouse
await gesture.moveTo(centerOfTheWidgetOffset);
await gesture.moveBy(moveOffset);

And that is all. You function is ready. Here is how the full code of the handler function will look like:

import 'dart:convert';
import 'package:flutter/widgets.dart';
import 'package:flutter/gestures.dart';
import 'package:flutter_test/flutter_test.dart';Future<String> dataHandler(String payload) async {
  if(payload == "moveMouseGesture"){
    final element = collectAllElementsFrom(
        WidgetsBinding.instance.renderViewElement,
        skipOffstage: false)
      .firstWhere((element) =>
        element.widget.key.toString()
          .contains(yourSearchedValueKey));
    final RenderBox box = element.renderObject as RenderBox;
    final centerOfTheWidgetOffset =
        box.localToGlobal(box.size.center(Offset.zero));
    var moveOffset =
        Offset(box.size.width / 2, box.size.height / 2);
    final wc = LiveWidgetController(
        WidgetsFlutterBinding.ensureInitialized());
    final gesture = await wc.createGesture(
        kind: PointerDeviceKind.mouse);
    await gesture.moveTo(centerOfTheWidgetOffset);
    await gesture.moveBy(moveOffset);
    return jsonEncode({
        "movedFrom": centerOfTheWidgetOffset.toString(),
        "movedBy": moveOffset.toString()});
  }
}
...
  enableFlutterDriverExtension(handler: dataHandler);

Now from the tests you can execute this command:

FlutterDriver driver = ...;
await driver.requestData("moveMouseGesture");

And that will simulate the mouse move event over the necessary widget within app.

Using gestures within data handler function of flutter driver extension

Written by Artur Korobeynyk