Pose Detection Components

The PoseDetector derived components provide a convenient way to detect what a user’s hand is doing. For example, detect when the fingers of a hand are curled or extended, whether a finger or palm is pointing in a particular direction, or whether the hand or fingertip is close to one of a set of target objects.

Detectors can be combined together using a Logic Gate. The LogicGateDetector is itself a pose detector component that logically combines two or more other detectors to determine its own state. Need a thumb’s up gesture? Combine a thumb extended detector with a thumb pointing upward detector using a logic gate.

Detectors generate event dispatchers calls when they activate or deactivate. This makes it easy to bind to a detector straight from blueprint.

Each detector can draw debug primitives (only visible in editor) that make it easy to see its configured conditions and whether it is currently active or inactive.

Using a PoseDetector component

To use a PoseDetector derived component, either attach the component to a hand’s SkeletalMeshComponent or create the component in a new actor, in which case, upon construction, setup it’s HandSkeletalMesh variable to point to the desired hand’s SkeletalMeshComponent.

The OnPoseDetected and OnPoseLost event dispatchers can then be bound to, in order to react to the detection event (for example thumbs up and then thumbs up no longer detected). See the example at UltraleapTracking/Content/PoseDetection/Pawn/PoseDetectionLowPolyHand for how to do this along with the example scene at UltraleapTracking/PoseDetection/ExampleScenes/PoseDetection.

Combining Pose Detectors

Combine multiple detectors to create more complex behaviour using the LogicGateDetector component. A logic gate takes any number of other detectors as input, and outputs a single boolean. It’s derived from the PoseDetector component, so it also dispatches OnPoseDetected and OnPoseLost events. You can set logic gates to be AND gates (all inputs must be true for the output to be true) or OR gates (the output is true if any input is true). You can also negate the output to configure the gate as a NAND or NOR gate.

Since a logic gate is, itself, a PoseDetector, you can hook up multiple logic gates to create arbitrarily complex logic.

Pose Detector Cookbook

The following collection of ideas illustrate how to use pose detectors to implement behaviors and interaction in your application.

Thumb’s Up

To detect a Thumb's Up use an ExtendedFingerDetector to check that the thumb is the only extended finger and a FingerDirectionDetector to check detect when the thumb is pointing up. Combine these detectors with an AND-type logic gate:

• ExtendedFingerDetector - configure the component so that the Thumb must be extended and the other fingers must not be extended.

• FingerDirectionDetector - configure the component so that:

  • Finger Name = [*Thumb bone name*]

  • Pointing Direction = (0, 0, 1)

  • Pointing Type = Relative To World

  • On and Off angles = set as desired

• LogicGateDetector - Set the Gate Type to AND Gate and bind to the event dispatchers to the Actor or Component that should react to the thumb’s up.

Camera-Facing Open Hand

To detect when a palm is facing the camera use an ExtendedFingerDetector to check that all the fingers are extended and a PalmDirectionDetector to check detect when the palm is facing the camera. Combine these detectors with an AND-type logic gate:

• ExtendedFingerDetector - configure the component so that all fingers must be extended.

• PalmDirectionDetector - configure the component so that:

  • Pointing Direction = (0, 1, 0)

  • Pointing Type = Relative To Horizon

  • On and Off angles = set as desired

• LogicGateDetector - Set the Gate Type to AND Gate and hook up the event dispatchers to the Actor or Component that should react to the thumb’s up.

Making Your Own Pose Detector

To create your own Pose Detector classes, create a child blueprint of the PoseDetector component. Then override the CheckPose function in your class to implement detection logic. Within the logic, call SetPoseActive when the pose is detected, and no longer detected. The parent PoseDetector component will then dispatch the relevant events for the two states. Reference the other PoseDetector derived components to see how a CheckPose implementation works.

If you want to draw primitives to help visualise the pose detection algorithm, override DrawDebug in the custom component.

Ready made Pose Detectors

ExtendedFingerDetector

Spheres turn yellow when detected

ProximityDetector

Middle sphere turns green when detected, detection lost when proximity further than blue sphere

PinchDetector

Middle pentagon turns green when detected, detection lost when pinch further than blue pentagon diameter

PalmDirectionDetector

Triggers when palm crosses direction threshold

FingerDirectionDetector

Triggers when a given finger points in a given direction over a given angle tolerance (see cone extending from the finger)