Modifying Hand Data¶

When building an application, there may be times when you want your virtual hand to do something that your real hands cannot. For instance:

Reach into the virtual environment to pick up an object that would be out of reach normally.
To have your real hand movements connect to robotic hand visuals that linearly interpolate.
Pose the virtual hand around virtual objects.

Note

Sample scene can be found in:

Assets/Samples/Ultraleap Tracking/x.x.x/XR Examples/1. Basics/3. Manipulating Hand Data.unity

Getting started with Modifying Hand Data¶

This page will show you how incoming tracking data can be manipulated before hand visuals are displayed. This is a powerful benefit when wanting to customize how hand tracking works in your project.

Note

For this example, we assume that you have just finished the Your First Project Guide.

It also assumes that you have an understanding of c# code.

This tutorial is going to teach you how to create your own PostProcessProvider to project your hands from your body.

So the first step is to create a new script.

Note

If you plan on doing more with your hands than is described in this demo, its worth familiarising yourself with functions in PostProcessProvider

../../../../../_images/create-new-script.png

You can delete the base Start and Update functions and inherit from the PostProcessProvider class. Your script should now look like the image below (with your chosen name instead of “TutorialPostProcessProvider”).

Note

If you are getting errors which say “PostProcessProvider cannot be found”, make sure to include “using Leap;” at the top of the script.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Leap;

public class TutorialPostProcessProvider : PostProcessProvider
{

}

Next we need to add “using Leap;” to the top of the script then override the ProcessFrame function from PostProcessProvider. You do this as shown below.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Leap;

public class TutorialPostProcessProvider : PostProcessProvider
{
    public override void ProcessFrame(ref Frame inputFrame)
    {
    }
}

At this point, your actual use case will greatly dictate how you manipulate the data given to you via the “inputFrame” reference. This holds all data about the hands in the latest frame. For more information on how to use this data, see Scripting Fundamentals

Note

If you are comfortable manipulating your own data, skip to step 8.

Add some private variables to the script and serialize them so they are visible in the inspector.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Leap;

public class TutorialPostProcessProvider : PostProcessProvider
{
    [SerializeField]
    private Transform headTransform;

    [SerializeField]
    private float projectionScale = 10f;

    [SerializeField]
    private float handMergeDistance = 0.35f;

    public override void ProcessFrame(ref Frame inputFrame)
    {
    }
}

Next, get the position of the head and calculate the basis of the shoulders inside the ProcessFrame function. This will allow us to project our hand based on the shoulder position.

public override void ProcessFrame(ref Frame inputFrame)
{
    // Calculate the position of the head and the basis to calculate shoulder position.
    if (headTransform == null)
    {
        headTransform = Camera.main.transform;
    }

    Vector3 headPos = headTransform.position;

    var shoulderBasis = Quaternion.LookRotation(
      Vector3.ProjectOnPlane(headTransform.forward, Vector3.up),
      Vector3.up);
}

Underneath this, within ProcessFrame, loop through all hands in the inputFrame and calculate where that hands shoulder is.

Note

We are using some magic numbers here but these are the numbers we have found best represent where the shoulder is.

You can change these numbers but you might get weird and unexpected results.

foreach (var hand in inputFrame.Hands)
{
    // Approximate shoulder position with magic values.
    Vector3 shoulderPos = headPos
                            + (shoulderBasis * (new Vector3(0f, -0.13f, -0.1f)
                            + Vector3.left * 0.15f * (hand.IsLeft ? 1f : -1f)));
}

We then process the hand data, calculating how far from the shoulder the hand is and thus, how far our projected hands should travel from their actual location.

foreach (var hand in inputFrame.Hands)
{
    // Approximate shoulder position with magic values.
    Vector3 shoulderPos = headPos
                            + (shoulderBasis * (new Vector3(0f, -0.13f, -0.1f)
                            + Vector3.left * 0.15f * (hand.IsLeft ? 1f : -1f)));

    // Calculate the projection of the hand if it extends beyond the
    // handMergeDistance.
    Vector3 shoulderToHand = hand.PalmPosition - shoulderPos;
    float handShoulderDist = shoulderToHand.magnitude;
    float projectionDistance = Mathf.Max(0f, handShoulderDist - handMergeDistance);
    float projectionAmount = 1f + (projectionDistance * projectionScale);
}

The final step in code is to set the position of the hand in the frame. We do this by calling “hand.SetTransform()” (which takes in a vector3 for hand position and a Quaternion for hand rotation) at the bottom of our ProcessFrame function.

hand.SetTransform(shoulderPos + shoulderToHand * projectionAmount,
    hand.Rotation);

If you have been following all steps up until this point, your script should look similar to the one in the dropdown below.

Add your new script to a new object inside your scene and add capsule hands as children of your new PostProcessProvider

../../../../../_images/add-hands-to-post-process-provider.png

Both capsule hands should have assigned your new PostProcessProvider to their LeapProvider field. If they have not, assign it now.

Finally, assign the values in your leap provider objects hierarchy.
- Assign your LeapProviderManager to “Input Leap Provider”
- Assign the Main Camera (Inside XRRig > Camera Offset) to “Head Transform”

With that taken care of, you should see two sets of hands in the scene, offset from one another and you are done. Press Play and try out your new projected hands!

../../../../../_images/post-process-provider-success.png