Face Tracking and DecalGeometry

🧑‍💻 Technical Support
1

Hi everyone,

I’m working with Face Tracking and trying to apply a “tattoo” or “sticker” effect on a face mesh. To do this, I started using THREE.DecalGeometry.

I created a function to detect when there’s an intersection between the user’s finger and the face mesh:

...
if (intersects.length > 0) {
  const { point, face, object } = intersects[0];
  const localPoint = this.el.object3D.worldToLocal(point.clone());
  const faceNormal = face.normal.normalize();

  this.createBox(localPoint, faceNormal); // Debug
  this.createDecal(localPoint, faceNormal, object);
}
...

First, I created a function to add a box at the intersection point between the face and the finger, which is working fine:

createBox(localPoint, faceNormal) {
  const boxEl = document.createElement('a-box');
  boxEl.setAttribute('position', localPoint);
  boxEl.setAttribute('depth', 0.1);
  boxEl.setAttribute('height', 0.1);
  boxEl.setAttribute('width', 0.1);
  boxEl.setAttribute('material', { transparent: true, opacity: 0.25 });

  boxEl.addEventListener('object3dset', () => {
    const { quaternion } = boxEl.object3D;
    quaternion.setFromUnitVectors(new THREE.Vector3(0, 1, 0), faceNormal);
  });

  this.el.appendChild(boxEl);
}

Next, I tried to achieve the same effect using DecalGeometry, but I encountered some issues with the position and rotation of the decal mesh created:

  • The decal mesh is placed above the head.
  • The decal mesh’s rotation is off by about 180 degrees.
  • The entire face mesh seems to be duplicated within the decal mesh.
createDecal(localPoint, faceNormal, object) {
    // object.geometry.computeVertexNormals(); // TEST

    const eye = localPoint.clone()
    eye.add(faceNormal)

    const rotation = new THREE.Matrix4()
    rotation.lookAt(eye, localPoint, THREE.Object3D.DefaultUp)
    const euler = new THREE.Euler()
    euler.setFromRotationMatrix(rotation)

    const s = 5
    const decalGeometry = new THREE.DecalGeometry(
      object,
      localPoint,
      rotation,
      new THREE.Vector3(s, s, s)
    )
    const decal = new THREE.Mesh(decalGeometry, this.decalMaterial.clone())
    this.el.object3D.add(decal)
}

I suspect the issues might be related to how the face object is being tracked, which could be affecting the position and rotation values I’m applying.

If anyone has feedback or ideas about what could be going wrong, it would be very helpful. Thank you for reading!

2

Would you be able to record a video of the current behavior?

3

Thank you for your reply, here a video link