Face-Tracking: split face effect

🧑‍💻 Technical Support
1

Hi everyone,

Based on the face texture project from Evan Carlson I’m trying to create a face filter effect with a “Split face” mesh.

I took the faceBlank GLB model you shared in different face tracking templates. Using Blender and the Bisect tool, I cut the mesh into two parts and saved these two parts as a single GLB file. These two parts are still forming a face mesh but I can split the face in 2 now.

My goal is to use the face tracking texture and apply this texture onto the two face meshes so each face mesh will render the expected face texture, matching the side of the face they are rendering. Finally, I can move and apply any animation to these two face meshes…

Here is the component I have:

HTML:

<a-entity gltf-model="#face-blank-cut" position="0 0 0" face-split-texture="debug: true"></a-entity>

Code:

export const faceSplitTextureComponent = {
  schema: {
    debug: {type: 'boolean', default: false},
  },
  init() {
    const faceTextureGltf_ = new THREE.Texture()
    const materialGltf = new THREE.MeshBasicMaterial({map: faceTextureGltf_, side: THREE.DoubleSide})

    let faceGltf

    this.el.addEventListener('model-loaded', () => {
      faceGltf = this.el.getObject3D('mesh')

      let isMoved = false
      faceGltf.traverse((node) => {
        if (node.material) {
          node.material = materialGltf

          if (this.data.debug) {
            if (!isMoved) {
              isMoved = true
              node.position.x += 0.5
            } else {
              node.position.x -= 0.5
            }
          }
        }
      })
    })

    const onxrloaded = () => {
      window.XR8.addCameraPipelineModule({
        name: 'cameraFeedPipeline',
        onUpdate: (processCpuResult) => {
          if (!processCpuResult) {
            console.log('no processCpuResult')
            return
          }
          const result = processCpuResult.processCpuResult
          if (result.facecontroller && result.facecontroller.cameraFeedTexture) {
            const {cameraFeedTexture} = processCpuResult.processCpuResult.facecontroller
            const texPropsGltf = this.el.sceneEl.renderer.properties.get(faceTextureGltf_)
            texPropsGltf.__webglTexture = cameraFeedTexture
          }
        },
      })
    }
    window.XR8 ? onxrloaded() : window.addEventListener('xrloaded', onxrloaded)

    const show = (event) => {
      const {uvsInCameraFrame} = event.detail

      const vertices = new Float32Array(event.detail.vertices.length * 3)

      faceGltf.children.forEach((_, index) => {
        // Update vertex positions.
        for (let i = 0; i < event.detail.vertices.length; ++i) {
          vertices[i * 3] = event.detail.vertices[i].x
          vertices[i * 3 + 1] = event.detail.vertices[i].y
          vertices[i * 3 + 2] = event.detail.vertices[i].z
        }
        faceGltf.children[index].geometry.setAttribute('position', new THREE.BufferAttribute(vertices, 3))
        faceGltf.children[index].geometry.attributes.position.needsUpdate = true

        const uvs = new Float32Array(uvsInCameraFrame.length * 2)
        for (let i = 0; i < uvsInCameraFrame.length; ++i) {
          uvs[i * 2] = uvsInCameraFrame[i].u
          uvs[i * 2 + 1] = uvsInCameraFrame[i].v
        }

        faceGltf.children[index].geometry.setAttribute('uv', new THREE.BufferAttribute(uvs, 2))

        // Update vertex normals.
        const normals = faceGltf.children[0].geometry.attributes.normal
        for (let i = 0; i < event.detail.normals.length; ++i) {
          normals[i * 3] = event.detail.normals[i].x
          normals[i * 3 + 1] = event.detail.normals[i].y
          normals[i * 3 + 2] = event.detail.normals[i].z
        }
        faceGltf.children[index].geometry.attributes.normal.needsUpdate = true
      })
    }

    this.el.sceneEl.addEventListener('xrfacefound', show)
    this.el.sceneEl.addEventListener('xrfaceupdated', show)
  },
}

Here is the result I’m getting:
wip_split_face (1)

  • The first mesh while updating vertices, UVs, and normals is rebuilt has a full face mesh
  • The second mesh is working ok

I’m not sure why I’m getting this result, maybe there is an issue when I’m exporting the GLB from Blender after using the Bisect tool.

I understand it’s a specific technical issue, I’m trying to figure it out, but if by any chance you have any advice or solutions on how to apply this face texture to both meshes, I would really appreciate it.

Thank you for reading.