How to make a lips image (.png) follow the mouth of the Face Mesh?

🧑‍💻 Technical Support
, , , ,
1

How to make a lips image (.png) follow the mouth of the Face Mesh?

I’m using the Studio, I don’t have the pro plan.
So I just created a new Project in the Studio, cloning the (Basic) Face Effects Project that comes with the Stereo Glasses as a .glb. and it has a .png in the Face Mesh.

I changed the Face Mesh texture to my own image, that has a mouth on it. Since it is linked straight to the Face Mesh I though it would automatically start to follow the Face’s mouth. But it didn’t.

What is missing to link the Actual Mesh to the Face ???
In the original project it has a text in the forehead that is “Static”. In my case it is a mouth, so it is “Flexible”. How could I make this link? What is missing??

image
image358Ă—725 62.4 KB

image
image1914Ă—940 123 KB

2

Hi @Gabriel_Magalhaes - thanks for your patience here. It seems like this was a bug in the sample project. We have fixed the issue so if you re-clone the project the face mesh will update with the user’s expressions.

This did uncover a separate issue with the “Mirrored Display” camera property, so that has been temporarily disabled in the sample project.

If you want to fix the problem in your existing project you need to:

  1. Disable “Mirrored Display” in the Camera properties
  2. Add a new Face to the scene
  3. Add a new Face Mesh from the Face properties
  4. Set “Color” texture (and “Opacity” texture, if desired)
  5. Remove the old Face entity

Hope this helps - I will follow up again once the issue with “Mirrored Display” has been resolved.

1 Like
3

It worked now. I had the mirroring issue, but I could fix it. Somehow when I left the project and went back again, it seems it returned but I just checked and unchecked the mirroring in the camera properties and it worked again.

Thanks for that.

I’m having issues with images now. But I might create another topic because it is related to another of the template projects.

4

I’m having more issues with the Face Mesh.

  1. The mesh is not polished enough, so when the mouth opens the edges are really sharp and visible. I tried subsurface dividing with three, but I couldn’t do that on Studio.
  2. The other problem is: the Face mesh for some reason has the option “Show Backface” checked. And it creates an issue big issue specifically with the mouth Mesh. It is the “Show Backface” option, it is not the “Backface Culling” as you can see in the image below.

image
image771Ă—489 52.4 KB

I had the same problem in the past when I was using another tool, that was using the same Face Mesh, uv mapping and all face binding related things, that your tool is using as well. I wonder it might be, ARCore, ARKit, A-Frame, THREE, I didn’t understand yet. But the fact is, both tools are having the same mal-functioning around the mouth in the Face Mesh deformation. When it comes to 3D face binding mesh.

In the other tool that I was using in the past I fixed this issue using my own custom mesh, that was nothing but the same mesh with Subdivision Surface (2) and the option “Show Backface” disabled (exported from blender).

Is there any chances to use my own custom Face Mesh on Niantic Studio (it need to still be well linked to the users face)?
If so, how could I do that?