I’ve tried many things since then and it worked, so I’ll share it with you.
targetPosition.set(detail.position.x, detail.position.y, detail.position.z)
stablePosition = targetPosition.sub(camera.position)
stablePosition.multiplyScalar(1 / detail.scale)
stablePosition.add(camera.position)
By using stablePosition created in this way, we can set the position without changing the scale of the target and the apparent size will not change.
This method seems to be a good choice for continuous physics simulation.
However, it is much more stable to use the normal position and scale for each.
I think this is a niche need like mine, so if anyone else has the same need, please give it a try.