Is it possible to disable world tracking for a single Aframe entity?

Hi everyone,
I notice significant jittering when using a 360 image inside a world-tracked effect. For this project, I am using a combination of world tracking and image tracking to enter a portal fixed to an image target position. The idea is to immerse the user in a 360 image once they enter the portal so that we avoid world tracking imperfections when the user is looking around.
When placing a-sky element (with the 360 image) in a scene with disableWorldTracking: true, the effect is smooth, however when having it in a scene with world tracking enabled, jitter in rotation is present.

As far as I know I cannot switch disableWorldTracking to false in runtime without reloading the whole scene. Is there any way I can overcome this issue?

The scene setup is just this with the a-sky element directly under the a-scene

<a-scene
  landing-page
  xrextras-loading
  xrextras-gesture-detector
  xrextras-runtime-error
  renderer="colorManagement: true;"
  controller
  xrweb="allowedDevices: any; disableDefaultEnvironment: true; disableWorldTracking: false">

  <a-assets>
...
  </a-assets>

  <a-camera
    id="camera"
    position="0 0 0"
    raycaster="objects: .cantap"
    cursor="fuse: false; rayOrigin: mouse;"
    proximity-trigger="target: .trigger"
    >
  </a-camera>

  <a-sky id="environment-sphere-camera" rotation="0 0 10" visible="false" material="src: #skybox-img; transparent: true; opacity: 1; fog:false;"></a-sky>

  <xrextras-named-image-target id="image-target" name="image-target"></xrextras-named-image-target>