HELP! Image Target Lost with video Issue — Visual Studio ECS Code

Hello, I am trying to pause/stop a video playing on a plane within an image target, once that image target is lost. The below code does not throw up errors, however it does not work and I never get the console.log to show ‘Image lost — resetting videoPlane material.’ PLEASE HELP!

 import * as ecs from '@8thwall/ecs'

ecs.registerComponent({
  name: 'PauseVideoOnExit',
  schema: {
    imageTarget: ecs.eid,
    videoPlane: ecs.eid,
  },

  stateMachine: ({world, eid, schemaAttribute}) => {
    let videoElement: HTMLVideoElement | null = null

    return ecs.defineState('default')
      .initial()
      .onEnter(() => {
        const {imageTarget, videoPlane} = schemaAttribute.get(eid)

        if (!imageTarget || !videoPlane) {
          console.warn('PauseVideoOnExit: Missing imageTarget or videoPlane.')
          return
        }

        // Delay to allow material and texture to initialize
        setTimeout(() => {
          const videoPlaneObject = world.three.entityToObject.get(videoPlane)

          if (!videoPlaneObject) {
            console.warn('PauseVideoOnExit: No three.js object for videoPlane entity.')
            return
          }

          const {material} = videoPlaneObject as any

          if (!material) {
            console.warn('PauseVideoOnExit: No material found on videoPlane object.')
            return
          }

          const materials = Array.isArray(material) ? material : [material]
          let foundVideoElement: HTMLVideoElement | null = null

          for (const mat of materials) {
            const texture = mat.map
            const image = texture?.image

            if (image instanceof HTMLVideoElement) {
              foundVideoElement = image
              break
            } else {
              console.log('PauseVideoOnExit: material.map.image is NOT a video element. Found:', image?.constructor?.name)
            }
          }

          if (!foundVideoElement) {
            console.warn('PauseVideoOnExit: No HTMLVideoElement found in any material map image.')
            return
          }

          videoElement = foundVideoElement

          world.events.addListener(imageTarget, 'xrimagelost', () => {
            console.log('Image target lost — pausing and resetting video.')

            if (videoElement && !videoElement.paused) {
              videoElement.pause()
              videoElement.currentTime = 0
            }
          })
        }, 500)  // Wait 0.5s to ensure texture has loaded
      })
  },
})

Hi! I’ve added a ‘Video’ space to our Image Target sample project. This should help get you started.

George Butler you’re a hero! Been beating my head against the wall trying to figure this out. Seriously this made my day.

Not to overstay my welcome, but I have used the UI Video Controls ECS code (template) in my project. So now if I pause the video then go off target, then return to the target, the video starts automatically with the play button back over the video in a weird way. Any way to combine these? Again, I don’t want to stretch my ask.

Here is the UI code (from 8th Wall):

import * as ecs from '@8thwall/ecs'

ecs.registerComponent({
  name: 'Video Controls UI',
  schema: {
    // @required
    background: ecs.eid,
    // @required
    playbackImage: ecs.eid,
  },
  schemaDefaults: {
  },
  data: {
  },
  stateMachine: ({world, eid, schemaAttribute, dataAttribute}) => {
    const toPause = ecs.defineTrigger()
    const toPlaying = ecs.defineTrigger()

    ecs.defineState('setup')
      .initial()
      .listen(eid, ecs.events.VIDEO_CAN_PLAY_THROUGH, (e) => {
        toPause.trigger()
      })
      .onTrigger(toPause, 'paused')

    ecs.defineState('paused')
      .onEnter(() => {
        const {background, playbackImage} = schemaAttribute.get(eid)

        ecs.Ui.mutate(world, background, (cursor) => {
          cursor.backgroundOpacity = 0.5
        })

        ecs.Ui.mutate(world, playbackImage, (cursor) => {
          cursor.image = 'assets/icons/play-button.png'
          cursor.opacity = 1
        })

        ecs.VideoControls.set(world, eid, {
          paused: true,
        })
      })
      .listen(eid, ecs.input.SCREEN_TOUCH_START, (e) => {
        toPlaying.trigger()
      })

      .onTrigger(toPlaying, 'playing')

    ecs.defineState('playing')
      .onEnter(() => {
        const {playbackImage} = schemaAttribute.get(eid)

        ecs.Ui.mutate(world, playbackImage, (cursor) => {
          cursor.image = 'assets/icons/pause-button.png'
        })

        ecs.VideoControls.set(world, eid, {
          paused: false,
        })
      })
      .onTick(() => {
        const {background, playbackImage} = schemaAttribute.get(eid)

        ecs.Ui.mutate(world, background, (cursor) => {
          cursor.backgroundOpacity -= 0.1
        })

        ecs.Ui.mutate(world, playbackImage, (cursor) => {
          cursor.opacity -= 0.1
        })
      })
      .listen(eid, ecs.input.SCREEN_TOUCH_START, (e) => {
        toPause.trigger()
      })
      .onTrigger(toPause, 'paused')
  },
})

1 Like

No problem! :smiley: That’s what I’m here for.

I would approach this by combining the two components, which you might find easier working from the Video sample project since it already has the UI elements built over the video. You just need to add the event listener to the playing state so that on lost it transitions to the “paused” state and on the paused state attach an event listener that goes to the “playing” state when the target is found.

If you get stuck I can look into updating the Video sample project with an Image Target example.

I am a bit stuck, is it possible to update that Video sample project? Thanks again George!

I’ve added a Image Targets space to the Video sample project: Studio: Video | 8th Wall | 8th Wall

Let me know if that helps!

Incredible George! Thank you!…

Quick question (maybe should start a new topic). In the 8th Wall project library I have been playing around with your “Spatial Audio” example. I cloned the project and also replicated the .ts code in my working project. In both cases, the audio requires me to click the button (boombox) twice to make it start.

It works great, I am just unsure if the need for two clicks is my device or this could be resolved? Any guidance would be deeply appreciated.

I’ll take a look at the project, I might have misconfigured the state machine or something similar during an update.

1 Like

Hey George, thanks again for all your help last week! We’re loving the idea of using video and audio in our experience. Based on your awesome guidance last week we were able to have the videos play/pause PLUS they stopped when the image target was lost…

Is it possible to do the exact same experience on Audio — where the sound stops when the target is lost?

I would do it like this:

import * as ecs from '@8thwall/ecs'

ecs.registerComponent({
  name: 'Pause Audio on Lost',
  schema: {
    imageTargetName: ecs.string,
  },
  schemaDefaults: {
    imageTargetName: 'My Image Target',
  },
  stateMachine: ({world, eid, schemaAttribute, dataAttribute}) => {
    ecs.defineState('default')
      .initial()
      .listen(world.events.globalId, 'reality.imagefound', (e) => {
        const {imageTargetName} = schemaAttribute.get(eid)
        const {name} = e.data as any

        if (name === imageTargetName) {
          ecs.Audio.mutate(world, eid, (c) => {
            c.paused = false
          })
        }
      })
      .listen(world.events.globalId, 'reality.imagelost', (e) => {
        const {imageTargetName} = schemaAttribute.get(eid)
        const {name} = e.data as any

        if (name === imageTargetName) {
          ecs.Audio.mutate(world, eid, (c) => {
            c.paused = true
          })
        }
      })
  },
})

You’d attach this component to the same entity that has the Audio component and also make sure to give it the correct name of the Image Target in the Inspector.


AMAZING, George! This worked brilliantly.

I am having a few other audio issues with my project — that also happen when I test in the 8th Wall " Spatial Audio" template. I have tried to learn/help myself and I keep running into issues:

I have 12 total image targets and would like a unique audio clip (in each) that would need to be triggered by clicking a play icon (similar to the ‘Spacial Audio’ template). But, when I put my audio in each target and apply the custom component either two things happen upon testing:

— Pushing one audio play icon plays an audio clip from a different target (yes I am certain my component is set correctly ; )
— …or… It takes two gesture taps to get the audio started.

Again, I am deeply grateful for any advice, and am trying to learn myself.

Follow up here in the spirit of trouble-shooting. It seems (which may be the core issue)… that if I tap anywhere — even before an image is targeted — one of my audio clips play.

Again, I am able to replicate this on the 8th Wall Spatial Audio example.

What if you turn off autoplay in the Audio properties?

I’ve tried ‘Autoplay’ on and off and still get same results.

I do have an update though that is headed in a positive direction. I replicated the 'Video UI Controls" ECS code and mimicked it for audio. Feel free to use the code (below). The first audio clip I try works with one click. However, whatever next audio clip(s) I try require double taps to start. We’re soooooo close!

The code:

import * as ecs from '@8thwall/ecs'

ecs.registerComponent({
  name: 'Audio Controls UI',
  schema: {
    background: ecs.eid,
    playbackImage: ecs.eid,
    audioPlayer: ecs.eid,
    imageTargetName: ecs.string,
  },
  schemaDefaults: {
    imageTargetName: '',
  },
  stateMachine: ({world, eid, schemaAttribute}) => {
    const toPause = ecs.defineTrigger()
    const toPlaying = ecs.defineTrigger()

    // Helper function to handle image lost event logic
    function onImageLost(e: any) {
      const {name} = e.data as any
      const {imageTargetName, audioPlayer, background, playbackImage} = schemaAttribute.get(eid)

      if (name === imageTargetName) {
        console.log('[imagelost] MATCH – pausing audio and updating UI')

        ecs.Audio.set(world, audioPlayer, {paused: true})

        ecs.Ui.mutate(world, playbackImage, (cursor) => {
          cursor.image = 'assets/icons/Audio-ON.png'
          cursor.opacity = 1
        })

        toPause.trigger()
      }
    }

    ecs.defineState('setup')
      .initial()
      .listen(schemaAttribute.get(eid).audioPlayer, ecs.events.AUDIO_CAN_PLAY_THROUGH, () => {
        console.log('[setup] AUDIO_CAN_PLAY_THROUGH fired — starting in paused state')
        toPause.trigger()
      })
      .listen(world.events.globalId, 'reality.imagelost', onImageLost)
      .onTrigger(toPause, 'paused')

    ecs.defineState('paused')
      .onEnter(() => {
        console.log('[paused] Entered')
        const {background, playbackImage, audioPlayer} = schemaAttribute.get(eid)

        ecs.Ui.mutate(world, playbackImage, (cursor) => {
          cursor.image = 'assets/icons/Audio-ON.png'
          cursor.opacity = 1
        })

        ecs.Audio.set(world, audioPlayer, {paused: true})
      })
      .listen(eid, ecs.input.SCREEN_TOUCH_START, () => {
        console.log('[paused] Tap – going to playing')
        toPlaying.trigger()
      })
      .listen(world.events.globalId, 'reality.imagelost', onImageLost)
      .onTrigger(toPlaying, 'playing')

    ecs.defineState('playing')
      .onEnter(() => {
        console.log('[playing] Entered')
        const {playbackImage, audioPlayer} = schemaAttribute.get(eid)

        ecs.Ui.mutate(world, playbackImage, (cursor) => {
          cursor.image = 'assets/icons/Audio-OFF.png'
          cursor.opacity = 1
        })

        ecs.Audio.set(world, audioPlayer, {paused: false})
      })
      .onTick(() => {
        const {background, playbackImage} = schemaAttribute.get(eid)

        ecs.Ui.mutate(world, playbackImage, (cursor) => {
          cursor.opacity = Math.max(0, cursor.opacity + 1)
        })
      })
      .listen(eid, ecs.input.SCREEN_TOUCH_START, () => {
        console.log('[playing] Tap – going to paused')
        toPause.trigger()
      })
      .listen(world.events.globalId, 'reality.imagelost', onImageLost)
      .onTrigger(toPause, 'paused')
  },
})

Can you land your changes and share the project with the support workspace so I can take a look?

1 Like

Got it, just did. The only two image targets that are close to working are “Anderson Ferry” and “Oscars” targets/scenes.

It’s probably because the page requires a user gesture before the audio can be played. This is likely why you’re only seeing it work after the second tap. (First tap is consumed and used as the OK signal for future audio).

Totally makes sense. Anyway around this?

Typically developers add a Splash Screen to the start of their project to get around this, ensuring that they’ve captured a user gesture before starting any of the content.

THANK YOU GEORGE!

Sorry it has taken sometime to get back to you. I was taking a small break.

Ok… so… the splash screen worked brilliantly and resolved my issues (just for a moment).

I am nearing completion of my project (SO CLOSE) and a new crop of issues have popped up. Mostly audio related.

Quick/simple description of the project and what I’m trying to achieve:
— I’ve got 12 image targets that relate to images from a printed brochure
— 10 of the targets house GLB models (with animations) and two have videos
— Each target also includes a playable audio file (cued by tapping an icon)
— All audio and animation elements stop on rolling off the image target, audio has to be re-tapped to start once the image is targeted again. This includes one of the videos, as well.

Everything worked fantastic when I was building the project up until my 9th/10th target. Then, the audio started working inconsistent. Sometimes the buttons would work, sometimes not. Every once and a while one button would even start the audio of another target (this was very rare, yet it would happen). Plus (also, just occasionally) an audio button from a random target would start one of the videos in another target.

Any insights?

Again, I am very, very, very grateful for your guidance through my project. Every step of the way you have helped me resolve issues and enhance the experience.

Nearing the finish line for sure!