Media3 ExoPlayer AudioSink receives PCM, but my Vosk live translation callback is never reached

I am developing an Android application in Kotlin using AndroidX Media3 / ExoPlayer. I am trying to build a live speech-to-text and translation pipeline using Vosk while video playback continues normally.

The problem is that the live translation never produces any visible result. I added several diagnostic checkpoints to determine exactly where the audio pipeline stops, but only the first test is ever displayed. The later tests are never reached.

I am looking for help identifying the exact point in the Media3 audio pipeline where the problem occurs.

Current architecture

The intended audio path is:

Media3 / ExoPlayer
        |
        v
Audio renderer
        |
        v
LiveVoskAudioSink
        |
        +----> normal DefaultAudioSink
        |          |
        |          v
        |       AudioTrack
        |
        +----> PCM callback
                   |
                   v
             VoskAudioBridge
                   |
                   v
          PCM frame alignment
                   |
                   v
          worker thread / queue
                   |
                   v
        16 kHz mono PCM conversion
                   |
                   v
            4096-byte batches
                   |
                   v
        LiveSpeechTranslationPipeline
                   |
                   v
                 Vosk
                   |
                   v
          speech recognition
                   |
                   v
             translation
                   |
                   v
          subtitleView on screen

The important requirement is that the speech-recognition path must never interrupt or block normal video/audio playback.

What I currently have

I replaced the normal audio sink with a wrapper called:

class LiveVoskAudioSink(
    private val delegate: DefaultAudioSink,
    private val onPcmBuffer: (ByteBuffer, Format) -> Unit
) : AudioSink

Its handleBuffer() currently does this:

override fun handleBuffer(
    buffer: ByteBuffer,
    presentationTimeUs: Long,
    encodedAccessUnitCount: Int
): Boolean {

    val format = configuredFormat

    if (format != null && buffer.hasRemaining()) {

        Log.e(
            "ELROMHY4V4_SINK_DIAG",
            "===== SINK HANDLEBUFFER ===== bytes=${buffer.remaining()} " +
                "rate=${format.sampleRate} " +
                "channels=${format.channelCount} " +
                "encoding=${format.pcmEncoding}"
        )

        try {
            val copy = ByteBuffer.allocate(buffer.remaining())
            val duplicate = buffer.duplicate()
            copy.put(duplicate)
            copy.flip()

            Log.e(
                "ELROMHY4V4_SINK_DIAG",
                "===== CALLING VOSK CALLBACK ===== bytes=${copy.remaining()}"
            )

            onPcmBuffer(copy, format)

            Log.e(
                "ELROMHY4V4_SINK_DIAG",
                "===== VOSK CALLBACK RETURNED OK ====="
            )

        } catch (error: Throwable) {

            Log.e(
                "ELROMHY4V4_SINK_DIAG",
                "===== VOSK CALLBACK THREW EXCEPTION =====",
                error
            )
        }
    }

    return delegate.handleBuffer(
        buffer,
        presentationTimeUs,
        encodedAccessUnitCount
    )
}

The delegate is a normal DefaultAudioSink.

I also disable audio offloading:

audioSink.setOffloadMode(
    AudioSink.OFFLOAD_MODE_DISABLED
)

The custom sink is returned from a DefaultRenderersFactory:

override fun buildAudioSink(
    context: Context,
    enableFloatOutput: Boolean,
    enableAudioTrackPlaybackParams: Boolean
): AudioSink {
    return audioSink
}

VoskAudioBridge

The VoskAudioBridge implements:

TeeAudioProcessor.AudioBufferSink

It contains an unbounded:

private val audioQueue =
    LinkedBlockingQueue<RawPcmPacket>()

The Media3 audio callback does not perform Vosk processing directly. It only copies/aligned PCM and puts it into the queue.

The worker thread then consumes the queue:

Media3 audio callback
        |
        v
Raw PCM queue
        |
        v
Vosk worker thread

The worker converts the PCM using a stateful converter:

workerPcmConverter.convert(
    packet.data,
    packet.sampleRate,
    packet.channels
)

The resulting audio is accumulated until it reaches:

VOSK_BATCH_BYTES

which is currently 4096 bytes.

Then it is passed to:

livePipeline.acceptPcm(voskBlock)

The important diagnostic result

I added several tests.

The first test is reached earlier in the application and appears correctly.

However, the second test is inside the VoskAudioBridge worker:

if (!audioToLiveTestSent && voskBlock.isNotEmpty()) {

    audioToLiveTestSent = true

    audioToLiveTestListener?.invoke(
        "[TEST 2] PCM وصل إلى VoskAudioBridge\nbytes=${voskBlock.size}"
    )
}

The listener is installed immediately after creating the bridge:

voskBridge =
    VoskAudioBridge(
        liveSpeechTranslationPipeline!!
    )

voskBridge?.setAudioToLiveTestListener { testText ->
    runOnUiThread {
        subtitleView.text = testText
        subtitleView.visibility = View.VISIBLE
        subtitleView.alpha = 1f
        subtitleView.bringToFront()
    }
}

voskBridge?.startVoskWorker()

But TEST 2 never appears.

None of the later tests appear either.

The only visible test is the first test.

This is what I am trying to determine:

> Is the audio data actually reaching LiveVoskAudioSink.handleBuffer() at all, or is my custom AudioSink not being used by Media3, or is the PCM callback/queue/worker path being bypassed somewhere before VoskAudioBridge?

Additional diagnostic logs

Inside LiveVoskAudioSink.handleBuffer() I added:

===== SINK HANDLEBUFFER =====
===== CALLING VOSK CALLBACK =====
===== VOSK CALLBACK RETURNED OK =====

Inside VoskAudioBridge.handleBuffer():

===== LIVE PCM QUEUED =====

Inside the Vosk worker:

===== VOSK WORKER RECEIVED PCM =====

Before sending to Vosk:

===== WORKER VOSK BATCH =====
===== SENDING PCM TO VOSK =====

The key question is that the application still plays the video/audio normally, but the live translation path produces no result and the diagnostic chain stops before TEST 2.

What I need help with

I would like someone experienced with Media3 ExoPlayer AudioSink / audio renderers / PCM processing to inspect this architecture and tell me:

1. Is overriding DefaultRenderersFactory.buildAudioSink() the correct way to intercept decoded PCM in current Media3?

2. Is LiveVoskAudioSink.handleBuffer() guaranteed to receive decoded PCM with this configuration?

3. Could DefaultAudioSink be receiving encoded audio or a different format than I expect?

4. Could the custom AudioSink be bypassed because of the renderer/audio configuration?

5. Is there another Media3 API that should be used instead, such as an AudioProcessor, TeeAudioProcessor, or another supported PCM interception point?

6. Is copying the ByteBuffer before passing it to the delegate safe in this location?

7. Could the AudioSink callback be running but my UI diagnostic be misleading because of thread/lifecycle issues?

8. Where exactly would you place diagnostic logging to prove the precise point where the audio stops?

9. What is the recommended architecture for extracting decoded PCM from Media3/ExoPlayer and feeding it asynchronously into Vosk without affecting playback?

I am specifically looking for the actual root cause, not just a general suggestion to add more logging.

The application is written in Kotlin, uses AndroidX Media3 / ExoPlayer, and the speech recognition engine is Vosk.

I can provide the complete relevant Kotlin source files and Logcat output if needed.