How to interleave a non-interleaved AudioBufferList inside a render callback?

I'm working on a project that involves streaming audio from an AVPlayer video player object into libpd using an MTAudioProcessingTap. For the process loop of the tap, I used PdAudioUnits render callback code as a guide; but I realized recently that the audio format expected by libpd is not the same as the audio coming from the tap — that is, the tap is providing two buffers of non-interleaved audio data in the incoming AudioBufferList, whereas libpd expects interleaved samples. I don't think I can change the tap itself to provide interleaved samples.

Does anyone know of a way I can work around this?

I think that I need to somehow create a new AudioBufferList or float buffer and interleave the samples in place; but I'm not quite sure how to do this and it seems like it would be expensive. If anyone could give me some pointers I would greatly appreciate it!

Here is my code for installing my tap:

- (void)installTapWithItem:(AVPlayerItem *)playerItem {            MTAudioProcessingTapCallbacks callbacks;            callbacks.version = kMTAudioProcessingTapCallbacksVersion_0;      callbacks.clientInfo = (__bridge void *)self;      callbacks.init = tap_InitCallback;      callbacks.finalize = tap_FinalizeCallback;      callbacks.prepare = tap_PrepareCallback;      callbacks.unprepare = tap_UnprepareCallback;      callbacks.process = tap_ProcessCallback;            MTAudioProcessingTapRef audioProcessingTap;      if (noErr == MTAudioProcessingTapCreate(kCFAllocatorDefault, &callbacks, kMTAudioProcessingTapCreationFlag_PreEffects, &audioProcessingTap))      {          NSLog(@"Tap created!");                    AVAssetTrack *audioTrack = [playerItem.asset tracksWithMediaType:AVMediaTypeAudio].firstObject;          AVMutableAudioMixInputParameters* inputParams = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:audioTrack];          inputParams.audioTapProcessor = audioProcessingTap;                    AVMutableAudioMix* audioMix = [AVMutableAudioMix audioMix];          audioMix.inputParameters = @[inputParams];          playerItem.audioMix = audioMix;      }  }  

And my tap_ProcessCallback:

static void tap_ProcessCallback(MTAudioProcessingTapRef tap, CMItemCount numberFrames, MTAudioProcessingTapFlags flags, AudioBufferList *bufferListInOut, CMItemCount *numberFramesOut, MTAudioProcessingTapFlags *flagsOut)  {      OSStatus status = MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut, flagsOut, nil, numberFramesOut);      if (noErr != status) {          NSLog(@"Error: MTAudioProcessingTapGetSourceAudio: %d", (int)status);          return;      }            TapProcessorContext *context = (TapProcessorContext *)MTAudioProcessingTapGetStorage(tap);            // first, create the input and output ring buffers if they haven't been created yet      if (context->frameSize != numberFrames) {          NSLog(@"creating ring buffers with size: %ld", (long)numberFrames);          createRingBuffers((UInt32)numberFrames, context);      }            //adapted from PdAudioUnit.m      float *buffer = (float *)bufferListInOut->mBuffers->mData;            if (context->inputRingBuffer || context->outputRingBuffer) {                    // output buffer info from ioData          UInt32 outputBufferSize = bufferListInOut->mBuffers[0].mDataByteSize;          UInt32 outputFrames = (UInt32)numberFrames;          //        UInt32 outputChannels = bufferListInOut->mBuffers[0].mNumberChannels;                    // input buffer info from ioData *after* rendering input samples          UInt32 inputBufferSize = outputBufferSize;          UInt32 inputFrames = (UInt32)numberFrames;          //        UInt32 inputChannels = 0;                    UInt32 framesAvailable = (UInt32)rb_available_to_read(context->inputRingBuffer) / context->inputFrameSize;          while (inputFrames + framesAvailable < outputFrames) {              // pad input buffer to make sure we have enough blocks to fill auBuffer,              // this should hopefully only happen when the audio unit is started              rb_write_value_to_buffer(context->inputRingBuffer, 0, context->inputBlockSize);              framesAvailable += context->blockFrames;          }          rb_write_to_buffer(context->inputRingBuffer, 1, buffer, inputBufferSize);                    // input ring buffer -> context -> output ring buffer          char *copy = (char *)buffer;          while (rb_available_to_read(context->outputRingBuffer) < outputBufferSize) {              rb_read_from_buffer(context->inputRingBuffer, copy, context->inputBlockSize);              [PdBase processFloatWithInputBuffer:(float *)copy outputBuffer:(float *)copy ticks:1];              rb_write_to_buffer(context->outputRingBuffer, 1, copy, context->outputBlockSize);          }                    // output ring buffer -> audio unit          rb_read_from_buffer(context->outputRingBuffer, (char *)buffer, outputBufferSize);      }  }  

https://stackoverflow.com/questions/65911406/how-to-interleave-a-non-interleaved-audiobufferlist-inside-a-render-callback January 27, 2021 at 08:52AM