Recently I wanted to play around with some audio stuff and looking for some audio library I have found CSCore. As I am newbie in coding sound at that abstraction level (quite low to me) I am struggling some issues with playback.
What I aim to achieve is to be able to play the same in memory sound buffer multiple times at the same time (in parallel). How is it intended to be done in CSCore?
At first I was struggling to play anything in parallel (two different sounds). What I finally understood reading CSCore samples and its source at GitHub is that in order to play anything in parallel using single SoundOut I need to create SampleSource or WaveSource that performs sound mixing. So with help of mixer sample that is provided in GitHub I have implemented my own simple sound mixer (that is ISampleSource mixing several ISampleSource-s). Great! It is playing two sounds at the same time from two different sources.
But here is where I got stuck. Loading single sound from disk into memory I want to be able to play it several times, possibly overlapping (parallel playback). Like I have 60 seconds sound that I start playing and after 5 seconds I want to start it another time that those are overlapping in what you can hear.
When I just added the same ISampleSource (my sound I want to play) two times into mixer list then I get strange output (sound is playing faster with glitches). I assume that it is due to reading the same stream (and moving its position forward) once by the first ISampleSource reference in mixer list and once by another one. To sum up, I cannot use the same ISampleSource to play it straight away in parallel as it is stateful.
Ok, so my second approach was to use Position to keep track of when a particular mixer list entry was when it was requested to Read(). I mean, when I add the same ISampleSource twice to mixer list it keeps additional data about Position for each list entry. So having the same ISampleSource given two times to mixer when it is asked to Read it goes through the list of sources to mix and it first sets its position to where it finished last time for this list entry and then it reads this source. So even if I use the same source twice its position is handled separately for each registration in mixer list. Great, I actually got what I expected - the same sound is playing several times in the same time (in parallel) but it is not crystal clear - I got some crackling in output like I would have physical cable defect. The problem is present even for a single sound being mixed and played. I spotted that when I comment out the line that is setting ISampleSource.Position then problem is gone. So I suppose that seeking the stream is a problem. Unfortunately I am not sure why exactly it is a problem. I saw in implementation of SampleSource/WaveSource that it's Position implementation aligns backing stream position with block-size alignment. Maybe that is the reason but I don't know for sure.
My next idea is to implement "SharedMemoryStream" that is something like MemoryStream but to be able to have the same actual memory buffer for several instances of such a stream. Then it would be tracking its position by itself (to avoid seeking that I have problem with) while having only one in-memory representation of loaded sound. I am not sure if it will work as expected (going to try it out soon) but some of my experiments showed me that it gonna be not so memory and CPU efficient - when using WaveFileReader it creates internally some WaveFileChunks that consume quite an amount of memory and time to be constructed. So I still see some other problems when using "SharedMemoryStream" to have separate stream with single buffer as I would need a WaveFileReader per stream and such a WaveFileReader would be needed per Play request (to enqueue ISampleSource in mixer).
Am I doing here something clearly wrong that only a total newbie would do?
PS. Sorry for long elaboration about my approaches and experiments. Just wanted to be clear where I am with my understanding of CSCore and audio processing in general. I am ready to remove unnecessary parts in my question/description.
Update: Added minimal code sample that let me reproduce the problem.
Here is a minimal example of my code that I have issue with.
// Program.cs
using CSCore;
using CSCore.Codecs;
using CSCore.SoundOut;
using CSCore.Streams;
namespace AudioProblem
{
internal static class Program
{
private static void Main(string[] args)
{
var soundOut = new WasapiOut();
var soundMixer = new SoundMixer();
var sound = LoadSound("Heroic Demise (New).mp3");
soundOut.Initialize(soundMixer.ToWaveSource());
soundOut.Play();
soundMixer.AddSound(sound);
// Use the same sample source to have the same sound in play after 5 seconds.
// So two sounds are playing at the same time but are phase shifted by 5 seconds.
//Thread.Sleep(TimeSpan.FromSeconds(5));
//soundMixer.AddSound(sound);
}
private static ISampleSource LoadSound(string filePath)
{
var waveFileReader = CodecFactory.Instance.GetCodec(filePath);
return new CachedSoundSource(waveFileReader).ToSampleSource();
}
}
}
// SoundMixer.cs
using System;
using System.Collections.Generic;
using CSCore;
namespace AudioProblem
{
internal class SoundMixer : ISampleSource
{
private readonly object _lock = new object();
private readonly List<SoundSource> _soundSources = new List<SoundSource>();
private float[] _internalBuffer;
public SoundMixer()
{
var sampleRate = 44100;
var bits = 32;
var channels = 2;
var audioEncoding = AudioEncoding.IeeeFloat;
WaveFormat = new WaveFormat(sampleRate, bits, channels, audioEncoding);
}
public int Read(float[] buffer, int offset, int count)
{
var numberOfSamplesStoredInBuffer = 0;
if (count > 0 && _soundSources.Count > 0)
lock (_lock)
{
Array.Clear(buffer, offset, count);
_internalBuffer = _internalBuffer.CheckBuffer(count);
for (var i = _soundSources.Count - 1; i >= 0; i--)
{
var soundSource = _soundSources[i];
// Here is the magic. Look at Read implementation.
soundSource.Read(_internalBuffer, count);
for (int j = offset, k = 0; k < soundSource.SamplesRead; j++, k++)
{
buffer[j] += _internalBuffer[k];
}
if (soundSource.SamplesRead > numberOfSamplesStoredInBuffer)
numberOfSamplesStoredInBuffer = soundSource.SamplesRead;
if (soundSource.SamplesRead == 0) _soundSources.Remove(soundSource);
}
}
return count;
}
public void Dispose()
{
throw new NotImplementedException();
}
public bool CanSeek => false;
public WaveFormat WaveFormat { get; }
public long Position
{
get => 0;
set => throw new NotSupportedException($"{nameof(SoundMixer)} does not support setting the {nameof(Position)}.");
}
public long Length => 0;
public void AddSound(ISampleSource sound)
{
lock (_lock)
{
_soundSources.Add(new SoundSource(sound));
}
}
private class SoundSource
{
private readonly ISampleSource _sound;
private long _position;
public SoundSource(ISampleSource sound)
{
_sound = sound;
}
public int SamplesRead { get; private set; }
public void Read(float[] buffer, int count)
{
// Set last remembered position (initially 0).
// If this line is commented out, sound in my headphones is clear. But with this line it is crackling.
// If this line is commented out, if two SoundSource use the same ISampleSource output is buggy,
// but if line is present those are playing correctly but with crackling.
_sound.Position = _position;
// Read count of new samples.
SamplesRead = _sound.Read(buffer, 0, count);
// Remember position to be able to continue from where this SoundSource has finished last time.
_position = _sound.Position;
}
}
}
}
Update 2: I have found solution that works for me - see below for details.
It looks like my initial idea about the problem is quite correct, but not 100% confirmed. I have introduced a counter that is executed in Read implementation of SoundSource to count how many samples was read in total to play the whole sound file. And what I got was different values for the case when I just played a stream directly and another case when I saved and restored position in each Read call. For the latter one I counted more samples than actual sound file contained therefore I assume some crackling appeared due to this over-carried samples. I suppose that position on ISampleSource level suffers from this issue as it aligns position to its internal block size, thus this property seems not sufficient to stop and continue at that level of precision.
So I have tried this idea with "SharedMemoryStream" to see if managing saving and restoration of position at a lower level would work. And it seems to do pretty well. Also my initial fear of heavy WaveSource/SampleSource creation using this approach seems not to be actually a problem - I have done some simple tests and it had very low CPU and Memory overhead.
Below is my code implementing this approach, if something is not clear or could be done better, or from the beginning it should be done in other way, please, let me know.
// Program.cs
using System;
using System.IO;
using System.Threading;
using CSCore;
using CSCore.Codecs.WAV;
using CSCore.SoundOut;
namespace AudioProblem
{
internal static class Program
{
private static void Main(string[] args)
{
var soundOut = new WasapiOut();
var soundMixer = new SoundMixer();
var sound = LoadSound("Heroic Demise (New).wav");
soundOut.Initialize(soundMixer.ToWaveSource());
soundOut.Play();
// Play first from shallow copy of shared stream
soundMixer.AddSound(new WaveFileReader(sound.MakeShared()).ToSampleSource());
Thread.Sleep(TimeSpan.FromSeconds(5));
// Play second from another shallow copy of shared stream
soundMixer.AddSound(new WaveFileReader(sound.MakeShared()).ToSampleSource());
Thread.Sleep(TimeSpan.FromSeconds(5));
soundOut.Stop();
}
private static SharedMemoryStream LoadSound(string filePath)
{
return new SharedMemoryStream(File.ReadAllBytes(filePath));
}
}
}
// SoundMixer.cs
using System;
using System.Collections.Generic;
using CSCore;
namespace AudioProblem
{
internal class SoundMixer : ISampleSource
{
private readonly List<SoundSource> _soundSources = new List<SoundSource>();
private readonly object _soundSourcesLock = new object();
private bool _disposed;
private float[] _internalBuffer;
public SoundMixer()
{
var sampleRate = 44100;
var bits = 32;
var channels = 2;
var audioEncoding = AudioEncoding.IeeeFloat;
WaveFormat = new WaveFormat(sampleRate, bits, channels, audioEncoding);
}
public int Read(float[] buffer, int offset, int count)
{
var numberOfSamplesStoredInBuffer = 0;
Array.Clear(buffer, offset, count);
lock (_soundSourcesLock)
{
CheckIfDisposed();
if (count > 0 && _soundSources.Count > 0)
{
_internalBuffer = _internalBuffer.CheckBuffer(count);
for (var i = _soundSources.Count - 1; i >= 0; i--)
{
var soundSource = _soundSources[i];
soundSource.Read(_internalBuffer, count);
for (int j = offset, k = 0; k < soundSource.SamplesRead; j++, k++)
{
buffer[j] += _internalBuffer[k];
}
if (soundSource.SamplesRead > numberOfSamplesStoredInBuffer)
numberOfSamplesStoredInBuffer = soundSource.SamplesRead;
if (soundSource.SamplesRead == 0)
{
_soundSources.Remove(soundSource);
soundSource.Dispose();
}
}
// TODO Normalize!
}
}
return count;
}
public void Dispose()
{
lock (_soundSourcesLock)
{
_disposed = true;
foreach (var soundSource in _soundSources)
{
soundSource.Dispose();
}
_soundSources.Clear();
}
}
public bool CanSeek => !_disposed;
public WaveFormat WaveFormat { get; }
public long Position
{
get
{
CheckIfDisposed();
return 0;
}
set => throw new NotSupportedException($"{nameof(SoundMixer)} does not support seeking.");
}
public long Length
{
get
{
CheckIfDisposed();
return 0;
}
}
public void AddSound(ISampleSource sound)
{
lock (_soundSourcesLock)
{
CheckIfDisposed();
// TODO Check wave format compatibility?
_soundSources.Add(new SoundSource(sound));
}
}
private void CheckIfDisposed()
{
if (_disposed) throw new ObjectDisposedException(nameof(SoundMixer));
}
private class SoundSource : IDisposable
{
private readonly ISampleSource _sound;
public SoundSource(ISampleSource sound)
{
_sound = sound;
}
public int SamplesRead { get; private set; }
public void Dispose()
{
_sound.Dispose();
}
public void Read(float[] buffer, int count)
{
SamplesRead = _sound.Read(buffer, 0, count);
}
}
}
}
// SharedMemoryStream.cs
using System;
using System.IO;
namespace AudioProblem
{
internal sealed class SharedMemoryStream : Stream
{
private readonly object _lock;
private readonly RefCounter _refCounter;
private readonly MemoryStream _sourceMemoryStream;
private bool _disposed;
private long _position;
public SharedMemoryStream(byte[] buffer) : this(new object(), new RefCounter(), new MemoryStream(buffer))
{
}
private SharedMemoryStream(object @lock, RefCounter refCounter, MemoryStream sourceMemoryStream)
{
_lock = @lock;
lock (_lock)
{
_refCounter = refCounter;
_sourceMemoryStream = sourceMemoryStream;
_refCounter.Count++;
}
}
public override bool CanRead
{
get
{
lock (_lock)
{
return !_disposed;
}
}
}
public override bool CanSeek
{
get
{
lock (_lock)
{
return !_disposed;
}
}
}
public override bool CanWrite => false;
public override long Length
{
get
{
lock (_lock)
{
CheckIfDisposed();
return _sourceMemoryStream.Length;
}
}
}
public override long Position
{
get
{
lock (_lock)
{
CheckIfDisposed();
return _position;
}
}
set
{
lock (_lock)
{
CheckIfDisposed();
_position = value;
}
}
}
// Creates another shallow copy of stream that uses the same underlying MemoryStream
public SharedMemoryStream MakeShared()
{
lock (_lock)
{
CheckIfDisposed();
return new SharedMemoryStream(_lock, _refCounter, _sourceMemoryStream);
}
}
public override void Flush()
{
}
public override long Seek(long offset, SeekOrigin origin)
{
lock (_lock)
{
CheckIfDisposed();
_sourceMemoryStream.Position = Position;
var seek = _sourceMemoryStream.Seek(offset, origin);
Position = _sourceMemoryStream.Position;
return seek;
}
}
public override void SetLength(long value)
{
throw new NotSupportedException($"{nameof(SharedMemoryStream)} is read only stream.");
}
// Uses position that is unique for each copy of shared stream
// to read underlying MemoryStream that is common for all shared copies
public override int Read(byte[] buffer, int offset, int count)
{
lock (_lock)
{
CheckIfDisposed();
_sourceMemoryStream.Position = Position;
var read = _sourceMemoryStream.Read(buffer, offset, count);
Position = _sourceMemoryStream.Position;
return read;
}
}
public override void Write(byte[] buffer, int offset, int count)
{
throw new NotSupportedException($"{nameof(SharedMemoryStream)} is read only stream.");
}
// Reference counting to dispose underlying MemoryStream when all shared copies are disposed
protected override void Dispose(bool disposing)
{
lock (_lock)
{
if (disposing)
{
_disposed = true;
_refCounter.Count--;
if (_refCounter.Count == 0) _sourceMemoryStream?.Dispose();
}
base.Dispose(disposing);
}
}
private void CheckIfDisposed()
{
if (_disposed) throw new ObjectDisposedException(nameof(SharedMemoryStream));
}
private class RefCounter
{
public int Count;
}
}
}