Realtime sound generation in Unity3D

Recently I have been digging into procedural generation of game resources for Unity games. And one of the aspects that isn't really covered anywhere is sound generation. Probably because most modern games don't need it, but it still seemed like fun. I've wanted to make a realtime sound oscillator in unity for quite a while and especially after seeing this blog entry about music generated with very short algorithms.

In this tutorial, however, we will be using a slightly different approach, since Unity's built in audio streams consist of float arrays. But most of simple formulas described in the article above still work.

To compose your own symphonies, you simply need to create a new AudioClip with AudioClip.Create(). But since we're generating the sound continuously, we also need to provide a PCMReaderCallback delegate.

Here's how it looks:

 public int buffer = 400;  
 public int channels = 2;  
 public int frequency = 16000;  
   
   
 AudioSource source;  
 AudioClip.PCMReaderCallback callback;  
   
 void Start() {  
 source = GetComponent<AudioSource>();  
   
 callback = delegate(float[] data) {  
    for (int i = 0; i < data.Length; i++) {  
       data[i] = Mathf.Sin(i/50);  
    }  
   };  
   
   AudioClip clip = AudioClip.Create("clip", buffer, channels, frequency, true, callback);  
   source.clip = clip;  
   source.Play();  
 }  
   

This will play a simple sinewave.

To make it work, simply put it in a MonoBehaviour class and add to any object with an AudioSource. I've made parameters of creation public so that you can easily adjust them from inspector.

Frequency defines a sample rate of our stream. Most audio files are saved at 44.1KHz sample rate, meaning that every second of your audio contains 44100 separate points also known as "samples".
Number of channels in audio clip can either be 1(mono) or 2(stereo). Setting it to any other values will simply not work.
Buffer defunes how many samples of our audio are going to be preprocessed. I wouldn't recommend setting it too high as it does have an impact on performance. In a finite clip it simply defines the length.

Now our callback is being executed in a separate thread every time audio clip reads data. Which in this case happens about 40 times a second. Threaded behaviour is an important thing to note, as it does put some limitations on the whole generation process.

But you must've already noticed something wrong with our newly generated sinewave. It sounds rather choppy and inconsistent. This is because we're generating the same piece of a wave for each block of data and the ends don't really match. To fix this, we're going to introduce a new variable called cycles.

 int cycles;  

And in our callback, we're going to increment it every time by one and then find the sample we're at globally by multiplying it by buffer length and adding i.

Here's what the callback looks like after these changes:

 callback = delegate(float[] data) {   
   cycles++;  
   int sample;  
   for (int i = 0; i < data.Length; i++) {  
     sample = cycles*buffer+i;  
     data[i] = Mathf.Sin(sample/50);   
   }   
};   

That sounds much better, doesn't it? But plain sinewaves quickly get boring. If you really want to capture the essence of procedural generation, you're going to need some white noise. And it might look simple, but that's where threading drawbacks come into play. Since we're not calculating our data in the main thread, we can't use UnityEngine.Random class. It just doesn't allow us to.

What we can do is use System.Random to generate random numbers. But it's not thread safe either, which means that if two separate threads use it at the same time, it will just break and output 0's. And if you're thinking of creating an instance of random for each call, or even a thread, this is a bad idea too. First approach will not give you random numbers and will have a huge impact on performance. Due to the nnature of System.Random, instances that are created too close in time will likely return same numbers. And this isn't what we want, is it?

Creating an instance for each thread might do the trick, but for me the produced noise had some metallic sounding artifacts in it each now and then. Using a simple lock however seems to produce the effect we need and even create negligible overhead. Sometimes the most obvious solution is the best.

So, create a new instance of System.Random, assign it on start and then use in the callback method.
To generate a random float value from -1 to 1, which is the range of all our waves by the way, we will use Random.Next int the range of -40 to 40 and then divide it by the same value. You can use any other number and it will just affect the precision of your noise. For my needs, 40 seems enough.

 float randomValue(System.Random random) {  
       int value = random.Next(-40, 40);  
       return (float)value/40f;  
    }  

Now in our callback

  callback = delegate(float[] data) {    
   cycles++;   
   int sample;   
   for (int i = 0; i < data.Length; i++) {   
    sample = cycles*buffer+i;   
    data[i] = randomValue(rndm);    
   }    
 };   

Where rndm is prevoiusly defined instance of System.Random. Now you can click play and enjoy the relaxing sounds of randomness.

Finally, if you want to define your waveform by hand, you can use animation curves.
Create one like that

 public AnimationCurve wave;  

Evaluate your sample like that

  data[i] = wave.Evaluate((sample%40)/40);   

And adjust it in realtime in editor. Example below shows a curve ranged from -1 to 1 as I'm evaluating a sinewave there. Your wave would start at 0 and end at 1. Pretty handy. And 40 actually defines the frequency of your wave, so play around with that too if you want.

Noisy texture in the background is a little rendering utility I made, just to have some visual feedback from my sounds. I'm not going to cover it for now as the whole concept still needs some polishing.

Without the visualisation, this oscillator seems pretty efficient and might help some retro style game to create a truly chiptune-like sound. It does stutter a little when the processor load increases though and I'm not sure how to fix that yet. Feel free to write me about your ideas on how to improve this technique.

Good luck, fellow experimenters.