Have us design and build your next software project.
Learn more →Software
Interactive Music for the Web
The Example here
another one
another one
One of our clients wanted to build a player for iOS to do custom playback for Wwise.
Here the design.
Goals:
- This
- This
- This
The project has some interesting challenges. We are going to impose that playback can start before all segments have been downloaded to get the most responsive start.
The playlist
A `playlist` is a list of segments and/or other playlists, with a type that specifies how to play the list.
There are four types of playlists:
1. Loop
2. Random
3. Shuffle
4. Sequential
{
"playlists": {
"ToySeq1": {
"type": "play",
"loop": 0,
"list": [
["toy_story_intro", 1],
{
"type": "play",
"loop": 0,
"list": [
["toy_story_DRUMS", 1],
["toy_story_loopNODRUMS", 1],
["toy_story_loopNOPIANODRUMS", 1],
["toy_story_loopNOPIANO", 1],
{
"type": "random",
"loop": 1,
"list": [
["toy_story_loopNOPIANODRUMS", 1],
["toy_story_loopNOPIANO", 1],
["toy_story_DRUMS", 1],
["toy_story_loopNODRUMS", 1]
]
}
]
}
]
}
}
}
Playlist machine
Let's build a state machine that will walk the playlist and return the next segment to play.
When the playlist is done, it will return null.
Let's start with a simple sequential playlist that optionally loops n times.
We are going to use the short hand notiation of `[segmentName, loopCount]` to represent a looping playlist.
function*s(m,S){let k=0;for(;k<m.loop||m.loop===0;){for(const _ of m.list)yield*y(_,ot({},S,{tempo:m.tempo}));k++}}
Rerun
intro
theme_drums
theme_nodrums_2x
theme_nodrums_2x
random_3
random_3
random_3
shuffle_1
shuffle_2
shuffle_2
shuffle_3
shuffle_3
shuffle_3
theme_drums
theme_nodrums_2x
...
How about a terminating version?
Rerun
intro
theme_drums
theme_nodrums_2x
theme_nodrums_2x
random_2
random_2
shuffle_3
shuffle_3
shuffle_3
shuffle_2
shuffle_2
shuffle_1
How about inherited properties? Let's refactor to merge properties as we go.
Rerun
intro {"tempo":[115,4,4]}
theme_drums {"tempo":[100,4,4]}
theme_nodrums_2x {"tempo":[100,4,4]}
theme_nodrums_2x {"tempo":[100,4,4]}
random_1 {"tempo":[220,4,4]}
shuffle_1 {"tempo":[80,4,4]}
shuffle_2 {"tempo":[80,4,4]}
shuffle_2 {"tempo":[80,4,4]}
shuffle_3 {"tempo":[80,4,4]}
shuffle_3 {"tempo":[80,4,4]}
shuffle_3 {"tempo":[80,4,4]}
Can I write this without generators?
Learn more →Ok we can walk playlists, halt and capture inherited properties as the list runs.
The timeline / the scheduler
A timeline at this point is essentially an independent scheduler tied to a playlist.
When the timeline is signalled to play, it will need to stitch together audio segments to create a continuous, on-time score.
Let's look more at the anatomy of a segment.
Segment
{
toy_story_loopNOPIANO: {
entrycue: 0.260869565217392,
exitcue: 31.5652173913043,
tracks: [
{
path: "ToyStory/toy_story_loopNOPIANO.ogg",
trimstart: 0.260869565217391,
trimend: 32.2173913043478,
playat: 0
}
]
}
}
A segment is a collection of tracks that are played together.
Each track has a path, a trim start and end, and a play at time.
The trim start and end are used to trim the audio file.
The play at time is used to indicate where the start of the track is played.**
The entry and exit indicate regions of possible overlap for the segment, these regions are called preroll and postroll.
However, first we need the track duration to be able to schedule the segments. We'll need to load the audio and fetch the durations. Lets do that now.
async function hr(e,n){return new Promise((t,r)=>e.decodeAudioData(n,a=>t(a),r))}
async function ut(e,n){return fetch(n).then(t=>t.arrayBuffer()).then(t=>hr(e,t))}
function g(m,S){return lt(Object.fromEntries(Object.entries(S).map(([k,_])=>[k,lt(_.tracks.map(F=>ut(m,"./interactivemusic/"+F.path))).pipe(Rt(F=>({..._,tracks:_.tracks.map((P,C)=>({...P,buffer:F[C]}))})))])))}
** Or have a way like code genius to hover and get more information about the code construction
What is this forkJoin?
Learn more →{
1Animals_PreRoll: {
tempo: [
128,
4,
4
],
entrycue: 1.875,
exitcue: 5.625,
tracks: [
{
path: "ExampleScene/1Animals_PreRoll.ogg",
buffer: "<AudioBuffer>"
}
]
}
}
Ok, now we can get the duration of each track by accessing `buffer.duration`, or the duration of the entire segment with `duration`.
You can see the duration doesn't match our exitcue time exactly, but we'll delay investigating this for now.
function v(m,S){return lt(Object.fromEntries(Object.entries(S).map(([k,_])=>[k,lt(_.tracks.map(F=>ut(m,"./interactivemusic/"+F.path))).pipe(Rt(F=>({..._,tracks:_.tracks.map((P,C)=>({...P,buffer:F[C]})),duration:F.reduce((P,C,D)=>{const R=_.tracks[D],V=R.trimend||C.duration,Z=R.trimstart||0;return Math.max(P,V-Z)},0)})))])))}
{
1Animals_PreRoll: {
tempo: [
128,
4,
4
],
entrycue: 1.875,
exitcue: 5.625,
tracks: [
{
path: "ExampleScene/1Animals_PreRoll.ogg",
buffer: "<AudioBuffer>"
}
],
duration: 5.629387755102041
}
}
** Here are a few example segments with varing trims. **
Ok we have the segment durations to begin assembling a time perfect score. Let's write version one of the sticher.
function*I(m,S,k){for(const[_,F]of S){const P=k[_],C=m-P.entrycue;m+=P.exitcue-P.entrycue,yield{queTime:m,segment:{segmentName:_,tempo:ot({},F.tempo,P.tempo),segment:P,startTime:C}}}}
Here are the results, mapped to a smaller set for readability.
stichCode(0, playlist, loadedSegments, 5);
[
{
segmentName: "1Animals",
startTime: -1.875,
duration: 5.629387755102041,
entrycue: 1.875
},
{
segmentName: "1Animals",
startTime: 1.875,
duration: 5.629387755102041,
entrycue: 1.875
},
{
segmentName: "2Animals",
startTime: 7.5,
duration: 3.7543764172335603,
entrycue: 0
},
{
segmentName: "2Animals",
startTime: 11.25,
duration: 3.7543764172335603,
entrycue: 0
},
{
segmentName: "2Animals_Loop2",
startTime: 15,
duration: 3.7500226757369615,
entrycue: 0
}
]
> In reality you would
validate the schema and ensure all the data checks out before publishing to web. Don't validate files on the client.
So the startTime for the first element is less than 0. We can't schedule audio for the past, so we'll need to move the time up.
Let's put a pin in this problem for now. (We could possibly send a flag into the sticher, but I try to avoid any special casing as much as possible.)
Here is the result rendered out.
It might cross your mind that the preroll for a second segment may be less than the start time for the prior segment. Here we will assume that this case is not possible.
Great- we have some data to schedule. Let's get some audio setup and get a scheduler going.
Web audio
Let's take our score from above, wire it up to an audio graph and play it.
Lets get a function going that will wire a segment to the audio graph.
function fe(){function m(k,_,F){F.tracks.map(P=>{var Z,Se;const C=k.createBufferSource();C.buffer=P.buffer,C.connect(k.destination);const D=(Z=P.trimstart)!=null?Z:0,V=((Se=P.trimend)!=null?Se:C.buffer.duration)-D;C.start(_,D,V)})}const S=y(kt.music.playlists.ANIMALSFULL);v(new AudioContext,Me).subscribe(k=>{const _=new AudioContext,F=I(0,S,k),{segment:P}=F.next().value,C=-P.startTime;m(_,_.currentTime,P.segment);let D=0;for(const{segment:R}of F)if(m(_,_.currentTime+C+R.startTime,R.segment),++D>5)break})}
Thats dope. This is how experential technology comes to life.
The scheduler
So you can pretty much see the scheduler in your mind. Keep pulling segments out of the score and connect them to the audio graph at the right moments.
function me(){const m=Math.max(...Object.values(Me).map(_=>_.entrycue||0)),S=1;function k(_,F){const P=d.currentTime;for(;;){const{value:C,done:D}=_.next();if(D)break;const{segment:R}=C,V=P+F+R.startTime;if(Et(d,V,R.segment),V-P>=m+S)break}setTimeout(()=>k(_,F))}k(score,offset)}