Algorithmic Performance: In your sequencer? Please Santa ... Demos Inside.

[YBaCuO]

Algorithmic performance: Does your sequencer do this? It should. I wish mine did. Demos below.

WARNING: most readers will want to just skip to the demos at the bottom, and if you like what you hear, petition the powers-at-be that you want it (my daughters just throw a tantrum, it works sometimes ...).

Here are some "quick-and-dirty" demos that feature two relatively simple algorithmic techniques to improve midi performances. Hopefully, these crude demos will encourage software vendors to implement similar ideas. I am practically shouting for them to do something. It must not be difficult to do, because I was able to rather easily, albeit messily, program them in the free KeyKit language (http://nosuch.com/keykit/).

These two ideas, which I'll call algorithmic tempo maps and "v-swell w/peak shift" expression/modulation control (v-swell for short), were featured in the old SuperConductor program headed by Dr. Manfred Clynes. I do not want to discuss this program but "more" info can be found here http://www.microsoundmusic.com/home.htm and purchased through here www.seermusic.com.

The v-swell algorithm is quite simple: hill-like expression/modulation midi data is applied to each note (except short/fast notes, which usually do not need it). Then the peak of this hill is shifted depending on the next note (in a continuous phrase). If the pitch of the next note is higher, then the peak of this hill is shifted towards the next note (to the right) with a greater peak shift for greater interval distances. Similarly, if the pitch of the next note is lower, then the peak of this hill is shifted away from the next note (to the left). In general, it is nice to have controls to create different hill shapes and/or peak shifts for different occasions or different note characteristics (duration mainly, velocity, host tempo, bar location, ends of phrases, note gap sizes, etc.).

However in these crude examples, simply ONE hill shape w/peak shift was applied to all notes for all instruments! Of course, slightly better results could be achieved if more shapes were used.

The algorithmic tempo maps are more complex and really beyond the scope of this post. One concept is the pulse, which is a sort of combination meta and micro rhythm structure and which Dr. Clynes refers to as a hierarchical Pulse. Different positions in the bar or among bars, get different tempos and amplitudes (faster and louder, slower and quieter, etc.). Dr. Clynes' research showed that performances of music by the masters could be approximated with this system. Each composer seemed to favor different kinds of pulses. Mozart preferred a strong-weak-strong-weak map, Beethoven strong-weak-strong-strong, Franck weak-strong-strong-weak, Schumann weak-strong-weak-strong and most of the others a variation of strong-weakest-less weak-strong. Programming such a pulse is straight-forward by creating a tempo map with user-selectable duration/amplitude sliders. However, applying a tempo map that makes musical sense is extremely difficult - part of the "mystery of the arts."

Most of the demos below use tempo maps that were created by Dr. Clynes. But notice that two demos used tempo maps created by myself. For Mozart's wind partita K.361, I simply used the standard Mozart pulse. For the original flute and tuba composition by TriestaOrange, I used a pulse similar to Franck (weak-strong-strong-weak), which suits so perfectly that the listener is not even aware that it exists.

Beyond the pulse described above, the SuperConductor program has some other nice features like crescendos/diminuendos and accelerandos/ritardandos that follow curves. And there were easy-to-use tools to manipulate these curves. The effect was intuitive and beautiful. And I ask, may we please have similar tools in our notation/sequencer programs?? Pretty please with lots of money on top ...

There are lots of small little tiny details that were necessary to interface all this different technologies together. And I also wish sequencer/notation programs would implement similar time saving features. In the demos, continuous lines of notes without a durational pause were automatically considered "slurred" and the appropriate pedal controller midi data were created (whether it was appropriate or not! The data was deleted only in the Bach demo). The program also detected short durational notes for the string sections and generated the appropriate keyswitch notes for the string sections (which seemed to only manifest itself in the Beethoven symphony demo). Some attempt was made to lessen the machine gun effect for repeated notes: some random note on/off variations, velocity, var1 and var2 midi data were automatically generated. Honestly, this last part did not seem to help much. But again, I want my sequencer/notation program to do this for me. And there were variables to adjust note lengths (so notes weren't end to end) and generate length parameter midi data. Sigh, why can't this be done automatically too (actually most sequencers can be set up to do this).

This algorithmic performance is straightforward to use, but tedious. My simple KeyKit program is text based (can you say batch file?) and although the SuperConductor program has a GUI, where-you-are-in-the-score is still text based. That makes all difficult to use.

DEMOS HERE:

The following excerpts are all algorithmic-based SuperConductor performances
by Dr. Manfred Clynes (except where noted), converted using a homemade
KeyKit program, mixed in the Muzys sequencer using the free 350seat or
1800seat impulses from www.noisevault.com and SIR. All instruments are from
Garritan Personal Orchestra (GPO).

DISCLAIMER: These are NOT intended to be polished demos. Instead, the intent is to demonstrate two simple algorithms for enhancing computer performance: algorithmic tempo maps and "v-swell w/peak shift" expression/modulation control.

Flute and Tuba: original composition by TriestaOrange, KeyKit performance
http://www.wusik.com/song.php?id=270

Cello and Piano: Brahms Cello Sonata
http://www.wusik.com/song.php?id=361

Violin and Piano: Franck Violin Sonata
http://www.wusik.com/song.php?id=362

Piano, Violin, Cello: Schubert Piano Trio
http://www.wusik.com/song.php?id=363

String Quartet: Haydn Quartet from Op.76
http://www.wusik.com/song.php?id=364

Wind Ensemble: Mozart Grand Partita K.361, KeyKit performance
http://www.wusik.com/song.php?id=365

Ensemble: Bach Brandenburg Concerto #1, mov. 1
http://www.wusik.com/song.php?id=375

Violin Concerto: Mozart
http://www.wusik.com/song.php?id=372

Violin Concerto: Mendelsohnn
http://www.wusik.com/song.php?id=366

Orchestra (slow): Schubert's Unfinished Symphony No.8, mov.1
http://www.wusik.com/song.php?id=367

Orchestra (fast): Beethoven Symphony No.3, mov.1
http://www.wusik.com/song.php?id=368

Enough performing for me now, back to my research

YBaCuO

[galvedro]

Wow! Your examples are quite impressive. They don't have any other time/expression processing than that from your algorithms, do they?

I observed by listening to the demos that you have programmed some kind of "random note on placement" in a per track basis that produces a very interesting desync between instruments, apart from the tempo changes. This makes very realistic some fragments of the music. This is clearly shown in the Brandemburg demo.

I think this performance techniques would be very useful, specially if they are smart enough to apply different algorithms depending on the style. For example, the program could have a parameter matrix, indexed by instrument and style, let's say for example baroque and romantic. Then, the user could select the baroque style, and each instrument would be played accordingly (strings would be played detache, etc...).

Anton

[YBaCuO]

galvedro wrote: "Wow! Your examples are quite impressive. They don't have any other time/expression processing than that from your algorithms, do they?"

Your ears are much better than mine! There are all kinds of little hidden details. There is code to do some random things, but for some reason the random note on start is not working, so instead I simply slid a whole track, so the viola, or 2nd instruments are always late compared to the violin or 1st part. There is some random velocity and also a higher in pitch is higher in velocity and volume (modulation). One difficult part was scaling the modulation dynamic range (how low or high in volume an instrument goes). You can hear often that an instrument gets too loud. I did not go back and fix anything, except for really obvious too loud parts.

Believe me all this is extremely easy to implement, I did it for fun and as all the parameters were changing as I played with the program, I do not know what settings were actually used for any given piece. I don't really care. I am not very interested in pursuing this project. The demos are not polished or really serious attempts at all. I am only hoping to attract someone who is interested in doing more.

I should point out that there is still a lot of time involved in each demo, even after all the tracks are created. The tracks have to be loaded into a sequencer, GPO instruments loaded and assigned, reverb added, and then the volume of each track has to be adjusted. This all takes time, probably an hours worth of time for a symphonic work.


galvedro wrote: "I think this performance techniques would be very useful, specially if they are smart enough to apply different algorithms depending on the style."

Yes! That is the spirit. There is a lot more that can be done. What I personally want to see developed is some sort of computer-aided performance system. So for example, if I wanted to make a violin slide form one note to another while I am performing, I just hit a knob or button and all the appropriate midi data is generated to perform the slide as I continue to play ...

While automatic computer performance is also interesting, I prefer to be able to "perform" myself, but managing 8 different knobs, pitch wheels, mod wheels while playing some keyboard all at the same time doesn't seem very fun to me.

Thanks for your time,
YBaCuO

[galvedro]

Danielle, I agree with you. At least to some degree.

But we all know how demanding is to do a realistic mockup. Some people like me, do the work by "layers": i.e. I write a scratch of what I want, and then I start to polish it. Wouldn't be great to directly write on a notation software like Overture, then press the play button and hear it one step forward automatically? Of course it would be needed to play every part individually afterwards to reach the highest level, but at least, you'd have a very good starting point to work with.

Anton

[YBaCuO]

Ah, Danielle, of course if someone has the time or ability to perform an entire symphony then algorithmic performance is not interesting but I do not have such time or ability, so can I pursue what interests me and maybe others in peace ... ???

But there are ideas here that anyone can use.

Take for example, the v-swell algorithm. Want to improve melodic or lyrical lines (especially string), then draw in or perform this "trick"? To understand it, I think of singing and feeling my muscle tension. Now I am singing "Music of the Night" from Phantom of the Opera. When the pitch goes down, I have to first relax my muscles and then retension them again. This takes time and the effect is almost a pause between the two notes. However, when the pitch goes up, all I need to do is tighten my muscles. I almost glide into the higher note, increasing in volume just before the higher note is sung. Gee, I just described exactly the v-swell w/pitch shift algorithm. While I have no idea why it sounds good, I would not be surprised that the vocal tension is the underlying reason. You can clearly hear this algorithm in all the slow violin parts in the demos, especially for the Mendelsohnn and the Franck. So, yes, it is a Romantic technique indeed. There is a comic effect in the Haydn quartet as the violin part, so busy doing a swell, that it is very late to the party.

Technical tip: the v-swell used was only 0-16 in midi units, which should be added to the underlying mod wheel data.

As for the rhytmic Pulse. Well, this is very deep and something to contemplate in your own compositions. I was unable to post entire movements in the demos, but I was able to hear them. And despite the poor midi performances, the always-the-same sounding samples and algorithms and the bad defects like the machine gun, the organ sound, etc, I was always just speechless after a piece was completed and I was eager to hit the play button again. The compositions were moving and I am convinced that the Pulse was the reason why. Mozart sounds like Mozart, Beethoven like Beethoven - why is that? Is it the "Pulse?" Mozart is rather unique with his symmetrical strong-weak-strong-weak pulse (think one bar at a time in 4/4 time). Mozart starts with a strong statement (usually just a repeated note or arpeggiation) in one bar, starts to move in the second bar (weak), arrives there in the 3rd bar (strong), and then flourishes with a cadential move, a kind of farewell (weak). But surprisingly, his themes throughout a whole piece continue the same pulse pattern (but not the exact example above), despite the amazing variety. The effect is hypnotic. What about other composers? the non-masters? I think of Prokovieff's Classical symphony - just a guess but no pulse there, and despite the brilliance in other areas, there is something missing in that piece.

What did you think of the performance of the WhenTheMoon... demo? There is a pulse there too.

gotta run

YBaCuO

gotta

[billp]

of course if someone has the time or ability to perform an entire symphony
Hmmm...I thought that was the point. :)

YBaCuO...are you one?

[YBaCuO]

Quote:
Originally Posted by YBaCuO
of course if someone has the time or ability to perform an entire symphony

billp wrote:
"Hmmm...I thought that was the point.

YBaCuO...are you one?"


Hee, hee - it is not me. The credits should be clearer: Dr. Clynes is responsible for almost all the performances - not only entering in all the notes but especially the interpretation - and there is a lot there: pulses defined, crescendos, ritardandos, etc, etc. The demos illustrate Dr. Clynes' musical genius more than anything.

I only wrote some conversion programs. Believe me, it was easy to do. I also did two performances, Mozart's Grand Partita for wind ensemble and WhentheMoon... But both were easy too.

Danielle and galvedro:
the performances are ALGORITHMIC but NOT AUTOMATIC.

Dr. Clynes, who did the interpretations, still had to spend many hours with the score, defining sections, fine tuning the pulses, adding crescendos/dimenuendos, accelerandos, pauses, tempo changes and so on. The algorithms make it easier and also limited and different than a live performance but the demo performances are still human ones.

The v-swell w/peak shift algorithm can be almost automatic. It improves 90% of standard midi playback and with a few more options it can get closer to 95%. It makes melodic lines sweeter and the output can be edited later.

As for the pulse: this is still a mystery for me. It works for the masters, but is it generally useful other than to provide some variation? I can only say that when I rendered two recently composed works by two different composers, well, by trial and error, I quickly found a pulse that gave excellent results. The demo WhentheMoon... for flute and tuba is one such work. So who knows? But again, the pulse is so very easy to program - why not include it as another available tool?

I want to emphasize again - these two algorithms are EASY to program. That is why I spent the time to create these demos and start this thread.

YBaCuO

[FireRaven]

not understanding this.

wish there was a video demo... i can always understand a video demo :D

[JonFairhurst]

This kind of algorithm work would be excellent in a notation program, and may be helpful in a sequencer.

I'd love to have a series of different envelopes available in a notation program. You could simply choose flat, increasing, decreasing, early, mid or late hill or early mid or late dip. If the notation program could make a best first guess, that would be even better. For the final level of tweakitute, allowing me to draw a custom envelope for any given note would be killer.

This could be handy for a sequencer as well, just ot get things started, but what I'd really like is a bezier curve model for the controllers. Heck, Corel 1.0 had bezier curves back in 1990 or so. Why does Sonar have the same damn interface for controllers in the piano roll window that Cakewalk has had for the better part of a decade? Sheesh!

Also, the interface only lets you magnify it so much, doing something smooth and subtle is out of the question. With Bezier curves, I could quickly make beautiful curving expressions very quickly. I'd still want the drawing ability too for the times that I want something fairly random. (The position control on Larry Seyer's Acoustic Drums comes to mind. Precision isn't needed there.)

Anyway, Notation programs really need something like the algorithms described. And if they can be easily tweaked to meet artistic intent, that's all the better. Sequencers could use it as a starting point, but faster, more precise manual control would be great as well.

-JF

[Markleford]

Missed this thread the first time...

Algorithmic performance: Does your sequencer do this? My sequencer (SONAR) can do some of that. As can any sequencer that supports MFX plugins (or similar standard). Tempo maps are unfortunately not able to be tweaked by MFX yet, though: you'd have to fake it by pre-buffering your entire track. That, and separate tempo maps per track in the same project, are something I'd like to see too!

Sometimes MFX is the right tool for the job over external drivers like KeyKit and MIDI-OX. Sometimes not!

An example collection of my (MFX-based) algorithmic and gestural stuff can be found here (http://www.ruccas.org/index.php?Markleford%20Friedman).

- m

[Markleford]

Why does Sonar have the same damn interface for controllers in the piano roll window that Cakewalk has had for the better part of a decade? Sheesh!Errm, SONAR *does* have envelope curves in the track view that can generate any controller data! Did you entirely miss that feature? They're not Beziers, but you really don't need that tight of control for contouring musical performance.

- m

[JonFairhurst]

Hi Markleford,

Yes I've seen (and used) the envelopes in the track view. They're cool, but somehow I'm more comfortable with working in the piano roll view. Maybe out of habit. Maybe it's because I really want to see the MIDI notes clearly. I'm not sure.

It's a step in the right direction. I have Sonar 3. Did they improve on the envelopes beyond 3 when they went to 4?

-JF

[Markleford]

It's a step in the right direction. I have Sonar 3. Did they improve on the envelopes beyond 3 when they went to 4?Same as 3. I haven't had much reason to want improvement on them, personally.

So you want envelope view in the piano roll view, then? I can't see why they couldn't make it happen, but I hadn't previously seen a need for it as I typically contour my performances from track view. Perhaps nobody has requested it yet!

- m

[JonFairhurst]

I think it depends on the countouring that you are doing. If you want something slow, or general, like a moving filter, a fade in, or fade out or overall expression, the track view is perfect. If you're trying to perfect a performance note by note, say with a solo violin, then I find that I need to start/end the envelope exactly on the MIDI start/end, and I may also need to tweak the overlap or separation between notes. In the second case, the piano roll view is the right place. For me anyway!

-JF

[Oksi Moron]

I think it depends on the countouring that you are doing. If you want something slow, or general, like a moving filter, a fade in, or fade out or overall expression, the track view is perfect. If you're trying to perfect a performance note by note, say with a solo violin, then I find that I need to start/end the envelope exactly on the MIDI start/end, and I may also need to tweak the overlap or separation between notes. In the second case, the piano roll view is the right place. For me anyway!

-JFThis debate is a little akin to the debate over breath- vs. wind-controllers. I've tried 'em both, and there are times when it seems that the little bit of extra per-note sculpting ability you get with a wind-controller is worth the expense. That may be because while few sample patches are programmed to change timbre under the control of the sampler's built-in ADSR, many are programmed to change timbre under cc control. And there are instruments, such as brass, that seem to me to really benefit from getting the maximum amount of timbral change over the life of the individual note, however long or short. It can be really hard to be a fast enough hand with a mod wheel to mimic the sort of per-note shaping I'm thinking of. Likewise, contouring similarly granular per-note effects with Sonar's Midi Shapes could be pretty tedious.

I'm kind of sorry to hear you talking against per-note contouring, Markleford, because I was just about to ask you to consider writing a variant on your control-cascade MFX plug.

Here's the idea: let's say I want an offline "virtual wind-controller" midi effect to apply to lines that I've played in with my keyboard instead of my wind controller. And let's say I'm only good enough a programmer to write a Cakewalk CAL script that applies a plausible cc "template" envelope, with a fixed peak amplitude, to each note. But I want more: I want to be able to draw in, as an overlay, a "phrasing" cc curve that spans several notes, and have it rescale my per-note envelopes accordingly.

Turns out I can almost get what I want using your controller cascade MFX. But the controller cascade MFX seems to perform an addition operation, and I think what I want would be a multiplication operation. In other words, it would be like convolving two sets of curves -- a per-note "tonguing" curve against a longer "breath" curve. ("Tongue curve" and "breath curve" are terms borrowed from Roger Dannenberg's fascinating study of trumpet envelopes (http://www-2.cs.cmu.edu/%7Erbd/papers/tptenv98/tptenv98.htm)) Doing it as a multiplication operation would assure that the spaces between the notes didn't get filled in with cc's. That way you preserve the attack- and release-sculpting benefit of having applied the per-note tongue curves in the first place.

You might well ask why I don't just think in terms of doing the tongue curves as cc2's, say, and then going in and drawing the longer breath curve as cc11, and leave it at that without trying to merge the two different cc's. I don't have a good answer for that, other than that some of my synths/samplers won't allow you to modulate one parameter (timbre is the one I care about here) with two different cc's.

Anyhow, Markeford, I love all your MFX toys, and I play with them all the time!

Greg

[Markleford]

I'm kind of sorry to hear you talking against per-note contouring, Markleford, because I was just about to ask you to consider writing a variant on your control-cascade MFX plug.Actually, I don't think I ever implied I was against it: I just meant that I tend to do per-note contouring in the track view of SONAR with envelopes, rather than the piano roll view with penciled controller curves. :)

Turns out I can almost get what I want using your controller cascade MFX. But the controller cascade MFX seems to perform an addition operation, and I think what I want would be a multiplication operation.I see what you mean: one envelope controls the countour of the line (breath curve), one acts relativistically to that on a per-note basis (tonguing curve). One macro, one micro.

That can certainly be done. I'll put it on the Ever Growing List of Things To Do Before I Die. ;)

- m

[Markleford]

For anyone still listening...

Finally had time to make this happen, and added a "relative" mode to TenCrazy.com MFX ControlCascade.

Check out the new thread here. (http://www.northernsounds.com/forum/showthread.php?t=31864)

- m

[Fabio]

I had the chance to try the human playback function of many notation programs, that is based on the same concept. It's year after year better.

I fully agree with the aim of a "first reading" humanization, that doesn't have the scope of a final performance (nor artist substitution), but can really make enjoyable the authomatic playback of a score just typed in or imported from step by step flat MIDI files.

The second level of action, is IMHO that to refine note by note a whole symphony would probably ask a lot of time, but if the first rough edging of a good algorithm has smoothed the piece, probably to finish it will be a matter of details and finesse adjustements, and "inpredictable" effect manual input where it's necessary. It should be faster, certainly.

Lots of professionals are pressed by timing, and a way to speed up the work is absolutely desirable.

Lots of arrangers are not excellent piano players (or are not keyboardist at all), and it's the last reason to speed up the work (mainly for very complex and fast passages).

Then I think that every true musician like Danielle is a little feared in front of the technology, because of the risk of cultural flattening, and the will of preserving the human source of the art, coming directly from our soul and our spirit.

But every technical instrument is only a tool in the hand of the artist: he will use it in the right way, being a musician.