First and foremost: This is really cool, and thank you for sharing! (It's also the first explanation of the terms in the matrix equation for SVD that I've happened across that has really clocked for me: much appreciated.)
Here's my worry, speaking as someone who knows physics but not all that much audio processing. I feel like this approach will inevitably put a whole lot of (unhelpful!) emphasis on the detailed phase information of the various sounds. The near-perfect cancellation of the average track illustrates this: two loops of the exact same bass drum beat offset by a fraction of a second will be treated as orthogonal or even opposite by this algorithm, but they're essentially identical as perceived by the listener.
Conceptually, I imagine what you'd want is some way of encoding the various loops whose average came out sounding like a real average, rather than as nearly silent due to phase cancellation. My first instinct is to say "take the FFT of each loop first, and then run your PCA on that". Maybe that's not the right answer: like I said, I'm not an audio processing expert. But I suspect that right now your analysis is spending a huge fraction of its effort effectively trying to get the first drum beat to happen at precisely the right fraction of a second, and separately to get the second drum beat to happen at precisely the right fraction of a second, and separately the third, and the fourth, and so on. And heaven help you if the different loops' bass drums are tuned to marginally different notes.
Edit: Only after writing this did it sink in that the top overall comment's remarks (by "highd") about earlier dimensional reduction were getting at the same issue. I'll leave this here, just in case the different framing is useful.
That's my thinking too. PCA completely ignores the phase shift symmetry and treats a phase-shifted sine wave as a different thing.
One could definitely get something better if they took the audio-loops as translation-invariant. I thought about trying to do a symmetry-invariant (or 'group-action-invariant') PCA but I could not find a good way of doing that. More advanced methods, e.g. 1-d convolutional networks, WaveNet etc. do have this translation-invariance built into them.
Interestingly, the second eigenvector is not just a phase-shifted version of the first eigenvector but contains slightly higher frequency information as well, but I agree totally with what you said.
Here's my worry, speaking as someone who knows physics but not all that much audio processing. I feel like this approach will inevitably put a whole lot of (unhelpful!) emphasis on the detailed phase information of the various sounds. The near-perfect cancellation of the average track illustrates this: two loops of the exact same bass drum beat offset by a fraction of a second will be treated as orthogonal or even opposite by this algorithm, but they're essentially identical as perceived by the listener.
Conceptually, I imagine what you'd want is some way of encoding the various loops whose average came out sounding like a real average, rather than as nearly silent due to phase cancellation. My first instinct is to say "take the FFT of each loop first, and then run your PCA on that". Maybe that's not the right answer: like I said, I'm not an audio processing expert. But I suspect that right now your analysis is spending a huge fraction of its effort effectively trying to get the first drum beat to happen at precisely the right fraction of a second, and separately to get the second drum beat to happen at precisely the right fraction of a second, and separately the third, and the fourth, and so on. And heaven help you if the different loops' bass drums are tuned to marginally different notes.
Edit: Only after writing this did it sink in that the top overall comment's remarks (by "highd") about earlier dimensional reduction were getting at the same issue. I'll leave this here, just in case the different framing is useful.