We have found pointer files to be surprisingly efficient as long as you don't have to actually materialize those files. (Git's internals actually very well done). Our mount mechanism does avoid materializing pointer files which makes it pretty fast even for repos with very large number of files.
For bigger annex repos with lots of pointer files, I just disable the git-annex smudge filters. Consider whether smudge filters are requirement, or a convenience. The smudge filter interface does not scale that well at all.
