Without benchmarking it, this is probably the biggest bottleneck of the design (with a large degree of fragmentation, iterating this list will become very slow, incurring a cache miss upon each traversal).
You could fix this by storing metadata outside of the allocated areas and using a doubly linked list to achieve O(1) removal.