SortCustom2D is using QuickSort algorithm. I dont know what version does it use exactly but some of them can run your system out of stack space even if you have just several thousands elements in your array, as it is massively recursive sort. To prevent this error there are implementations of QuickSort that use iteration with stack instead of recursion, but you would have to write it yourself or ask someone to make it for you.