You can't - the contents of an
ArrayList are outside of a plugin's memory space; the only way a plugin is allowed to access it is through the provided natives, and none of them give you that kind of memory access.
ArrayList.Set lets you update specific entries in an array stored in an
ArrayList; otherwise you will need to copy data to / from your plugin (as you've already discovered).
What's your particular use case? Without further context this sounds like a preemptive attempt at optimization.