[Core] Actor/Task Memory resource allocation should be separate from Object Store allocation

[alexeykudinkin]

What happened + What you expected to happen

Currently, when scheduling Actors or Tasks with memory resource specified i'm expecting that they will be allocated based on memory available on the node for use as private process heap.

This is problematic in the following scenario in Ray Data:

1. I'm configuring region of node memory to be dedicated for the Object Store (say 35%)
2. Remainder of the memory i want to be split b/w Tasks/Actors based on their corresponding memory allocation.

Instead i observed that we schedule based on the total available memory that also includes the Object Store.

Versions / Dependencies

2.41

Reproduction script

See above

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[alexeykudinkin]

Here's an example of this kind of allocation where 2 actors that aren't supposed to fit on the same node are scheduled onto one: