MemoryConstrained
MemoryConstrained[expr,b] evaluates expr, stopping if more than b bytes of memory are requested.
MemoryConstrained[expr,b,failexpr] returns failexpr if the memory constraint is not met.
Please visit the official Wolfram Language Reference for more details and examples on core symbols.