8055289: Internal Error: mallocTracker.cpp:146 fatal error: Should not use malloc for big memory block, use virtual memory instead Summary: Return NULL if memory allocation size is bigger than MAX_MALLOC_SIZE when NMT is on Reviewed-by: coleenp, gtriantafill