When that happens to me I just shut down Boinc, open VB and make certain all tasks stop, then start Boinc again. If that doesn't work, you can always use brute force and restart your system.
If you do nothing at all, I believe the unmanageable tasks will resume processing in 24 hours.

Are you trying to crunch with all available cores? It sure sounds like it.
Momentary spikes in memory usage is the main suspect in triggering the messages you are seeing.
I found through trial and error that my system was at it's most stable running on only 50% of the processors.

I think with that much memory, boinc may be thinking there's virtual memory (swap) getting used.
I'm running 32 threads right now, and I can tell you, you don't need more than 500MB per thread for QuChemPi.
On a system with 64 cores, I would assign between 32GB and the next value up (40GB or 48GB or so).
More is just wasting memory needlessly.

I set my RAM usage to 95% when not used and 90% when in use.
Yet of the 32GB I have, only 14-16 GB are used.