Quchem multithreading

Message boards : Number crunching : Quchem multithreading
Message board moderation

To post messages, you must log in.

AuthorMessage
ProDigit

Send message
Joined: 16 Nov 19
Posts: 44
Credit: 21,290,949
RAC: 0
Message 915 - Posted: 30 Jun 2020, 15:01:26 UTC

Good news!
My first two, real multithreaded, 8 CPU QChem Long WUs arrived and are crunching with success!
Instead of 3 days for 1CPU, they seem to crunch at around 7.5 to 10 hours est...

The boinc site, "Max # of CPUs for this project" wasn't set, but has a maximum setting of #8 cores.
8 is a good setting, perhaps 12 would also be possible, and slightly better.
24 is too much, as it'll hog up an entire 3900x, and most of a 3950x.
The good thing about such WUs is on Ryzens, they should have shared L-3 cache, allowing data to be shared between cores...
Would be nice if Quchempedia could use that advantage, though current 3000 series chipsets have 2 CPU blocks of 6 or 8 cores.
4000 chipsets should be able to share that data between all cores.
ID: 915 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jim1348

Send message
Joined: 3 Oct 19
Posts: 153
Credit: 32,412,973
RAC: 0
Message 916 - Posted: 30 Jun 2020, 18:22:32 UTC - in response to Message 915.  
Last modified: 30 Jun 2020, 18:24:52 UTC

Instead of 3 days for 1CPU, they seem to crunch at around 7.5 to 10 hours est.

That is just due to normal variations. I crunch with one core per work unit (t1) on a Ryzen 3600 (Ubuntu 18.04), and see 6 to 28 hours in the log at the moment.
ID: 916 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProDigit

Send message
Joined: 16 Nov 19
Posts: 44
Credit: 21,290,949
RAC: 0
Message 920 - Posted: 3 Jul 2020, 4:17:50 UTC - in response to Message 916.  
Last modified: 3 Jul 2020, 4:20:18 UTC

First 4 core WUs also processing well!
My PPD count for the day went up by 2x (I do share projects)

I believe 4 core WUs are a big improvement, especially on AMD Ryzen 3000 series CPUs, over the single threaded WUs!
Shared l-cache might be helping out a lot!

Good job Quchempedia team!
ID: 920 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jim1348

Send message
Joined: 3 Oct 19
Posts: 153
Credit: 32,412,973
RAC: 0
Message 924 - Posted: 3 Jul 2020, 13:17:45 UTC - in response to Message 920.  

I believe 4 core WUs are a big improvement, especially on AMD Ryzen 3000 series CPUs, over the single threaded WUs!
Shared l-cache might be helping out a lot!

OK, I will give it a try on my Ryzen 3600. Sometimes the PPD does not correspond to the work done though. It is a strange credit system, and I normally pay no attention to it.
But if it gets more work done, that is another matter.
ID: 924 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
xii5ku

Send message
Joined: 21 Jun 20
Posts: 24
Credit: 68,559,000
RAC: 0
Message 927 - Posted: 11 Jul 2020, 13:02:44 UTC

"NWChem long" is not a multithreaded application in the stricter sense, in which one process maintains several threads which operate on shared data. Instead, the application spawns one or more separate processes, each operating on own data. These sub-processes are synchronizing with each other only occasionally, via message passing. Therefore I presume that processor cache does *not* make a difference for what little or large overhead there might be for running "NWChem long" with more than a single thread.

Architecturally, a single job of this application could even be spread over an Ethernet cluster of computers, or an Infiniband cluster. When the message passing has to go via a network like this, then there might be a bigger performance impact from communications latency though.
ID: 927 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jim1348

Send message
Joined: 3 Oct 19
Posts: 153
Credit: 32,412,973
RAC: 0
Message 928 - Posted: 11 Jul 2020, 14:18:39 UTC - in response to Message 927.  
Last modified: 11 Jul 2020, 14:22:12 UTC

Thanks. I am back to a single core. I haven't figured out why the multi-core version is even offered, as I haven't seen an advantage.

EDIT: I see your other post. It makes sense if you need a shorter work unit.
ID: 928 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProDigit

Send message
Joined: 16 Nov 19
Posts: 44
Credit: 21,290,949
RAC: 0
Message 939 - Posted: 16 Jul 2020, 5:14:32 UTC

I'm wondering if those multi thread WUs are the ones giving validation errors?
I've got 900 of those, where I presumably not get any points for.
ID: 939 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jim1348

Send message
Joined: 3 Oct 19
Posts: 153
Credit: 32,412,973
RAC: 0
Message 940 - Posted: 16 Jul 2020, 11:58:00 UTC - in response to Message 939.  
Last modified: 16 Jul 2020, 11:59:33 UTC

I'm wondering if those multi thread WUs are the ones giving validation errors?
I've got 900 of those, where I presumably not get any points for.

I don't see that in your results. Your validation errors are all t1, aren't they?
(Strictly speaking, I guess they are all "multi-threaded", just with t=1).

But they are high on some of your machines, at over 5%. I normally get few invalids.
And I don't see much difference in the hardware to account for it. Maybe bad luck.
https://quchempedia.univ-angers.fr/athome/results.php?hostid=2356

You aren't overclocking, are you?
ID: 940 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
jozef j

Send message
Joined: 14 Oct 19
Posts: 1
Credit: 1,037,065
RAC: 0
Message 942 - Posted: 16 Jul 2020, 19:42:40 UTC

hi,, task are very long,, sometimes falling, error in validation, other probllems when computer is slowed with VM runs, and other probllems(low credit for one day task)
will back later when you develop non vm task and more stable.
thanks
ID: 942 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jim1348

Send message
Joined: 3 Oct 19
Posts: 153
Credit: 32,412,973
RAC: 0
Message 943 - Posted: 16 Jul 2020, 19:56:05 UTC - in response to Message 939.  

I'm wondering if those multi thread WUs are the ones giving validation errors?

And another thing is that you are running either older (4.5) or newer (5.4) version of Linux than is standard in Ubuntu 18.04.4, which is 5.3.0-62 at the moment.
Maybe that leads to incompatibilities with certain libraries? I really don't know, but throw it out.
ID: 943 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Number crunching : Quchem multithreading

©2024 Benoit DA MOTA - LERIA, University of Angers, France