https://quchempedia.univ-angers.fr/athome/workunit.php?wuid=3453985

What I don't understand is one of the failures I checked says this:
[...]
Which doesn't look like an error to me.

To fix this issue, I edited the file /lib/systemd/system/boinc-client.service and replaced ProtectSystem=strict by ProtectSystem=full

systemctl stop boinc-client
sed -i 's/ProtectSystem=strict/ProtectSystem=full/g' /lib/systemd/system/boinc-client.service
systemctl daemon-reload
systemctl start boinc-client

I have 2 hosts on Debian 11, where one (#10506) works fine and the other (#10563) returns invalid workunits after ~3 seconds. Looks like the difference came down to the BOINC client's systemd service file. 10506 has "PrivateTmp=true" whereas 10563 has "#PrivateTmp=true #Block X11 idle detection". Everything else in the file is the same, including "ProtectSystem=strict". After changing 10563 to use PrivateTmp, it has started returning valid results.

Just checked the boinc-client packages on Debian today. Only 7.16.17+dfsg-2 (no longer available for amd64) included a service file that uses PrivateTmp. The other recent versions (7.16.16+dfsg-1, 7.18.1+dfsg-4) have it commented out and the old version (7.14.2+dfsg-3) doesn't even have that line.

EDIT: Looks like PrivateTmp was commented out to fix idle detection (issue, pull request). Apparently other projects have similar issues. It seems the long-term fix is for the QuChem program to write to the slot folder instead of /tmp.

Volunteers frequently affected by the postponed issue may try a different vboxwrapper.

BOINC's wiki pages mention communication problems between vboxwrapper and VirtualBox 6.x, especially on Windows.
They offer premade executables that may solve the problems:
https://boinc.berkeley.edu/trac/wiki/VboxApps#Premadevboxwrapperexecutables

It would be the job of the project developers to test those vboxwrappers and distribute them to the clients.
As long as this is not done volunteers could use the following steps as a workaround:

1. Download an alternative vboxwrapper from the page mentioned above (or use one you got from another project, e.g. LHC@home)
2. Start the BOINC client but suspend computing
3. Change to the project directory, e.g. projects/www.cosmologyathome.org, and replace the vboxwrapper there with the test version; the filename must be the name of the old vboxwrapper
4. Resume computing -> check the logfiles of tasks started after the patch


Each restart of the BOINC client will replace the patch with the original vboxwrapper from the project server.
This can be avoided setting <dont_check_file_sizes>1</dont_check_file_sizes> in cc_config.xml, but then all other automatic updates will also not work.

I seem to be getting tasks that get stuck and don't really go anywhere. For example, I have one task that has been running for 8 days and 6 hours, but has only registered 01:24:00 of CPU time. These tasks also never seem to exceed 3 seconds of CPU time since the last checkpoint.

Does anybody know what exactly is going on here? Should I just abort these tasks?

#!/bin/bash

# Edit this:
#    a list of hosts, each optionally with GUI port number appended
#    (may be just a single host, or dozens of hosts)
hosts=(
	"localhost"
	"computer_a"
	"computer_b:31420"
)

# Edit this:
#    the password from gui_rpc_auth.cfg
#    This script expects the same password on all hosts.
#    Can be set to "" if you have empty gui_rpc_auth.cfg's.
password="$(cat /var/lib/boinc/gui_rpc_auth.cfg)"

# Edit this if you want to apply this to a different project.
project_url="https://boinc.bakerlab.org/rosetta/"

# Change this from "abort" to "suspend" if you prefer.
task_op="abort"

# Before a task hasn't been executing for some time, other task stats
# may still be imprecise.  The script therefore does not touch any
# tasks which haven't been executing for at least this many seconds.
# You can use integer numbers here, but not floating point numbers.
# E.g.: 5 * 60 for 5 minutes.
min_elapsed_time=$((5 * 60))

# After tasks were aborted, boinc-client may cease to request
# new work due to "Communication deferred". To avoid this, should a
# project update be forced after one or more tasks were aborted?
# Set to 1 for yes, 0 for no.
force_project_update=1

# Loop intervals.
# You probably don't need to edit these.
check_every_n_minutes=10
timestamp_every_n_minutes=120

# That's it; there is probably no need to edit anything from here on.
delay=$((${check_every_n_minutes}*60/${#hosts[*]}+1))
ts=${timestamp_every_n_minutes}

echo "Monitoring ${hosts[*]}."
for ((;;))
do
	(( (ts += check_every_n_minutes) >= timestamp_every_n_minutes )) && { date; ts=0; }

	for host in ${hosts[*]}
	do
		# Edit this if you run on Cygwin:
		#    boinccmd="/cygdrive/c/Program*Files/BOINC/boinccmd --host ${host} --passwd ${password}"
		if [ -n "${password}" ]
		then
			boinccmd="boinccmd --host ${host} --passwd ${password}"
		else
			boinccmd="boinccmd --host ${host}"
		fi

		tasks=$(${boinccmd} --get_tasks) || { sleep ${delay}; continue; }

		unset name url state ett cct
		while read line
		do
			case ${line} in
		             		[1-9]* )	 i=${line%)*};;
		        	     "name: "* )  name[$i]=${line#*"name: "};;
			      "project URL: "* )   url[$i]=${line#*"project URL: "};;
			"active_task_state: "* ) state[$i]=${line#*"active_task_state: "};;
			"elapsed task time: "* )       tmp=${line#*"elapsed task time: "}; ett[$i]=${tmp%.*};;
			 "current CPU time: "* )       tmp=${line#*"current CPU time: "};  cct[$i]=${tmp%.*};;
			esac
		done <<< "${tasks}"

		n=0
		for j in ${!name[*]}
		do
			# Skip tasks
			#   - which do not belong to this project,
			#   - which are not currently running,
			#   - which have been running for less than $min_elapsed_time seconds,
			#   - which have a CPU time of more than 50% of elapsed time.
			[ "${url[$j]}"   != "${project_url}" ] && continue
			[ "${state[$j]}" != "EXECUTING"      ] && continue
			e=${ett[$j]}; ((e < min_elapsed_time)) && continue
			c=${cct[$j]}; ((e < 2*c)) && continue

			printf "${host}: ${task_op} ${name[$j]}\t"
			printf "(elapsed: %02d:%02d:%02d," $((e/3600)) $((e%3600/60)) $((e%60))
			printf " CPU: %02d:%02d:%02d)\n"   $((c/3600)) $((c%3600/60)) $((c%60))
			${boinccmd} --task "${project_url}" "${name[$j]}" "${task_op}"
			((n++))
		done

		((force_project_update && n)) && { sleep 1; ${boinccmd} --project "${project_url}" update; }

		sleep ${delay}
	done
done

I tweak feeder and scheduler again. Normally, retries are accelerated AND the oldest jobs are preferred. I also doubled the capacity in shared memory. Let's see !

I have added the config file and set my preference to only crunch the longs, but still have not received a work unit.

it seems the thread count does not have to be a power of two. I started a 14-threaded tasks successfully, but it has yet to complete, and to validate of course.

Most likely harmless to the NWChem (...long) application.

PrivateTmp=true

In theory this should be OK for NWChem long.

ProtectSystem=strict

This is probably the culprit! As I understand the documentation, this will make /tmp read-only.
Either relax this from strict to full, or append

-/tmp

to the ReadWritePaths line.
Then restart the boinc-client service. Or maybe you even need to reboot, I don't know.
Then fetch one QuChemPedIA task and see if it runs normally.

ProtectControlGroups=true

In theory this should be OK.

(Documentation of the systemd service file format is spread over "man systemd.unit", "man systemd.service", and "man systemd.exec".)

I WANT a long work unit, but just one. I have set already 1 CPU.

<app_config>
    <app>
        <name>nwchem_long</name>
        <max_concurrent>1</max_concurrent>
    </app>
</app_config>

<app_config>
    <project_max_concurrent>1</project_max_concurrent>
</app_config>

Max # jobs: 1

The limit on tasks in progress is enforced by the server, i.e. works independently of any client-side settings.