EXCALIBUR_SPACK_ENV on supported systems

[tkoskela]

While helping @cjlegg debug excalibur-tests it came up that setting EXCALIBUR_SPACK_ENV to an environment inside the repo (e.g. EXCALIBUR_SPACK_ENV=excalibur-tests/stage/kathleen/compute-node) doesn't work. The error is this

[ReFrame Setup]
  version:           4.6.2
  command:           '/lustre/scratch/ucapcjg/excalibur-tests-env/bin/reframe -c benchmarks/apps/omb -r --system=kathleen --tag latency --performance-report'
  launched by:       ucapcjg@login01
  working directory: '/lustre/scratch/ucapcjg/excalibur-tests'
  settings files:    '<builtin>', '/home/ucapcjg/Scratch/excalibur-tests/benchmarks/reframe_config.py'
  check search path: '/lustre/scratch/ucapcjg/excalibur-tests/benchmarks/apps/omb'
  stage directory:   '/lustre/scratch/ucapcjg/excalibur-tests/stage'
  output directory:  '/lustre/scratch/ucapcjg/excalibur-tests/output'
  log files:         '/run/user/259482/rfm-5cu3c1gp.log'

[==========] Running 1 check(s)
[==========] Started on Wed Aug 28 12:00:55 2024+0100

[----------] start processing checks
[ RUN      ] Osu_latency /f47ce12a @kathleen:compute-node+default
[     FAIL ] (1/1) Osu_latency /f47ce12a @kathleen:compute-node+default
==> test failed during 'compile_wait': test staged in '/lustre/scratch/ucapcjg/excalibur-tests/stage/kathleen/compute-node/default/Osu_latency'
[----------] all spawned checks have finished

[  FAILED  ] Ran 1/1 test case(s) from 1 check(s) (1 failure(s), 0 skipped, 0 aborted)
[==========] Finished on Wed Aug 28 12:00:58 2024+0100
=================================================================================================================================================================================================================
SUMMARY OF FAILURES
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
FAILURE INFO for Osu_latency (run: 1/1)
  * Description:
  * System partition: kathleen:compute-node
  * Environment: default
  * Stage directory: /lustre/scratch/ucapcjg/excalibur-tests/stage/kathleen/compute-node/default/Osu_latency
  * Node list:
  * Job type: batch job (id=None)
  * Dependencies (conceptual): []
  * Dependencies (actual): []
  * Maintainers: []
  * Failing phase: compile_wait
  * Rerun with '-n /f47ce12a -p default --system kathleen:compute-node -r'
  * Reason: build error: stdout: 'rfm_build.out', stderr: 'rfm_build.err'
--- rfm_build.err (first 10 lines) ---
==> Error: Detected 1 missing include path(s):
   /lustre/scratch/ucapcjg/excalibur-tests/stage/kathleen/compute-node/default/common.yaml
--- rfm_build.err ---

It looks like reframe falls back on a default spack environment. While the documentation does tell users to only set this on unsupported systems, this seems like an easy mistake to make (and took us half a day to debug). We should either

1. Fix

   excalibur-tests/benchmarks/modules/utils.py

   Lines 218 to 240 in f41b992

   	def identify_build_environment(current_partition):
   	# Select the Spack environment:
   	# * if `EXCALIBUR_SPACK_ENV` is set, use that one
   	# * if not, use a provided spack environment for the current partition
   	# * if that doesn't exist, create a persistent minimal environment
   	if os.getenv('EXCALIBUR_SPACK_ENV'):
   	env_dir = cp_dir = os.getenv('EXCALIBUR_SPACK_ENV')
   	subdir = ''
   	else:
   	system, partition = current_partition.fullname.split(':')
   	cp_dir = os.path.realpath(
   	os.path.join(os.path.dirname(__file__), '..', 'spack',
   	system))
   	subdir = partition
   	env_dir = os.path.join(cp_dir, partition)
   	if not os.path.isdir(cp_dir):
   	cmd = run_command(["spack", "env", "create", "--without-view", "-d", env_dir])
   	if cmd.returncode != 0:
   	raise BuildSystemError("Creation of the Spack "
   	f"environment {env_dir} failed")
   	getlogger().info("Spack environment successfully created at"
   	f"{env_dir}")
   	return env_dir, cp_dir, subdir

   to work when EXCALIBUR_SPACK_ENV points to an environment inside the repo, or
2. Make it very clear in

   excalibur-tests/docs/setup.md

   Lines 123 to 133 in f41b992

   	#### Set `EXCALIBUR_SPACK_ENV` variable
   	To use the new Spack environment in ReFrame,
   	set the environment variable `EXCALIBUR_SPACK_ENV` to the path of the directory
   	where the environment is, i.e.
   	```sh
   	export EXCALIBUR_SPACK_ENV=/path/to/environment
   	```
   	If this is not set, ReFrame will try to use the
   	environment for the current system if known, otherwise it will automatically
   	create a very basic environment (see [Usage on unsupported systems](use.md#usage-on-unsupported-systems).

   that it doesn't.