`auto_accounts` requires all slurm clusters to share the same database

[simonLeary42]

AccountCache.accounts uses just one slurm adapter:

https://github.com/OSC/ondemand/blob/4acd0a2a3ac3cb46787f757005255d1890be10f2/apps/dashboard/app/lib/account_cache.rb#L13-L14

      cluster = Configuration.job_clusters.select(&:slurm?).first
      cluster.nil? ? [] : cluster.job_adapter.accounts

In contrast, queues_per_cluster loops over each slurm adapter:

https://github.com/OSC/ondemand/blob/4acd0a2a3ac3cb46787f757005255d1890be10f2/apps/dashboard/app/lib/account_cache.rb#L149-L150

        Configuration.job_clusters.each do |cluster|
          hash[cluster.id] = cluster.job_adapter.queues

This works (I think) because scontrol is specific to one cluster, while sacctmgr is not.

I think that if accounts were stored in more than one slurm database, any account in any job adapter but the 1st would be omitted from auto_accounts.

Would it not be more natural to make the slurm adapter to limit its sacctmgr output to the relevant cluster, and loop over each cluster in AccountCache.accounts, just like how we looop over each cluster in AccountCache.queues_per_cluster? This would also remove the need to store cluster information in each AccountInfo, just like how there's no need to store cluster information in each QueueInfo.

It would be easier to reason about a job adapter if it was definitely/always limited to one cluster, especially if an adapter needs to be scheduler-agnostic.

[simonLeary42]

AccountCache.dynamic_accounts does loop over each job adapter, but it still relies on AccountCache.accounts, which might lead to errors if dynamic_accounts tries to work with accounts that are not found in accounts.

[johrstrom]

I think you were right to put it into the ondemand repository. The way these adapters work is per cluster - i.e., the adapter itself represents a way to interact with a single cluster.

So it seems to me the best route is to have adapters pull accounts for their cluster, then have the upper layer (like OnDemand) stitch it all together.

[simonLeary42]

I think the 1st step would be to add a cluster=... argument to the sacctmgr command in the slurm adapter here in the core, then do the stitching in ondemand. Maybe it doesn't matter which repo the issue is in.

[johrstrom]

I think the 1st step would be to add a cluster=... argument to the sacctmgr command in the slurm adapter

Are you sure that's supported? I already fixed an issue using -M <cluster>

https://github.com/OSC/ood_core/blame/930e5ccf30e562b53f690654ce1e6333805e6036/lib/ood_core/job/adapters/slurm.rb#L479

and

#806

[simonLeary42]

sacctmgr show assoc where user=$USER cluster=unity works on my system.

[simonLeary42]

the slurm CLI tools are notoriously inconsistent with each other.

[johrstrom]

sacctmgr show assoc where user=$USER cluster=unity works on my system.

Cool, works for me too. OK cool - so we can tighten up the query here then update it upstream to stitch it all together.

[simonLeary42]

ok, I'll start on that once I manage to write tests for ondemand/4471 without getting distracted 🙃