Apparently simple resource request not possible in canonical jobspec?

[SteVwonder]

@grondo's original suggestion for a feature to flux-jobspec:

... isn't the key difference between flux jobspec srun and sbatch that sbatch should insert a flux start or flux broker before the provided COMMAND (which is assumed to be a batch script), and ensure that the tasks run once per node? (or something similar

@SteVwonder's reply:

... To make that last piece about one per node work, I think --nodes would have to be a required argument. If it is optional and the user only specifies --cores, then the tool will have to make a determination/assumption about the number of cores per node......actually, now that I think about this, is it even possible to specify that kind of request in the canonical jobspec? The request of thinking of is along the lines of: "I want N cores. I don't care about the number of nodes, but I want 1 process per node."

@grondo's reply:

It does feel like we got ourselves into a bind by requiring that the slot shape be defined as part of the resource request. There are certain things that are now not possible or very difficult to express (though it could just be my misunderstanding of current jobspec definition). In general, I don't see how you can define a slot at a parent of the resource type you want to request without affect the exact count (and exclusivity) of the parent resource type (if that makes any sense, specific example is what we're discussing here where you want N cores with a slot of a node, but don't care how many nodes. In fact, the exact slot shape can only be determined from a concrete resource set)

A couple ideas on how to handle this come to mind:

• add a concept of predefined slots that match resource names, e.g. node, socket, core that allow the slot to be left out of the jobspec resources section, e.g. in the case of flux jobspec sbatch you could emit a resources section with no slot and a task: with slot: label node:
• add a special value for count: like any or 0 that indicates no preference for the number of this type, and any slot that contains an any resource as a direct child would by definition apply to exactly 1 of that resource.

[SteVwonder]

One other option that comes to mind, allowing under the count of a task the use of a reference to a resource within the task's slot. For example:

resource:
- type: slot
  name: foo
  count: 1
  with:
  - type: node
    count: {min: 1}
    with:
    - type: core
      count: 30
tasks:
- command: foo
  slot: foo
  count:
    per_resource: # option A
      type: node
      count: 1
    per_resource: {node: 1} # option B

[grondo]

Ah, yes, I'm always forgetting that the task section can specify more than one task per slot.

I like the simplicity of your suggestion, and the nice thing about this approach is that you could use a semi-canned tasks: section for launching a new instance, e.g. something like

tasks:
- command: [ "flux", "start" ]
  slot: $slot_name
  count:
    per_resource: { "type": "node", "count": 1 }

You just have to somehow substitute the user-provided slot name.

The only thing I don't like, is this seems like a kludge around the slot concept. In these cases the job is always just one big slot, and you leave it up to the execution system to distribute your N tasks across the megaslot, and I could imagine cases where things are not going to go the way you expect (granted, the per-node case is straightforward)

Note also since you are forced to define a slot at the node level, you can only request nodes exclusively in this case, which may not be what a user wants if they just want an instance with access to 30 cores.

On the other hand, if we had some reserved, predefined slots like node socket core, then the tasks: section for an instance really is canned:

tasks:
- command: [ "flux", "start" ]
  slot: node

You could also make a slot definition optional in the resources: section when requesting a new instance (or really anytime), which not only makes it more straightforward to craft the resource request, but works around the node exclusivity problem.

The entire example is reduced to:

resource:
- type: core
  count: 30
task:
- command: [ "flux", "start" ]
  slot: node
  count:
    per_slot: 1

[grondo]

Note also since you are forced to define a slot at the node level, you can only request nodes exclusively in this case,

Oh, I do feel a bit silly now -- exclusivity can be disabled via the exclusive field on the node resource that is a child of the slot (at least I think that is how it is supposed to work).

However, I think that requiring users to set things up in this way will run us into trouble (nobody will remember to set exclusive: false even when that is what they want >50% of the time), though I suppose as long as we mostly have utilities to generate jobspec and tools to validate, it won't be a problem in practice.

[SteVwonder]

Summarizing a coffee convo @grondo, @garlick, and I had (please feel free to correct me/ add stuff I missed).

We discussed that we should probably continue down the path of making the canonical jobspec as explicit and easy to statically verify as possible. We can hide the verbosity behind front-end tools like flux-jobspec or put syntactic sugar in a different spec that then compiles down to canonical jobspec. Given that, I think we agreed to keep the explicit slot and add the per_resource: {type: X, count: N} option to tasks: count in order to satisfy the resource/job requirements listed in the OP.

One additional thing we discussed, any resources referenced in the tasks section must explicitly exist in the resources section. For example, the following jobspec would be invalid since node is referenced in tasks but not in resources:

resources:
- type: slot
  name: foo
  count: 1
  with:
  - type: core
    count: 30
tasks:
- command: foo
  slot: foo
  count:
    per_resource: {type: node, count: 1}

Additionally, it is not enough for the referenced resource to exist in the resources section. If a resource is referenced by a particular task in the tasks section, the resource must explicitly appear in the slot referenced by that particular task. For example, the following jobspec would be invalid since the node is not inside the foo slot:

- type: node
  count: {min: 1}
  with:
  - type: slot   
    name: foo
    count: 1
    with:
    - type: core
      count: 30
tasks:
- command: foo
  slot: foo
  count:
    per_resource: {type: node, count: 1}

[SteVwonder]

Given that this would require the inclusion of ranges for counts, are we OK with not supporting this use case in the V1 of jobspec (#149)? Not supporting this case would also eliminate the need to include per_resource (it's relatively easy, in V1, to get N tasks per-node or per-core without it).

One task per node (for a fixed number of nodes, N):

- type: node
  count: N
  with:
  - type: slot   
    name: foo
    count: 1
    with:
    - type: core
      count: 1
tasks:
- command: foo
  slot: foo
  count:
    per_slot: 1

One task per core (for a fixed number of cores, C):

- type: slot   
  name: foo
  count: C
  with:
  - type: core
    count: 1
tasks:
- command: foo
  slot: foo
  count:
    per_slot: 1

Or is this use-case important enough that we want to support it on TOSS4 and in V1 of the jobspec (and thus require support for min counts in both the simple-sched and flux-sched sched)?

[SteVwonder]

PS - If we decide to not support this in V1 of jobspec, I think this issue can be closed, since #154 was merged.

EDIT

PS PS - Looks like per_resource will be required by @trws's proposal in #2213.

[trws]

It would be if we want to support running things like flux instances in a natural way when the resource request isn't phrased in terms of nodes. That doesn't mean we need it for the first cut, but it would make certain things pretty awkward.