Flexible scheduling: p_item data deduplication

[jpbruinsslot]

Testing if we can solve some p_item data duplication.

Place data (BoefjeTask, NormalizerTask, etc.) in Task definition instead of embedding a p_item in a Task and other models. Then reference the Task in TaskRun, TaskSchedule and PrioritizedItem.

Alternative 1

erDiagram
task_runs {
    uuid id PK
    uuid task_id FK
    timestamp_with_time_zone created_at
    character_varying scheduler_id
    character_varying type
    taskstatus status
    timestamp_with_time_zone modified_at
}
tasks {
    uuid id PK
    jsonb data
    character_varying hash
}
task_schedules {
    uuid id PK
    uuid task_id FK
    boolean enabled
    character_varying scheduler_id
    character_varying cron_expression
    timestamp_with_time_zone deadline_at
    timestamp_with_time_zone evaluated_at
    timestamp_with_time_zone created_at
    timestamp_with_time_zone modified_at
}
p_items {
    uuid id PK
    uuid task_id FK
    timestamp_with_time_zone modified_at
    timestamp_with_time_zone created_at
    character_varying scheduler_id
    character_varying hash
    integer priority
}


task_runs }o--|| tasks: ""
task_schedules ||--|| tasks: ""
p_items ||--|| tasks: ""

Loading

Alternative 2

erDiagram
tasks {
    uuid id PK
    uuid schema_id FK
    timestamp_with_time_zone created_at
    character_varying scheduler_id
    character_varying type
    taskstatus status
    timestamp_with_time_zone modified_at
}
schemas {
    uuid id PK
    jsonb data
    character_varying hash
    character_varying schedule
    boolean enabled
    timestamp_with_time_zone deadline_at
    timestamp_with_time_zone created_at
    timestamp_with_time_zone modified_at
}
p_items {
    uuid id PK
    uuid task_id FK
    timestamp_with_time_zone modified_at
    timestamp_with_time_zone created_at
    character_varying scheduler_id
    character_varying hash
    integer priority
}


tasks }o--|| schemas: ""
p_items ||--|| tasks: ""

Loading

Alternative 3

As per discussion with @Donnype

erDiagram
task {
    uuid id PK
    uuid task_definition_id FK
    character_varying scheduler_id
    integer priority
    timestamp_with_time_zone created_at
    character_varying type
    taskstatus status
    timestamp_with_time_zone modified_at
}
task_definition {
    uuid id PK
    jsonb data
    character_varying hash
    character_varying schedule
    boolean enabled
    timestamp_with_time_zone deadline_at
    timestamp_with_time_zone created_at
    timestamp_with_time_zone modified_at
}

task }o--|| task_definition: ""

Loading

TODO:

• Flexible scheduling: p_item data deduplication > data model definitions #2650
• Flexible scheduling: p_item data deduplication > refactor migrations #2651
• Flexible scheduling: p_item data deduplication > refactor schedulers #2652
• Flexible scheduling: refactor server #2653
• Flexible scheduling: refactor tests #2654
• Flexible scheduling: p_item data deduplication > update filter functionality #2711
• Update cascading rules
• Update runner scheduler service connections
• Update rocky scheduler service connections (different api endpoint, different data models)
• Make sure filters work with new data model

[jpbruinsslot]

@Donnype as per our discussion, I've added your suggestion as alternative 3. While trying out an implementation I came across some difficulties, perhaps nice to discuss this in a call?

[jpbruinsslot]

Closing this issue in favour of #204 together with @Donnype we're proposing the third alternative mentioned here and picking this up in #204