feat (ui): pipeline fetch/publish using mapping set

[lordmallam]

Fetch kernel artefacts and stores them as pipeline -> contracts grouped by mapping sets
Allows pipeline create, edit and publish from UI to kernel using mapping sets
Revalidate grouped contracts on pipeline update
Refactor identity mapping to use mapping sets as contracts
Enforce read only pipelines (on input(pipeline) and contract levels individually)

Related JIRA Tickets:
AET-346
AET-342
AET-277
AET-347
AET-343

[obdulia-losantos]

Same comment as in the serializers.py file... Parlez vous français? Anglais? Espagnol? 🤷‍♀️

[obdulia-losantos]

another styling comment... lines with several indentations are hard to follow.

outcome['exists'].append({
    entity_type['name']: '{} schema with id {} exists'.format(
        entity_type['name'],
        contract.kernel_refs['schemas'][entity_type['name']]),
})

As an example... obviously there are more and better options

[obdulia-losantos]

This is checking the pipeline or the contract? Because if it's the contract the pipeline is accessible via contract.pipeline and you can remove the "duplicated" argument.

Also the outcome could be declared inside and not include as an argument (it's the returned value)

[obdulia-losantos]

Same comment, publishing pipeline and all its contracts or only one contract?

[obdulia-losantos]

I think that the project id should be extracted from the pipeline mapping set and not kept within the contract...

    # check linked Project I (using pipeline mapping set)
    project_id = None
    mappingset_id = None
    
    if pipeline.mappingset:
        mappingset_id = pipeline.mappingset
        try:
            kernel_mappingset = kernel_data_request(f'mappingsets/{pipeline.mappingset}/')
            project_id = mappingset.get('project')
            # take from kernel_mappingset the data to be push back to kernel
        except Exception:
            # take from pipeline the data to be pushed to kernel
            pass
    else:
        mappingset_id = str(uuid.uuid4())
        # take from pipeline the data to be pushed to kernel

    # check linked Project II (using given project name)
    if not project_id and project_name:
        try:
            projects = kernel_data_request(f'projects/?name={project_name}')['results']
            aux_project = projects[0] if len(projects) else {}
            project_id = aux_project.get('id')
        except Exception as e:
            pass

    # check linked Project III (if everything else failed...)
    if not project_id:
        project_id = str(uuid.uuid4())

and continue with contract...

[lordmallam]

The algorithm here is to retrieve the linked project using pipeline.mappingset or contract.kernel_refs['project'] if previously publish, or create and link an Aux project to all new pipelines coming from the ui

[obdulia-losantos]

You can skip these lines just calling the the method list
return self.list(request)

[lordmallam]

I'll stick with this implementation. Don't want to deal with pagination

[obdulia-losantos]

You can easily skip pagination with this option 😉

class PipelineViewSet(viewsets.ModelViewSet):
    pagination_class = None

The PRO of using return self.list(request) is that if you have searches/filters/permissions/ordering/... you don't need to implement them twice.

As an example /pipelines/fetch/?fields=id should return only the list of ids, with your manual approach it doesn't. 🤷‍♀️

[obdulia-losantos]

If you are not using the returned pipelines, you can save memory and resources not creating the array.

[obdulia-losantos]

Any action here?

[obdulia-losantos]

I feel like I'm the only one caring about translations... 🙄

[obdulia-losantos]

🇩🇪 🇬🇧 🇫🇷 🇪🇸

[obdulia-losantos]

Do we try to achieve a 1000 chars line? 😧

[obdulia-losantos]

Merge conflict?

[lordmallam]

Should the ui depend on kernel which in turn depends on db or just db?

[obdulia-losantos]

To avoid "healthy" problems we removed the containers dependencies, only the databases are included, but if you prefer to have it, the most important dependency is the database 😉

  ui:
    extends:
      file: ./docker-compose-base.yml
      service: ui-base
    depends_on:
      db:
        condition: service_healthy
      kernel:
        condition: service_started
      # use this dependency with HRM, otherwise comment it out
      # ui-assets:
      #   condition: service_started
    networks:
      internal:
        aliases:
          - ui.aether.local

[obdulia-losantos]

I see sass-lint complaining...

> sass-lint --verbose

./css/_pipelines.scss
  100:1  warning  Space expected between blocks  empty-line-between-blocks

./css/base/_global.scss
  236:1   warning  Space expected between blocks                  empty-line-between-blocks
  242:1   warning  Space expected between blocks                  empty-line-between-blocks
  242:13  warning  Pseudo-elements must start with double colons  pseudo-element
  243:18  warning  Pseudo-elements must start with double colons  pseudo-element

✖ 5 problems (0 errors, 5 warnings)

[obdulia-losantos]

could you use an id that indicates that the text is "read only"? ;-)

[obdulia-losantos]

😬

[obdulia-losantos]

Could you merge current parallel-extraction into this branch? All tests may pass now 🤞

[obdulia-losantos]

Simpler: if outcome.get('error'): 😉

[obdulia-losantos]

I still think that there are a couple of things to be done/rethought... but so far it's great.
👏