Add Index-based adaptor for integrations

[Swiddis]

Description

Index Adaptor is implemented as wrapper around the JSON reader. Solution seems to work fine for the current integrations, but may need to be refactored later for larger counts. The PR also includes refactoring to start using the adaptor as a source for integrations in the TemplateManager, and several smaller lint fixes as part of gradual work on #1363. When merged, users can add integrations by POSTing their serialized forms to .kibana. A Zip reader would still be helpful since integration serialization is specific to this code, but once the serialization is ported to another tool it should be straightforward to upload bundles like this. I expect this to be the method used by the Catalog repository to perform automatic uploading, instead of a zip file which is easier for manual uploads.

I ran into issues while testing the functionality with document length, many existing integrations are too large to be serialized directly. It turns out OS has a document length limit of 1 MB. As a short-term workaround, optimizing images and removing sample data is sufficient to get all current integrations under 0.5 MB (image optimization in general would help in a future PR). In the long term we plan to move images and sample data out of the integration data entirely, they'll be handled by the catalog.

This PR also makes integrations objects manageable in the Saved Objects table:

Issues Resolved

Resolves #1380

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: 15 lines in your changes are missing coverage. Please review.

Comparison is base (4bffb00) 53.87% compared to head (111f9a9) 53.97%.

Files	Patch %	Lines
server/routes/integrations/integrations_router.ts	16.66%	10 Missing ⚠️
...tors/integrations/repository/index_data_adaptor.ts	88.88%	3 Missing ⚠️
...rver/adaptors/integrations/integrations_manager.ts	85.71%	1 Missing ⚠️
...ver/adaptors/integrations/repository/repository.ts	95.83%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1381      +/-   ##
==========================================
+ Coverage   53.87%   53.97%   +0.10%     
==========================================
  Files         316      317       +1     
  Lines       11309    11349      +40     
  Branches     3014     3018       +4     
==========================================
+ Hits         6093     6126      +33     
- Misses       5167     5174       +7     
  Partials       49       49              

Flag	Coverage Δ
dashboards-observability	53.97% <80.26%> (+0.10%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mengweieric]

I'm not quite familiar with how the entire index adaptor works, but just wondering if there could have any performance issues as I saw there are recursive file traversals as well as file system operations. How did we perform performance tests and would this be a potential issue to the overall cluster health if the cluster is undergoing high traffic with less computing resource available.

[Swiddis]

Short answer is that it's not very performant: #1389. Among other things I want to try adding a cache in a future PR, which should fix the bulk of the issues, but I still need to figure out a framework for more thorough profiling to make sure I'm optimizing the right things.

Altogether there shouldn't be more than a couple dozen megabytes of integrations content at any point (the total of the 11 we have so far is 2.5MB and I don't imagine users adding 100+ integrations), so I don't see any issues with adding an in-memory cache, though I'm not sure how I'm going to handle cache invalidation.