Add ELK stack to docker-compose files

[Killpit]

Changes

-Added ELK stack for people who wants to work with ELK stack with OpenTelemetry
-Combined the additions with both Elastic stack documentation, Docker Hub and the docker-compose and .env file structures on implementing changes.

[cla-checker-service]

💚 CLA has been signed

[rogercoll]

Thanks for creating this PR @Killpit.

I see that most of the changes are in the docker-compose.yml file, which we try not to change/diverge from the upstream fork. The upstream repository is heavily forked and to improve the maintainability (we have nightly auto-syncs) the forked versions rely mostly on environment variables, specifically the .env.override file. Elastic's fork changes should be done using environment variables or by adding new files to avoid upstream conflicts.

In addition, what would happen for cloud deployments? With these changes, users using Elasticsearch cloud would be deploying a local instance too. What do you think if we provide a separate docker-compose file (e.g. docker-compose.elk.yaml) or we update the README to point to some public documentation for local deployments?

[Killpit]

Thanks for creating this PR @Killpit.

I see that most of the changes are in the docker-compose.yml file, which we try not to change/diverge from the upstream fork. The upstream repository is heavily forked and to improve the maintainability (we have nightly auto-syncs) the forked versions rely mostly on environment variables, specifically the .env.override file. Elastic's fork changes should be done using environment variables or by adding new files to avoid upstream conflicts.

In addition, what would happen for cloud deployments? With these changes, users using Elasticsearch cloud would be deploying a local instance too. What do you think if we provide a separate docker-compose file (e.g. docker-compose.elk.yaml) or we update the README to point to some public documentation for local deployments?

I can create a separate Docker Compose file as a solution, I think it would work and then we can update the public documentation alongside with it. What is your opinion on that @rogercoll

[Killpit]

Thanks for creating this PR @Killpit.

I see that most of the changes are in the docker-compose.yml file, which we try not to change/diverge from the upstream fork. The upstream repository is heavily forked and to improve the maintainability (we have nightly auto-syncs) the forked versions rely mostly on environment variables, specifically the .env.override file. Elastic's fork changes should be done using environment variables or by adding new files to avoid upstream conflicts.

In addition, what would happen for cloud deployments? With these changes, users using Elasticsearch cloud would be deploying a local instance too. What do you think if we provide a separate docker-compose file (e.g. docker-compose.elk.yaml) or we update the README to point to some public documentation for local deployments?

I'll create a new docker-compose file and can make the same pull request here, or change it @rogercoll

[rogercoll]

@Killpit I have discussed this PR with the team and we think that to ease the maintenance we could just reference existing tools that actively maintain ELK docker compose files. For example, this is a very new repository that creates a custom ELK docker compose file from a bash script: https://github.com/elastic/start-local

What do you think if we add a new section in the README.md referencing the start-local repository for local ELK deployments?

[Killpit]

@Killpit I have discussed this PR with the team and we think that to ease the maintenance we could just reference existing tools that actively maintain ELK docker compose files. For example, this is a very new repository that creates a custom ELK docker compose file from a bash script: https://github.com/elastic/start-local

What do you think if we add a new section in the README.md referencing the start-local repository for local ELK deployments?

Sounds good to me if I get the credit