Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking β€œSign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds docker install for testing #590

Merged
merged 153 commits into from
Mar 25, 2025
Merged
Changes from 21 commits
Commits
Show all changes
153 commits
Select commit Hold shift + click to select a range
92efe42
#390 Install v2 pipeline (#392)
cbaxley Aug 8, 2024
626aa75
Upgrade 1x to 2.0 (#428)
cbaxley Sep 16, 2024
bd08785
Updated API and Selenium tests to validate Raw Access Read panel on U…
rishagg01 Sep 16, 2024
5212353
Added encrpyption at rest option for users
Sep 25, 2024
162bf11
Install pipeline and tests (#429)
cbaxley Oct 3, 2024
ead411d
Upgrade API tests for Release 2 (#465)
rishagg01 Oct 9, 2024
c6bedb2
Merge 2.0 into vault user password encryption (#458)
cbaxley Oct 10, 2024
05cba6a
Documentation update to volume and index management (#468)
aarz-snl Oct 15, 2024
f6d6416
add image to docs
aarz-snl Oct 15, 2024
e1d00af
Update index-management.md
aarz-snl Oct 15, 2024
1733589
Fix tests after password encryption (#466)
cbaxley Oct 17, 2024
abba9d3
Mreeve 461 post install (#477)
mreeve-snl Oct 23, 2024
a2cacab
Sysmon Install Powershell Script (#480)
rgbrow1949 Oct 23, 2024
a5196ac
Mreeve elastalert 2 (#483)
mreeve-snl Oct 24, 2024
31039b6
Make the pipeline use the post install script (#481)
cbaxley Oct 24, 2024
4339de8
Ddiabe refactored dashboards 2.0 (#486)
ddiabe Oct 25, 2024
0d1d9db
Updates some obscure bugs on install (#487)
cbaxley Oct 25, 2024
587014c
Remove breaking health check for fleet server
aarz-snl Oct 28, 2024
981c0b1
Harden the pipline steps (#493)
cbaxley Oct 29, 2024
e96a52c
Delete dashboard_refactor directory
ddiabe Oct 30, 2024
352251d
Sleep a bit longer before running the tests
cbaxley Nov 6, 2024
a310de9
Merge branch 'release-2.0.0' into develop
cbaxley Nov 6, 2024
a6ece91
Fix typo
causand22 Nov 6, 2024
f8a41a2
Remove files that were modified in 2.0.0
cbaxley Nov 6, 2024
93e9b19
LME 2 Dashboard Update and Bug Fixes (#501)
aarz-snl Nov 6, 2024
fdabeb0
Merge branch 'release-2.0.0' into develop
cbaxley Nov 6, 2024
8a1fc47
Selenium Test Release 2 Changes (#499)
rishagg01 Nov 7, 2024
976db4b
Merge branch 'release-2.0.0' into develop
cbaxley Nov 7, 2024
34fb759
Remove the cluster run from automatic running.
cbaxley Nov 7, 2024
d465e10
adding in lme-2-docs content (#506)
mreeve-snl Nov 8, 2024
6b56ae0
Merge branch 'release-2.0.0' into develop
cbaxley Nov 8, 2024
119e4bd
Merge branch 'main' into develop
mitchelbaker-cisa Nov 8, 2024
46d729f
Merge branch 'main' into develop
mitchelbaker-cisa Nov 15, 2024
6bb0e9a
Creates a windows qcow for minimega (#515)
cbaxley Nov 22, 2024
9933eb7
Update test_user_h_r_dashboard_lo.py
cbaxley Nov 22, 2024
717874f
Made enhancements to API and Sel tests (#522)
rishagg01 Dec 2, 2024
c516f0e
Made a single change to skip API test that failed on cluster run on d…
rishagg01 Dec 3, 2024
34c1b2c
Skipping API test on metrics search (#525)
rishagg01 Dec 3, 2024
5d4ed3b
Merge branch 'main' into develop
mitchelbaker-cisa Dec 7, 2024
417dff6
Merge branch 'main' into develop
mitchelbaker-cisa Dec 7, 2024
24ef2fc
Remove inline comments (#532)
aarz-snl Dec 13, 2024
ead34e8
Merge branch 'main' into develop
mitchelbaker-cisa Dec 17, 2024
2da8cb0
Create minimega windows instance and install elastic agent (#534)
cbaxley Dec 18, 2024
a724796
Adds docker for temporary install
cbaxley Dec 27, 2024
b2946ca
Adds customer Docker config
cbaxley Dec 30, 2024
d4eef2c
Updates the post install script to be more robust
cbaxley Dec 31, 2024
b1d26d6
Merge branch 'cbaxley-517-docker-for-customers' of https://github.com…
cbaxley Dec 31, 2024
4a5b3a4
Fix the docker build from conflicts
cbaxley Dec 31, 2024
e8df867
Init lme when starting docker container
cbaxley Jan 2, 2025
272c56e
Updates how we wait for fleet to be ready
cbaxley Jan 2, 2025
e00e32d
Modify the docker build to build as much as it can of the lme install
cbaxley Jan 3, 2025
f9b23d9
Remove temporary files
cbaxley Jan 3, 2025
098e6a9
Split up the install of nix and lme into two separate steps in docker
cbaxley Jan 7, 2025
22d0c6d
Adds scripts to check the status of the LME setup in docker
cbaxley Jan 7, 2025
8b8bf6c
Merge branch 'main' into develop
mitchelbaker-cisa Jan 13, 2025
e4cb054
Creates a persistent docker container for LME
cbaxley Jan 15, 2025
ffbbf0d
Updates the ansible script to remove the warnings
cbaxley Jan 20, 2025
68bb273
Update the source dashboard path to be absolute
cbaxley Jan 21, 2025
b8ee020
Adds port check for kibana with retries
cbaxley Jan 22, 2025
347db0e
Fixes the check not to count times it has failed
cbaxley Jan 22, 2025
d7dbcad
Revert "Creates a persistent docker container for LME"
cbaxley Jan 23, 2025
b64bfb8
fixing docs, somehow diffs got included, and we just want to include …
mreeve-snl Jan 25, 2025
da5c8f3
Update the README.md file for the docker setup
cbaxley Jan 28, 2025
0ac4f54
Clean up the README.md file for the docker setup
cbaxley Jan 28, 2025
ff69cb3
Adds a 24.04 docker setup and install
cbaxley Jan 28, 2025
fdd1031
Increment the group id if it already exists
cbaxley Jan 28, 2025
65d1d12
Increment the user id if it already exists
cbaxley Jan 28, 2025
625b54d
Updates the overlayfs in install_lme_local.yml
cbaxley Jan 29, 2025
3fce454
Put back in the shell argument for bash arguments in installers
cbaxley Jan 29, 2025
987b16d
Adds the ability to pass the HOST_IP to the docker-compose.yml file
cbaxley Feb 3, 2025
e87f679
Updates the lme-init.sh file to use the HOST_IP variable
cbaxley Feb 3, 2025
c9747f5
Adds the ability to pass the HOST_IP to the init script
cbaxley Feb 3, 2025
49b76cf
Replace the HOST_IP variable with the HOST_IP environment variable
cbaxley Feb 3, 2025
79572e0
Sets the HOST_IP environment variable in the lme-setup.service file
cbaxley Feb 3, 2025
a012def
Sets the HOST_IP environment variable in the lme-setup.service file
cbaxley Feb 3, 2025
8f37e36
Sets the HOST_IP environment variable in an env file
cbaxley Feb 3, 2025
53a4acd
Debug why the server hosts results are not using env
cbaxley Feb 3, 2025
285de22
Export the IPVAR environment variable in the lme-init.sh file
cbaxley Feb 4, 2025
e758548
Set the IPVAR environment variable more explicitly in the lme-init.sh…
cbaxley Feb 4, 2025
c83afd9
Remove the lme-environment.env file from the base install
cbaxley Feb 4, 2025
6a46cad
Copy in the updated lme-environment.env file to opt/lme/
cbaxley Feb 4, 2025
872b055
Change where the kibana port is checked for
cbaxley Feb 4, 2025
02c85e8
Merge hotfix from main into develop (#570)
mitchelbaker-cisa Feb 7, 2025
6a29858
Updates the readme to reflect the new non-persistent docker setup.
cbaxley Feb 11, 2025
963f4f2
Update main and docker readmes
cbaxley Feb 11, 2025
74cdc27
Remove environment.sh from 22.04 and add example files for 22.04 and …
cbaxley Feb 12, 2025
59431e2
Update docker readme to include instructions for environment.sh
cbaxley Feb 12, 2025
b867216
Mreeve 552 lme quality of life fixes readme (#581)
mreeve-snl Feb 14, 2025
71605ca
Merge branch 'develop' into cbaxley-517-2-non-persistant-docker
cbaxley Feb 18, 2025
1ea2e6a
Leave running assets on cluster run for debugging
cbaxley Feb 18, 2025
07f06e7
Merge branch 'main' into cbaxley-517-2-non-persistant-docker
cbaxley Feb 19, 2025
2cefce5
Fix post install to watch for fleet server hosts and verify they are …
cbaxley Feb 21, 2025
b44a940
Try testing that fleet server is ready.
cbaxley Feb 21, 2025
927986a
Increase the size of the Azure instance to E16_v4
cbaxley Feb 25, 2025
29f85ec
Wait between running local install and post install
cbaxley Feb 25, 2025
afadaa4
Wait more before the post install, check fleet setup
cbaxley Feb 26, 2025
e8565f4
Adds additional checks for fleet setup
cbaxley Feb 26, 2025
a0e75bd
Removes sleep from post install
cbaxley Feb 26, 2025
3bab8bb
Reintroduce cleaning up azure resources after build
cbaxley Feb 27, 2025
f8892fe
Try not installing minimega to see if failures are a networking issue
cbaxley Feb 27, 2025
552c77c
Checks the available versions of ansible and python
cbaxley Feb 27, 2025
bcdf24a
Clean the apt cache before running apt-get update
cbaxley Feb 27, 2025
8ff9134
Enable universe repository
cbaxley Feb 27, 2025
3869ba0
Break out the apt commands into separate lines
cbaxley Feb 27, 2025
3bc6d03
Remove the Kibana and Elasticsearch URLs from the replace_home_in_con…
cbaxley Feb 27, 2025
ba16526
Remove the build containers from the github actions server
cbaxley Feb 27, 2025
2550f2c
Reintroduce minimega into the cluster workflow
cbaxley Feb 28, 2025
9bdc610
Install LME before minimega
cbaxley Feb 28, 2025
b4eb9df
Try a different vm size
cbaxley Feb 28, 2025
2f8b6b6
Hide environment exporter file
cbaxley Feb 28, 2025
e8ace3e
Update cluster.yml to install LME later in the workflow
cbaxley Feb 28, 2025
55cd659
Change fleet setup to accept only 200 status code
cbaxley Mar 2, 2025
15b1cb3
Adds more checks before enabling Fleet in Kibana
cbaxley Mar 3, 2025
af5a533
Wait longer for Kibana and Elasticsearch to start
cbaxley Mar 3, 2025
8f8c0bc
Replace home in config with IP
cbaxley Mar 3, 2025
45a6d24
Revert the local ips
cbaxley Mar 3, 2025
d4dd489
Replace the local ips with the docker dns names
cbaxley Mar 3, 2025
930e14a
Removes setting local_kbn_url and local_es_url to docker dns
cbaxley Mar 4, 2025
08307a9
Remove a checker in install.sh that may be causing the cluster.yml wo…
cbaxley Mar 4, 2025
ae888ec
Open the firewall to all IP addresses for testing
cbaxley Mar 5, 2025
9c6b837
Makes it so the fleet-server container is not started automatically o…
cbaxley Mar 6, 2025
5bcdbc4
Add the clone directory to the post_install_local.yml file
cbaxley Mar 6, 2025
75ee047
Reintroduce the minimega commands
cbaxley Mar 6, 2025
380c648
Move the conditionpathexists to the unit section of the fleet-server.…
cbaxley Mar 6, 2025
cc8a3d8
Remove the minimega install again to check if it is a route problem
cbaxley Mar 6, 2025
34fe378
Do not remove the azure resources for testing purposes
cbaxley Mar 6, 2025
5005627
Set the elasticsearch host in the fleet server container
cbaxley Mar 7, 2025
ba30fca
Add a sleep to the cluster.yml workflow to allow the VM to boot
cbaxley Mar 7, 2025
1059959
Add more sleep for the cluster.yml workflow after the VM is built
cbaxley Mar 7, 2025
3bdef0f
Adds back in the minimega install on the cluster.yml workflow
cbaxley Mar 7, 2025
ab4861d
Sleep even longer for the cluster workflow because of larger machine
cbaxley Mar 7, 2025
bbcf35f
Remove minimega install from cluster.yml workflow
cbaxley Mar 7, 2025
1e7ac04
Wait for the larger machine to fully boot before installing LME
cbaxley Mar 10, 2025
379894d
Install on smaller machine
cbaxley Mar 10, 2025
7c6ca49
Use the larger machine with ssd drive
cbaxley Mar 10, 2025
8bb8053
Increases the size of the cluster machine to 16 cores
cbaxley Mar 10, 2025
cf52f62
Use an intermediate size machine for the cluster
cbaxley Mar 10, 2025
15f7d46
Only start the fleet server if the setup is finished
cbaxley Mar 10, 2025
d8e3cee
Use a larger machine for the cluster
cbaxley Mar 10, 2025
a02d801
Try a full cluster run with the new setup
cbaxley Mar 10, 2025
832cbd8
Install LME on the cluster after the minimega setup is complete
cbaxley Mar 10, 2025
69fea29
Attempt to replace the elasticsearch hosts variable with an environme…
cbaxley Mar 10, 2025
80658cd
Merge branch 'main' into cbaxley-517-2-non-persistant-docker
cbaxley Mar 10, 2025
2ad1fca
Wait for the fleet container to be ready
cbaxley Mar 11, 2025
06be209
Updates the readme for the docker installation
cbaxley Mar 12, 2025
3868ae1
Decrease the VM size for the cluster run
cbaxley Mar 13, 2025
5b54d51
Increases the vm size in cluster to a Standard_E16d_v4
cbaxley Mar 13, 2025
d9fa364
Adds back in the restriction on the IP address
cbaxley Mar 13, 2025
78e46f8
Fixes the missing ip address in the azure cluster workflow
cbaxley Mar 13, 2025
51bfd0e
Merge branch 'main' into cbaxley-517-2-non-persistant-docker
cbaxley Mar 17, 2025
0161a9e
Updating expected output to match now that fleet-server will install
mreeve-snl Mar 18, 2025
7d840a5
Update README.md
tylmorr-snl Mar 18, 2025
06e8e08
Disable integration assistant
aarz-snl Mar 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .devcontainer/python_development/devcontainer.json
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
{
"name": "Python Development",
"dockerComposeFile": [
"../../testing/development/docker-compose.yml"
"../../testing/v2/development/docker-compose.yml"
],
"service": "ubuntu",
"shutdownAction": "none",
"workspaceFolder": "/lme",
"workspaceFolder": "/root/LME",
"customizations": {
"vscode": {
"extensions": [
@@ -15,5 +15,5 @@
]
}
},
"remoteUser": "admin.ackbar"
"remoteUser": "root"
}
30 changes: 8 additions & 22 deletions .github/ISSUE_TEMPLATE/bug-or-error-report.md
Original file line number Diff line number Diff line change
@@ -10,38 +10,24 @@ assignees: ''
## **BEFORE CREATING THE ISSUE, CHECK THE FOLLOWING GUIDES**:
- [ ] [FAQ](https://github.com/cisagov/LME/blob/main/docs/markdown/reference/faq.md)
- [ ] [Troubleshooting](https://github.com/cisagov/LME/blob/main/docs/markdown/reference/troubleshooting.md)
- [ ] Search current/closed issues for similar questions and utilize github/google search to see if an answer exists for the error you are encountering.
- [ ] Search current/closed issues for similar questions, and utilize github/google search to see if an answer exists for the error I'm encountering.

If the above did not answer your question, proceed with creating an issue below:

## Describe the bug
<!-- A clear and concise description of what the software flaw you are experiencing looks like, or what the behavior is. -->

## Expected behavior
A clear and concise description of what you expected to happen.
<!-- A clear and concise description of what the bug is. -->

## To Reproduce
<!-- Steps to reproduce the behavior. These should be clear enough that our team can understand your running environment, software/operating system versions, and anything else we might need to debug the issue. -->
<!-- Good examples can be found here: [Issue 1](https://github.com/cisagov/LME/issues/15) [Issue 2](https://github.com/cisagov/LME/issues/19). -->

### Please complete the following information

#### **Setup**
- Are you running the LME machines in a virtual environment (i.e. Docker) or are you running natively on the machines?
- Which version of LME are you installing?
- Is this a first-time installation or are you upgrading? If upgrading, what was your previous version?

#### **Desktop:** (Client Machines)
- OS: [e.g. Windows 10]
- Browser: [e.g. Firefox Version 104.0.1]
- Software version: [e.g. Sysmon v15.0]

#### **Domain Controller:**
- OS: [e.g. Windows Server]
- Browser: [e.g. Firefox Version 104.0.1]
- Software version: [e.g. Winlogbeat 8.11.1]
#### **Desktop:**
- OS: [e.g. Windows 10]
- Browser: [e.g. Firefox Version 104.0.1]
- Software version: [e.g. Sysmon v15.0, Winlogbeat 8.11.1]

#### **ElasticSearch/Kibana Server:**
#### **Server:**
- OS: [e.g. Ubuntu 22.04]
- Software Versions:
- ELK: [e.g. 8.7.1]
@@ -65,7 +51,7 @@ Increase the number of lines if your issue is not present, or include a relevant
## Expected behavior
A clear and concise description of what you expected to happen.

## Screenshots
## Screenshots **OPTIONAL**
If applicable, add screenshots to help explain your problem.

## Additional context
17 changes: 7 additions & 10 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -18,13 +18,11 @@
## πŸ§ͺ Testing

<!-- How did you test your changes? How could someone else test this PR? -->
<!-- Include details of your testing environment, and the tests you ran to. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->

## βœ… Pre-approval checklist ##
- [ ] There is a [gitIssue](https://github.com/cisagov/LME/issues) that this PR resolves
- [ ] Git Issue that this PR solves has been selected in the Development section
- [ ] The PR's base branch has been modified to be the proper branch.

- [ ] Changes are limited to a single goal **AND**
the title reflects this in a clear human readable format
- [ ] Issue that this PR solves has been selected in the Development section
@@ -35,12 +33,11 @@

## βœ… Pre-merge Checklist

- [ ] All tests pass.
- [ ] PR has been tested and the documentation for testing is above.
- [ ] Squash and merge all commits into one PR level commit.
- [ ] All tests pass
- [ ] PR has been tested and the documentation for testing is above
- [ ] Squash and merge all commits into one PR level commit

## βœ… Post-merge Checklist

- [ ] Delete the branch to keep down number of branches.
- [ ] The PR is labeled with `feat` for an added new feature, `update` for an update, **OR** `fix` for a fix.
- [ ] The PR contains `Resolves #<issue #>` so that merging it closes out the corresponding issue. For example `Resolves #132`.
- [ ] Delete the branch to keep down number of branches

Loading