Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make use of tempfile when writing stdout content for 'buildtest build' #1605

Merged
merged 2 commits into from
Aug 29, 2023

Conversation

shahzebsiddiqui
Copy link
Member

This PR will address an issue when running buildtest build command in parallel typically 2 background process will result in some error as shown below.

FileNotFoundError: [Errno 2] No such file or directory: '/Users/siddiq90/Documents/github/buildtest/var/output.txt'

[2]  + 27972 exit 1     buildtest bd -b tutorials/sleep.yml > /dev/null

With this PR it will address this bug, see example below

(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] buildtest bd -b tutorials/sleep.yml 1>/dev/null & 
[1] 29565
(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] buildtest bd -b tutorials/sleep.yml 1>/dev/null & 
[2] 29724
(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] 
[1]  - 29565 done       buildtest bd -b tutorials/sleep.yml > /dev/null
(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] 
[2]  + 29724 done       buildtest bd -b tutorials/sleep.yml > /dev/null

In the history we will see both runs captured

(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] buildtest hy list                                 
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ id ┃ hostname                     ┃ user     ┃ system  ┃ date                ┃ pass tests ┃ fail tests ┃ total tests ┃ command                                           ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 0  │ DOE-7086392.vpn-dhcp.lbl.gov │ siddiq90 │ generic │ 2023/08/29 11:18:41 │ 1          │ 0          │ 1           │ /Users/siddiq90/Documents/github/buildtest/bin/bu │
│    │                              │          │         │                     │            │            │             │ ildtest bd -b tutorials/sleep.yml                 │
├────┼──────────────────────────────┼──────────┼─────────┼─────────────────────┼────────────┼────────────┼─────────────┼───────────────────────────────────────────────────┤
│ 1  │ DOE-7086392.vpn-dhcp.lbl.gov │ siddiq90 │ generic │ 2023/08/29 11:18:43 │ 1          │ 0          │ 1           │ /Users/siddiq90/Documents/github/buildtest/bin/bu │
│    │                              │          │         │                     │            │            │             │ ildtest bd -b tutorials/sleep.yml                 │
└────┴──────────────────────────────┴──────────┴─────────┴─────────────────────┴────────────┴────────────┴─────────────┴───────────────────────────────────────────────────┘

With this change there will be no such file $BUILDTEST_ROOT/var/output.txt instead this will be a temporary file

(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] ls $BUILDTEST_ROOT/var/output.txt
ls: /Users/siddiq90/Documents/github/buildtest/var/output.txt: No such file or directory

The output.txt file is copied over and still present

(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] ls -l $BUILDTEST_ROOT/var/.history/*/output.txt
-rw-r--r--@ 1 siddiq90  92503  10839 Aug 29 11:18 /Users/siddiq90/Documents/github/buildtest/var/.history/0/output.txt
-rw-r--r--@ 1 siddiq90  92503  10839 Aug 29 11:18 /Users/siddiq90/Documents/github/buildtest/var/.history/1/output.txt

We can still query the content so long as file exists

(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] buildtest hy query 0 --output
╭────────────────────────────────────────── buildtest summary ───────────────────────────────────────────╮                                                                  
│                                                                                                        │                                                                  
│ User:               siddiq90                                                                           │                                                                  
│ Hostname:           DOE-7086392.vpn-dhcp.lbl.gov                                                       │                                                                  
│ Platform:           Darwin                                                                             │                                                                  
│ Current Time:       2023/08/29 11:18:38                                                                │                                                                  
│ buildtest path:     /Users/siddiq90/Documents/github/buildtest/bin/buildtest                           │                                                                  
│ buildtest version:  1.5                                                                                │                                                                  
│ python path:        /Users/siddiq90/.local/share/virtualenvs/buildtest-Ir4AdrfC/bin/python3            │                                                                  
│ python version:     3.10.12                                                                            │                                                                  
│ Configuration File: /Users/siddiq90/Documents/github/buildtest/buildtest/settings/config.yml           │                                                                  
│ Test Directory:     /Users/siddiq90/Documents/github/buildtest/var/tests                               │                                                                  
│ Report File:        /Users/siddiq90/Documents/github/buildtest/var/report.json                         │                                                                  
│ Command:            /Users/siddiq90/Documents/github/buildtest/bin/buildtest bd -b tutorials/sleep.yml │                                                                  
│                                                                                                        │                                                                  
╰────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                  
─────────────────────────────────────────────────────────────────────────  Discovering Buildspecs ──────────────────────────────────────────────────────────────────────────
                      Discovered buildspecs                       
╔════════════════════════════════════════════════════════════════╗
║ buildspec                                                      ║
╟────────────────────────────────────────────────────────────────╢
║ /Users/siddiq90/Documents/github/buildtest/tutorials/sleep.yml ║
╚════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
──────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/Users/siddiq90/Documents/github/buildtest/tutorials/sleep.yml: VALID
Total builder objects created: 1
                                                                   Builders by type=script                                                                    
┏━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder        ┃ type   ┃ executor           ┃ compiler ┃ nodes ┃ procs ┃ description     ┃ buildspecs                                                     ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ sleep/d798dda7 │ script │ generic.local.bash │ None     │ None  │ None  │ sleep 2 seconds │ /Users/siddiq90/Documents/github/buildtest/tutorials/sleep.yml │
└────────────────┴────────┴────────────────────┴──────────┴───────┴───────┴─────────────────┴────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────── Building Test ───────────────────────────────────────────────────────────────────────────────
sleep/d798dda7: Creating test directory: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/sleep/sleep/d798dda7
sleep/d798dda7: Creating the stage directory: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/sleep/sleep/d798dda7/stage
sleep/d798dda7: Writing build script: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/sleep/sleep/d798dda7/sleep_build.sh
────────────────────────────────────────────────────────────────────────────── Running Tests ───────────────────────────────────────────────────────────────────────────────
Spawning 1 processes for processing builders
─────────────────────────────────────────────────────────────────────────────── Iteration 1 ────────────────────────────────────────────────────────────────────────────────
sleep/d798dda7 does not have any dependencies adding test to queue
In this iteration we are going to run the following tests: [sleep/d798dda7]
                                                 Test Summary                                                 
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder        ┃ executor           ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime  ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ sleep/d798dda7 │ generic.local.bash │ PASS   │ None None None                      │ 0          │ 2.495698 │
└────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴──────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /Users/siddiq90/Documents/github/buildtest/var/report.json
Writing Logfile to: /Users/siddiq90/Documents/github/buildtest/var/logs/buildtest_zwhm1zem.log


…t build'. This will avoid

issue where multiple buildtest build commands are running in parallel since previously it was writing to same file
@shahzebsiddiqui shahzebsiddiqui merged commit 20ad993 into devel Aug 29, 2023
31 of 34 checks passed
@shahzebsiddiqui shahzebsiddiqui deleted the write_buildtest_build_stdout_to_temporary_file branch August 29, 2023 17:53
@@ -183,7 +182,11 @@ def main():
if cmd.build_success():
build_history_dir = cmd.get_build_history_dir()

shutil.move(fname, os.path.join(build_history_dir, "output.txt"))
shutil.copyfile(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be a good idea to close the file before you copy it in case any write buffering or something else random happens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants