Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows Terminalで異常な文字がある場合、途中で出力されなくなる #382

Closed
YamatoSecurity opened this issue Jan 30, 2022 · 7 comments · Fixed by #378 or #485
Closed
Assignees
Labels
bug Something isn't working Priority:High
Milestone

Comments

@YamatoSecurity
Copy link
Collaborator

YamatoSecurity commented Jan 30, 2022

Windows Terminalで異常な文字が結果にある場合、途中で出力されなくなって、CTRL+Cを押して強制終了する必要があります。
デフォルトルールとサンプルEVTXファイルで再現できます。Windows 10のWindows Terminalで検証しています。
./target/release/hayabusa -d ../hayabusa-sample-evtx -r ../hayabusa-rules
因みに、カラー出力する際は無事に出力されます。

Windows Terminal Output Bug

2019-07-20 00:11:26.852 +09:00,MSEDGEWIN10,1,informational,Process Creation,"Command: ""C:\Windows\system32\cmd.exe"" /c ""ntdsutil “ac i ntds� “ifm� “create full C:\Atomic_Red_Team q q"" : Path: C:\Windows\System32\cmd.exe : User: MSEDGEWIN10\IEUser : Parent Command: powershell",../hayabusa-rules/hayabusa/sysmon/events/1_ProcessCreation.yml,../hayabusa-sample-evtx/EVTX-ATTACK-SAMPLES/AutomatedTestingTools/PanacheSysmon_vs_AtomicRedTeam01.evtx
の行で出力が止まってしまいます。

コントロールコードのHex: 0x9Dが問題っぽいです。

@YamatoSecurity YamatoSecurity added bug Something isn't working Priority:Medium labels Jan 30, 2022
@YamatoSecurity YamatoSecurity added this to the v1.1 milestone Jan 30, 2022
@hitenkoku
Copy link
Collaborator

hitenkoku commented Jan 30, 2022

この問題はWindows Terminal側の問題なので、同様の事象がないかどうか確認してみます

@YamatoSecurity
Copy link
Collaborator Author

もう少し調べました。コントロール文字が原因のようです。
microsoft/terminal@6742965
で修正しているけど、いつTerminalをアップデートしたら直るか分かりません。(fixはまだdevブランチにしかないようです)
(修正されるまでにworkaroundないかな? 因みにhttps://github.com/microsoft/terminal/issues/10310 で議論されていました。)

取りあえずReadmeで「Terminalはまだバグっているので、現在使えない可能性がある。」
1.1が出た時に「カラー出力したら無事に出力されるので、出力が途中で止まる場合は-cを付けてください」という風に説明しましょうか?

@hitenkoku
Copy link
Collaborator

hitenkoku commented Jan 30, 2022

ご確認ありがとうございます。 一応 https://github.com/microsoft/terminal/blob/main/doc/terminal-v2-roadmap.md でロードマップは出てますが……そのときに入るんでしょうかね……

ひとまずReadmeの方は、ご指摘の通り、修正をかけて、対応する形を取ったほうが良さそうですね(流石にterminalをdevからもってこいとは言えない……)

@hitenkoku
Copy link
Collaborator

こちらの対応ですが、 #378 で対応をさせていただこうと思います

@hitenkoku hitenkoku linked a pull request Feb 3, 2022 that will close this issue
hitenkoku added a commit that referenced this issue Feb 9, 2022
* added color code emit_csv test

* replaced HashMap and HashSet to hashbrown #368

* removed debug output in test #368

* added color option #376

* fixed process of output check #376

* removed color output check from test #376

* english updates

* colored detections and rules count output by level #384

* refactoring in colored output process #384

* update usage #364 #376

* fixed markdown lint

* added windows terminal bug evasion way #382

* update readme

* fixed colored output test

Co-authored-by: Tanaka Zakku <[email protected]>
@hitenkoku
Copy link
Collaborator

closed via #378

@hitenkoku hitenkoku added this to the v1.1 milestone Mar 7, 2022
@hitenkoku
Copy link
Collaborator

Windows Terminal v1.12.1073 でもこの事象は解決していないことを確認したのでメモ代わりにコメントしておきます

@hitenkoku
Copy link
Collaborator

#481 (comment) の記載にある通り、文字列からコントロール文字列を削除することで対応することが可能であるかを検討するため本issueをreopenします

ref #468 #464

@hitenkoku hitenkoku reopened this Apr 5, 2022
@hitenkoku hitenkoku modified the milestones: v1.1, v1.2 Apr 5, 2022
hitenkoku added a commit that referenced this issue Apr 5, 2022
* added control character filter in details #382

* fixed document

- removed fixed windows teminal caution in readme
@hitenkoku hitenkoku added the waiting merge main branch already resolved develop branch. label Apr 5, 2022
hitenkoku added a commit that referenced this issue Apr 15, 2022
* Fix/fix clippy warn (#434)

- Fixed following Clippy Warnings(previous warning count: 671 -> after: 4)
  - clippy::needless_return
  - clippy::println_empty_string
  - clippy::redundant_field_names
  - clippy::single_char_pattern
  - clippy::len_zero
  - clippy::iter_nth_zero
  - clippy::bool_comparison
  - clippy::question_mark
  - clippy::needless_collect
  - clippy::unnecessary_unwrap
  - clippy::ptr_arg
  - clippy::needless_collect
  - clippy::needless_borrow
  - clippy::new_without_default
  - clippy::assign_op_pattern
  - clippy::bool_assert_comparison
  - clippy::into_iter_on_ref
  - clippy::deref_addrof
  - clippy::while_let_on_iterator
  - clippy::match_like_matches_macro
  - clippy::or_fun_call
  - clippy::useless_conversion
  - clippy::let_and_return
  - clippy::redundant_clone
  - clippy::redundant_closure
  - clippy::cmp_owned
  - clippy::upper_case_acronyms
  - clippy::map_identity
  - clippy::unused_io_amount
  - clippy::assertions_on_constants
  - clippy::op_ref
  - clippy::useless_vec
  - clippy::vec_init_then_push
  - clippy::useless_format
  - clippy::bind_instead_of_map
  - clippy::bool_comparison
  - clippy::clone_on_copy
  - clippy::too_many_arguments
  - clippy::module_inception
  - fixed clippy::needless_lifetimes
  - fixed clippy::borrowed_box (Thanks for helping by hach1yon!)

* Merge main and output fix#443#444 (#445)

* removed tools/sigmac (#441)

* removed tools/sigmac

- moved tools/sigmac to hayabusa-rules repo

* fixed doc link tools/sigmac

* fixed submodule track

* fixed submodule track from latest to v1.1.0 tag

* fixed link

* erased enter #444

* erased enter #444

* reverted logo enter

* fixed rules submodule target commit #444

Co-authored-by: Yamato Security <[email protected]>

* readme update screenshots etc (#448)

* Opensslを静的にコンパイルするためにCargo.tomlの設定変更 (#437)

* cargo update - openssl static

* updated cargo

* macos2apple

* cargo update

* cargo update

* aliasキーがない場合もEvent.EventDataを自動で走査する (#442)

* add no event key

* support not-register-alias search

* added checking EventData when key do not match in alias #290

- added checking key in Event.EventData, if key is not exist in eventkey_alias.txt.

* cargo fmt

* fixed panic when filter files does not exists

* fixed errorlog format when filter config files does not exist

Co-authored-by: DustInDark <[email protected]>

* changed downcast library from mopa to downcast_rs #447 (#450)

* Fixed Clippy Warnings (#451)

* fixed clippy warn

* fixed cargo clippy warnging

* fixed clippy warngings in clippy ver 0.1.59

* fixed clippy warnings clippy::unnecessary_to_owned

* added temporary blackhat arsenal badge

* added rust report card badges #453

* added repository maintenance levels badge #453

* documentation update macOS usage etc

* update

* added clippy workflow #428 (#429)

* added clippy workflow #428

* fixed action yaml to run clippy #428

* fixed indent

* fixed workflow

* fixed workflow error

* fixed indent

* changed no annotation #428

* adujusted annotation version

* fixed clippy::needless_match

* remove if let exception

* removed unnecessary permission check #428

* statistics event id update (#457)

* Feature/#440 refactoring #395 (#464)

* updated submodule

* fix degrade for pull req #464 (#468)

* fix degrade for pull req #464

* add trim

* Fearture/ added output update result#410 (#452)

* add git2 crate #391

* added Update option #391

* updated readme #391

* fixed cargo.lock

* fixed option if-statement #391

* changed utc short option and rule-update short option #391

* updated readme

* updated readme

* fixed -u long option & version number update #391

* added fast-forwarding rules repository #391

* updated command line option #391

* moved output logo prev update rule

* fixed readme #391

* removed recursive option in readme

* changed rules update from clone and pull to submodule update #391

* fixed document

* changed unnecessary clone recursively to clone only

* English message update.

* cargo fmt

* English message update. ( 4657c35 cherry-pick)

* added create rules folder when rules folder is not exist

* fixed gitmodules github-rules url from ssh to https

* added output of updated file #420

* fixed error #410

* changed update rule list seq

* added test

* fixed output #410

* fixed output and fixed output date field  when  modified field is lacked #410

* fixed compile error

* fixed output

- added enter after Latest rule update output
- added output when no exist new rule
- fixed Latest rule update date format
- changed output from 'Latest rule update' to 'Latest rules update'

* fixed compile error

* changed modified date source from rules folder to each yml rule file

* formatting use chrono in main.rs

* merge develop clippy ci

* fixed output when no update rule #410

- removed Latest rule update

- no output "Rules update successfully" when No rule changed

* Change English

Co-authored-by: Tanaka Zakku <[email protected]>

* Remove unnecessary code from timeline_event_info and rename files for… (#470)

* Remove unnecessary code from timeline_event_info and rename files for issue462

* Remove unnecessary code #462

* add equalsfield pipe (#467)

* Enhancement: add config config #456 (#471)

* added config option #456

* added process of option to speicifed config folder #456

following files adjust config option.

* noisy_rules.txt

* exclude_rules.txt

* fixed usage in readme

* updated rules submodule:

* fixed process when yml file exist in .git folder

* ignore when yml file exist in .git folder

* Add: --level-tuning option's outline

* Add: read Rule files

* Add: input rule_level.txt files & read rules

* cargo fmt

* Add: level-tuning function

* Reface: split to options file

* WIP: Text overwrite failed...

* Fix: Text overwrite was failed

* Add: Error handlings

* Add: id, level validation

* mv: IDS_REGEX to configs file

* fix: level tuning's file name

* Cargo fmt

* Pivot Keyword List機能の追加 (#412)

* add get_pivot_keyword() func

* change function name and call it's function

* [WIP] support config file

* compilete output

* cargo fmt

* [WIP] add test

* add test

* support -o option in pivot

* add pivot mod

* fix miss

* pass test in pivot.rs

* add comment

* pass all test

* add fast return

* fix output

* add test config file

* review

* rebase

* cargo fmt

* test pass

* fix clippy in my commit

* cargo fmt

* little refactor

* change file input logic and config format

* [WIP] change output

* [wip] change deta structure

* change output & change data structure

* pass test

* add config

* cargo fmt & clippy & rebase

* fix cllipy

* delete /rules/ in .gitignore

* clean comment

* clean

* clean

* fix rebase miss

* fix rebase miss

* fix clippy

* file name output on -o to stdout

* add pivot_keywords.txt to ./config

* updated english

* Documentation update

* cargo fmt and clean

* updated translate japanese

* readme update

* readme update

Co-authored-by: DustInDark <[email protected]>
Co-authored-by: Tanaka Zakku <[email protected]>

* Add: test

* Add: README.md

* Cargo fmt

* Use
#[cfg(test)]

* Fixed output stop when  control char exist in windows terminal (#485)

* added control character filter in details #382

* fixed document

- removed fixed windows teminal caution in readme

* fixed level tuning test and added test files #390

* changed level_tuning.txt header from next_level to new_level

* fixed convert miss change to low level

* added run args rules path to check test easy #390

* fixed comment out processing in level_tuning.txt

* fixed config to show level-tuning option

* fixed level-tuning option usage from required to option

* reduce output mitre attack detail tachnique No. by config file (#483)

* reduced mitre attck tag output by config file #477

* prepared 1.2.0 version toml

* added test files and mitre attck strategy tag file #477

* fixed cargo.toml version

* updated cargo.lock

* output tag english update

* cargo fmt

Co-authored-by: Tanaka Zakku <[email protected]>

* Fix: test file's path was incorrect

* Add: add test_files/config/level_tuning.txt

* Add: Flush method.

* inserted debug data

* reverted config usage

* fixed test yaml file path

* Feature/#216 output allfields csvnewcolumn (#469)

* refactoring

* refactoring

* under constructing

* underconstructing

* under construction

* underconstructing

* fix existing testcase

* finish implement

* fmt

* add option

* change name

* fix control code bug

* fix disp

* change format and fix testcase

* fix help

* Fix: show usage when hayabusa has no args

* rm: debug line

* Enhance/warning architecture#478 (#482)

* added  enhance of architecture check #478

* changed check architecture process after output logo #478

* English msg update

* fixed detect method of os-bit to windows and linux

* removed mac and unix architecture and binary and updated its process of windows

* fix clippy

* added check on Wow64 env #478

* Update contributors.txt

Co-authored-by: Tanaka Zakku <[email protected]>

* added --level-tuning option to usage

* Revert "added --level-tuning option to usage"

This reverts commit e6a7409.

* readme update

* Update README-Japanese.md

* readme, version, cargo update

* typo fix

* typo fix

* rm: duplicated test & fix test name

* Add: show logo, and some infos

* small english fix

* twitter link fix (#486)

* added feature of tag output reducing to agg condition #477 (#488)

* changed level output from informational to info #491

* updated rules submodule

* v1.2 changelog update (#473)

* changelog update

* Update CHANGELOG.md

added contributor in "Fields that are not defined in eventkey_alias.txt will automatically be searched in Event.EventData."

ref #442

* Update CHANGELOG-Japanese.md

Fields that are not defined in eventkey_alias.txt will automatically be searched in Event.EventData.

added contributor in "Fields that are not defined in eventkey_alias.txt will automatically be searched in Event.EventData."

ref #442

* Update CHANGELOG.md

added bug fixes (#444) and `Performance and. accuracy`  add contributor ref(#395)

* Update CHANGELOG-Japanese.md

* Translated v1.2 change log to Japanese

v1.2の内容を日本語に修正

* fixed typo

added lacked back quote.

* added description

added following issue and pr description to readme

- #216 / #469 L8
- #390 / #459 L9
- #478 / #482 L19
- #477/ #483 L20

* added description README.md

added following issue and pr description to readme

- #216 / #469 L8
- #390 / #459 L9
- #478 / #482 L19
- #477/ #483 L20

* changelog update

* changelog update

* update

Co-authored-by: DustInDark <[email protected]>

* updated rules #493 (#494)

* Resolve conflict develop (#496)

* removed tools/sigmac (#441)

* removed tools/sigmac

- moved tools/sigmac to hayabusa-rules repo

* fixed doc link tools/sigmac

* fixed submodule track

* fixed submodule track from latest to v1.1.0 tag

* fixed link

* fixed rules submodule targe #444

* updated submodule

* updated rules submodule

Co-authored-by: Yamato Security <[email protected]>

Co-authored-by: Yamato Security <[email protected]>
Co-authored-by: kazuminn <[email protected]>
Co-authored-by: James / hach1yon <[email protected]>
Co-authored-by: garigariganzy <[email protected]>
Co-authored-by: itiB <[email protected]>
@hitenkoku hitenkoku removed the waiting merge main branch already resolved develop branch. label May 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Priority:High
Projects
None yet
2 participants