Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zip file fix #34399

Merged
merged 33 commits into from
May 26, 2024
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
42d2e3e
Ignore ZipFile in native image
talzich May 16, 2024
57e410d
Merge branch 'master' of github.com:demisto/content into ZipFile-fix
talzich May 16, 2024
c362037
Merge branch 'master' of github.com:demisto/content into ZipFile-fix
talzich May 16, 2024
3438b55
Add UT to ZipFile
talzich May 19, 2024
39ff726
Merge branch 'master' of github.com:demisto/content into ZipFile-fix
talzich May 19, 2024
97e70b7
Update Packs/CommonScripts/Scripts/ZipFile/ZipFile.py
talzich May 20, 2024
2d58aa5
Address CR for ZipFile Script
talzich May 20, 2024
a774887
Merge branch 'master' of github.com:demisto/content into ZipFile-fix
talzich May 20, 2024
1842448
Resolve conflicts
talzich May 20, 2024
3fdc801
Remove test-files from ZipFile
talzich May 20, 2024
30b9686
Fix bug in ZipFile
talzich May 21, 2024
d7eb5da
Merge branch 'master' of github.com:demisto/content into ZipFile-fix
talzich May 21, 2024
0c40809
Merge branch 'master' of github.com:demisto/content into ZipFile-fix
talzich May 21, 2024
a61bb23
Add RN and update version for CommonScripts
talzich May 21, 2024
ae30a06
Fix RN and and pre-commit changes
talzich May 21, 2024
a612a08
Merge branch 'master' of github.com:demisto/content into ZipFile-fix
talzich May 21, 2024
aed276b
address validation errors
talzich May 21, 2024
3735344
Merge branch 'master' into ZipFile-fix
talzich May 22, 2024
e7724a1
Address CR on ZipFile test
talzich May 22, 2024
4900cf3
Merge branch 'master' of github.com:demisto/content into ZipFile-fix
talzich May 22, 2024
5ef51b0
Merge branch 'ZipFile-fix' of github.com:demisto/content into ZipFile…
talzich May 22, 2024
05e7984
Merge branch 'master' into ZipFile-fix
talzich May 22, 2024
29ae1a6
Add troubleshooting section to README
talzich May 22, 2024
bd3856f
Merge branch 'ZipFile-fix' of github.com:demisto/content into ZipFile…
talzich May 22, 2024
549daef
Merge branch 'master' of github.com:demisto/content into ZipFile-fix
talzich May 22, 2024
d6f56df
Fix pre-commit errors
talzich May 22, 2024
e9c9e9d
Empty commit
talzich May 22, 2024
54395a5
Update docker image
talzich May 23, 2024
be0adf5
Merge branch 'master' of github.com:demisto/content into ZipFile-fix
talzich May 23, 2024
69e5a43
Update docker image and fix validations error
talzich May 23, 2024
0cc6f59
Fix conflicts
talzich May 23, 2024
3094d86
Update RN and pack version for CommonScripts
talzich May 23, 2024
5b9c40f
Merge branch 'master' into ZipFile-fix
talzich May 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Packs/CommonScripts/.pack-ignore
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ ignore=RN114
ignore=SC106,BA124

[known_words]
pyminizip
pyzipper
unzipfile
zipstrings
extractinbetween
Expand Down
5 changes: 5 additions & 0 deletions Packs/CommonScripts/ReleaseNotes/1_14_49.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@

#### Scripts

##### ZipFile
- Updated the Docker image to: *demisto/py3-tools:1.0.0.94214*.
talzich marked this conversation as resolved.
Show resolved Hide resolved
51 changes: 30 additions & 21 deletions Packs/CommonScripts/Scripts/ZipFile/ZipFile.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,40 @@

import re
import shutil
import zipfile
import pyminizip
import pyzipper
from os.path import isfile

ESCAPE_CHARACTERS = r'[/\<>"|?*]'


def test_compression_succeeded(zip_name: str, password: str = None):
with pyzipper.AESZipFile(zip_name) as zf:
# testing for file integrity
if password:
zf.setpassword(bytes(password, 'utf-8'))
ret = zf.testzip()
if ret is not None:
raise DemistoException('There was a problem with zipping the file: ' + ret + ' is corrupted')
talzich marked this conversation as resolved.
Show resolved Hide resolved


def compress_multiple(file_names: List[str], zip_name: str, password: str = None):
"""
Compress multiple files into a zip file.
:param file_names: list of file names to compress
:param zip_name: name of the zip file to create
:param password: password to use for encryption
"""
compression = pyzipper.ZIP_DEFLATED
talzich marked this conversation as resolved.
Show resolved Hide resolved
encryption = pyzipper.WZ_AES if password else None
talzich marked this conversation as resolved.
Show resolved Hide resolved
demisto.debug(f'zipping {file_names=}')
with pyzipper.AESZipFile(zip_name, mode='w', compression=compression, encryption=encryption) as zf:
talzich marked this conversation as resolved.
Show resolved Hide resolved
zf.pwd = bytes(password, 'utf-8') if password else None
for file_name in file_names:
zf.write(file_name)
test_compression_succeeded(zip_name, password)
zf.close()


def escape_illegal_characters_in_file_name(file_name: str) -> str:
if file_name:
file_name = re.sub(ESCAPE_CHARACTERS, '-', file_name)
Expand All @@ -20,10 +47,6 @@ def escape_illegal_characters_in_file_name(file_name: str) -> str:


def main():
try: # in order to support compression of the file
compression = zipfile.ZIP_DEFLATED
except Exception:
compression = zipfile.ZIP_STORED
try:
args = demisto.args()
zipName = None
Expand Down Expand Up @@ -75,21 +98,7 @@ def main():
zipName = fileCurrentName + '.zip'

# zipping the file
if password:
pyminizip.compress_multiple(file_names, ['./'] * len(file_names), zipName, password, 5)

else:
zf = zipfile.ZipFile(zipName, mode='w')
try:
for file_name in file_names:
zf.write(file_name, compress_type=compression)
# testing for file integrity
ret = zf.testzip()
if ret is not None:
raise DemistoException('There was a problem with the zipping, file: ' + ret + ' is corrupted')

finally:
zf.close()
compress_multiple(file_names, zipName, password)

with open(zipName, 'rb') as f:
file_data = f.read()
Expand Down
10 changes: 5 additions & 5 deletions Packs/CommonScripts/Scripts/ZipFile/ZipFile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ args:
isArray: true
name: entryID
required: true
- description: 'Name of the output file, for example: zipName="test" would result in output file "test.zip"'
- description: 'Name of the output file, for example: zipName="test" would result in output file "test.zip".'
name: zipName
- description: 'Used to create a password protected zip file. Example: password="abcd"'
- description: 'Used to create a password protected zip file. Example: password="abcd".'
name: password
comment: Zip a file and upload to war room
comment: Zip a file and upload to war room.
commonfields:
id: ZipFile
version: -1
Expand Down Expand Up @@ -54,7 +54,7 @@ tags:
timeout: '0'
type: python
subtype: python3
dockerimage: demisto/py3-tools:1.0.0.49703
dockerimage: demisto/py3-tools:1.0.0.94214
fromversion: 5.0.0
tests:
- ZipFile-Test
- ZipFile-Test
55 changes: 54 additions & 1 deletion Packs/CommonScripts/Scripts/ZipFile/ZipFile_test.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
import os
import pytest
from ZipFile import escape_illegal_characters_in_file_name
import pyzipper
from ZipFile import escape_illegal_characters_in_file_name, compress_multiple
import tempfile

ESCAPE_CHARACTERS_PACK = [
('/Users/user/Downloads/b/a/testingfile.txt', '-Users-user-Downloads-b-a-testingfile.txt'),
Expand All @@ -9,6 +12,56 @@
]


def unzip(zip_file_path: str, password: str = None):
with tempfile.TemporaryDirectory() as unzip_dir, pyzipper.AESZipFile(zip_file_path) as zf:
zf.pwd = bytes(password, 'utf-8') if password else None
zf.extractall(path=unzip_dir)


@pytest.mark.parametrize(('input_name', 'output_name'), ESCAPE_CHARACTERS_PACK)
def test_escape_characters_in_file_name(input_name, output_name):
assert escape_illegal_characters_in_file_name(input_name) == output_name


def test_compress_multiple_with_password():
test_data_dir = './test_data'
file_names = [os.path.join(test_data_dir, f) for f in os.listdir(test_data_dir) if
os.path.isfile(os.path.join(test_data_dir, f))]
with tempfile.NamedTemporaryFile(suffix='.zip') as tmp_zip:
zip_name = tmp_zip.name
compress_multiple(
file_names=file_names,
zip_name=zip_name,
password='123'
)


def test_unzip():
talzich marked this conversation as resolved.
Show resolved Hide resolved
talzich marked this conversation as resolved.
Show resolved Hide resolved
test_data_dir = './test_data'
file_names = [os.path.join(test_data_dir, f) for f in os.listdir(test_data_dir) if
os.path.isfile(os.path.join(test_data_dir, f))]
with tempfile.NamedTemporaryFile(suffix='.zip') as tmp_zip:
zip_name = tmp_zip.name
compress_multiple(
file_names=file_names,
zip_name=zip_name,
password='123'
)
unzip(zip_name, '123')


def test_unzip_wrong_password():
test_data_dir = './test_data'
file_names = [os.path.join(test_data_dir, f) for f in os.listdir(test_data_dir) if
os.path.isfile(os.path.join(test_data_dir, f))]
with tempfile.NamedTemporaryFile(suffix='.zip') as tmp_zip:
zip_name = tmp_zip.name
compress_multiple(
file_names=file_names,
zip_name=zip_name,
password='123'
)
with pytest.raises(Exception) as e:
unzip(zip_name, '1234')

assert 'Bad password' in e.value.args[0]
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file not shown.
Binary file not shown.
1 change: 1 addition & 0 deletions Packs/CommonScripts/Scripts/ZipFile/test_data/test_txt.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
test
talzich marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 1 addition & 1 deletion Packs/CommonScripts/pack_metadata.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"name": "Common Scripts",
"description": "Frequently used scripts pack.",
"support": "xsoar",
"currentVersion": "1.14.48",
"currentVersion": "1.14.49",
"author": "Cortex XSOAR",
"url": "https://www.paloaltonetworks.com/cortex",
"email": "",
Expand Down
8 changes: 8 additions & 0 deletions Tests/docker_native_image_config.json
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,14 @@
"native:dev",
"native:candidate"
]
},
{
"id":"ZipFile",
"reason":"see CIAC-10361",
"ignored_native_images":[
"native:8.6",
"native:candidate"
]
}
],
"flags_versions_mapping":{
Expand Down
Loading