Add standard interface for handling file uploads in the Dioptra client #708
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jkglasbrenner
changed the title
jkglasbrenner
force-pushed
the
add-uploading-support-to-client
branch
5 times, most recently
from
ec75e73 to
9374ea7
Compare
This update adds file uploading support to the Dioptra client library through a new DioptraFile
class, supporting utility functions, and extending the client's post method to accept `data` and
`files` arguments. The changes enable users to upload single files, multiple files, or all files
within a directory while enforcing strict path validation in the associated filenames to prevent
server-side directory traversal attacks.
The new functionality includes:
- A DioptraFile class that validates filenames to ensure they are normalized, POSIX-compliant
relative paths without traversal components (e.g., "..").
- File selection utility functions:
- select_files_in_directory: Recursively selects files from a directory with optional regex
pattern filtering
- select_one_or_more_files: Selects individual files with optional renaming support to handle
filename collisions
- Extended POST request in the client to handle multipart form data that contains regular form
fields and file uploads. All file uploads, whether they're single file or multiple files, are
streamed to the server.
The DioptraSession implementation for the requests-based client and the Flask test client have both
been updated to support the new functionality.
jkglasbrenner
force-pushed
the
add-uploading-support-to-client
branch
from
9374ea7 to
3164d79
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses client-side solution for #694
Added requests-toolbelt as a new project dependency to support streaming of multi-file uploads.
This branch does not include any new unit tests for this new functionality, as it will start being exercised via the REST API integration tests once we integrate the solutions for #694 with #658. However, in the interest of verifying this new feature for the client works, I created this branch, https://github.com/usnistgov/dioptra/tree/test-client-uploading-support, that adds a simple
/echoesendpoint that allows you to upload files, and the server echoes back the name and size of the files you upload. I tested it locally and I also created two simple REST API integration tests based on this endpoint to validate the test client implementation. Both seem to be working as expected.Todos
DioptraFileobjects for upload. The filenames will retain relative filepaths with respect to the source directory's root path, and be standardized in the POSIX format.DioptraFileobject/list ofDioptraFileobjects to upload. The filenames will be stripped of any directory paths.