Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[multipart] Support bytes[]/content type #2380

Merged
merged 49 commits into from
Feb 2, 2024
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
94e6e3d
init
msyyc Jan 23, 2024
a45a47c
rename file
msyyc Jan 23, 2024
575568f
lint
msyyc Jan 23, 2024
b8fd3ff
review
msyyc Jan 25, 2024
97932ba
update
msyyc Jan 26, 2024
ca46a17
fix cspell
msyyc Jan 26, 2024
44747f3
Merge branch 'main' into multipart-file-array-dev5
msyyc Jan 29, 2024
fba4a91
update test
msyyc Jan 29, 2024
f7d4ce6
update test
msyyc Jan 29, 2024
f0eb959
accept files input, doesn't work
Jan 29, 2024
7280ce8
don't use unreleased tcgc
Jan 29, 2024
a847978
Merge branch 'main' of https://github.com/Azure/autorest.python into …
Jan 30, 2024
6493106
Merge branch 'main' of https://github.com/Azure/autorest.python into …
Jan 30, 2024
3d78277
revert test changes to multipart
Jan 30, 2024
fbc88e6
try to pass payload multipart, waiting on tcgc
Jan 31, 2024
e72641e
can generate list of multipart file types
Jan 31, 2024
5518329
regen
Jan 31, 2024
9c4db12
update to dev 4 version
Feb 1, 2024
445d4a3
use dev.5
Feb 1, 2024
3e57197
passes multipart test
Feb 1, 2024
3e70bac
almost there with legacy
Feb 1, 2024
913c3f5
remove file_properties
Feb 1, 2024
34d9725
add back doc formatting
Feb 1, 2024
915f34b
black
Feb 1, 2024
7062043
Merge branch 'main' of https://github.com/Azure/autorest.python into …
Feb 1, 2024
4a887a1
linting checks
Feb 1, 2024
b41e539
fix gen and add dpg changes
Feb 1, 2024
ca66468
remove multipartFile section from types
Feb 1, 2024
8734335
run black
Feb 1, 2024
fe19141
remove files added to azure mock api tests
Feb 1, 2024
0375e3d
use eggs for azure-core and corehttp in main
Feb 1, 2024
1d977fd
import email.utils
Feb 1, 2024
8f53d03
generate with optional filetype
Feb 1, 2024
0da11ba
give typehint to _data
Feb 1, 2024
7bfd08f
fix typing for files
Feb 1, 2024
6704313
bump azure-core versions
Feb 1, 2024
dcf59fa
regen with fixed files typing
Feb 1, 2024
3eb1f38
prepare changelog and package.json
Feb 1, 2024
2c256e6
black
Feb 1, 2024
6d3da87
fix corehttp dep in setup.py
Feb 2, 2024
2533326
mypy fix
Feb 2, 2024
c15d6b4
use FilesType typing for _files
Feb 2, 2024
43520d4
fix mypy hopefully
Feb 2, 2024
e8d1b7a
regen with correct relative import of FileType
Feb 2, 2024
ad3174d
improve code structure
Feb 2, 2024
9731549
fix generation for autorest
Feb 2, 2024
e49b726
add complex tests
Feb 2, 2024
7ad0dbb
fix mypy and pyright of generated helper functions
Feb 2, 2024
a7a41c9
remove unused imports in generated code
Feb 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 1 addition & 15 deletions packages/autorest.python/autorest/codegen/models/operation.py
Original file line number Diff line number Diff line change
Expand Up @@ -583,23 +583,9 @@ def imports(self, async_mode: bool, **kwargs: Any) -> FileImport:
)
file_import.add_import("json", ImportType.STDLIB)
else:
file_import.add_submodule_import(
relative_path, "_model_base", ImportType.LOCAL
)
file_import.add_submodule_import("io", "IOBase", ImportType.STDLIB)
file_import.add_submodule_import(
f"{relative_path}_vendor",
"multipart_file",
ImportType.LOCAL,
)
file_import.add_submodule_import(
f"{relative_path}_vendor",
"multipart_data",
ImportType.LOCAL,
)
file_import.add_submodule_import(
f"{relative_path}_vendor",
"handle_multipart_form_data_model",
"handle_multipart_form_data_body",
ImportType.LOCAL,
)
if self.default_error_deserialization or any(
Expand Down
23 changes: 22 additions & 1 deletion packages/autorest.python/autorest/codegen/models/parameter.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@
from .constant_type import ConstantType
from .utils import add_to_description
from .combined_type import CombinedType
from .model_type import JSONModelType
from .model_type import JSONModelType, DPGModelType
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wit this dpg PR Azure/typespec-azure#166 all we have to do is create a new type MultipartFileType and then type this type as Union[binary, Tuple[string, binary], Tuple[string, binary, string]]

from .primitive_types import ByteArraySchema
from .list_type import ListType

if TYPE_CHECKING:
from .code_model import CodeModel
Expand Down Expand Up @@ -278,6 +280,25 @@ def has_json_model_type(self) -> bool:
return self.type.target_model_subtype((JSONModelType,)) is not None
return isinstance(self.type, JSONModelType)

@property
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also we can actually support data and files entered, and pass them as HttpRequest(data=data, files=files) once this pr gets merged and released Azure/azure-sdk-for-python#34021

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before #2380 (comment), it passes data and files. Now the code merge all part into files and it works. SDK users will not be influenced no matter SDK merge all in files or pass data/files so I think current logic is OK.

def file_properties(self) -> List[str]:
model_type = None
if isinstance(self.type, CombinedType):
model_type = self.type.target_model_subtype((JSONModelType, DPGModelType))
elif isinstance(self.type, (JSONModelType, DPGModelType)):
model_type = self.type
if model_type is None:
return []
return [
prop.wire_name
for prop in model_type.properties
if isinstance(prop.type, ByteArraySchema)
or (
isinstance(prop.type, ListType)
and isinstance(prop.type.element_type, ByteArraySchema)
)
]

@classmethod
def from_yaml(
cls, yaml_data: Dict[str, Any], code_model: "CodeModel"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -755,12 +755,8 @@ def _serialize_body_parameter(self, builder: OperationType) -> List[str]:
body_param = cast(BodyParameter, builder.parameters.body_parameter)
if body_param.is_form_data:
return [
f"if isinstance({body_param.client_name}, _model_base.Model):",
f" _body = handle_multipart_form_data_model({body_param.client_name})",
"else:",
f" _body = {body_param.client_name}",
"_files = {k: multipart_file(v) for k, v in _body.items() if isinstance(v, (IOBase, bytes))}",
"_data = {k: multipart_data(v) for k, v in _body.items() if not isinstance(v, (IOBase, bytes))}",
'file_properties = ["' + '", "'.join(body_param.file_properties) + '"]',
msyyc marked this conversation as resolved.
Show resolved Hide resolved
f"_files = handle_multipart_form_data_body({body_param.client_name}, file_properties)",
]
retval: List[str] = []
body_kwarg_name = builder.request_builder.parameters.body_parameter.client_name
Expand Down Expand Up @@ -977,7 +973,6 @@ def _create_request_builder_call(
f"{' # type: ignore' if type_ignore else ''}"
)
if request_builder.has_form_data_body:
retval.append(" data=_data,")
retval.append(" files=_files,")
elif request_builder.overloads:
seen_body_params = set()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -131,10 +131,11 @@ def serialize_vendor_file(self, clients: List[Client]) -> str:
ImportType.SDKCORE,
)
if self.code_model.has_form_data:
file_import.add_submodule_import("typing", "List", ImportType.STDLIB)
file_import.add_submodule_import("typing", "Tuple", ImportType.STDLIB)
file_import.add_submodule_import("typing", "Union", ImportType.STDLIB)
file_import.add_submodule_import("typing", "Any", ImportType.STDLIB)
file_import.add_submodule_import("io", "IOBase", ImportType.STDLIB)
file_import.add_submodule_import("io", "BytesIO", ImportType.STDLIB)
file_import.add_import("uuid", ImportType.STDLIB)
file_import.add_import("json", ImportType.STDLIB)
file_import.add_mutable_mapping_import()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,37 +66,63 @@ def prep_if_none_match(etag: Optional[str], match_condition: Optional[MatchCondi
return None
{% endif %}
{% if code_model.has_form_data %}
class NamedBytesIO(BytesIO):
def __init__(self, name: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.name = name

def multipart_file(file: Union[IOBase, bytes]) -> IOBase:
if isinstance(file, IOBase):
JSON = MutableMapping[str, Any] # pylint: disable=unsubscriptable-object

# file-like tuple could be `(filename, IO (or bytes))` or `(filename, IO (or bytes), content_type)`
FileType = Union[IOBase, bytes]
FileTuple = Union[Tuple[str, FileType], Tuple[str, FileType, str]]
MultiPartFile = Union[IOBase, bytes, FileTuple]
HandledMultiPartFile = Union[IOBase, FileTuple]

def multipart_file(file: MultiPartFile) -> HandledMultiPartFile:
if isinstance(file, (IOBase, tuple)):
return file
return NamedBytesIO("auto-name-" + str(uuid.uuid4()), file)
return ("auto-name-" + str(uuid.uuid4()), file)

def multipart_data(data: Any) -> Any:
if isinstance(data, (list, tuple, dict, Model)):
return json.dumps(data, cls=SdkJSONEncoder, exclude_readonly=True)
return data

def handle_multipart_form_data_model(body: Model) -> MutableMapping[str, Any]: # pylint: disable=unsubscriptable-object
"""handle first layer of model.
def handle_multipart_form_data_model(body: Model, file_properties: List[str])-> JSON: # pylint: disable=unsubscriptable-object
"""handle first layer of model.
If its value is bytes or IO, replace it with raw value instead of serialized value.

:param body: The model to handle.
:type body: ~payload.multipart._model_base.Model
:type body: ._model_base.Model
:param file_properties: The properties of the model that are file type.
:type file_properties: list[str]
:return: The handled model.
:rtype: MutableMapping[str, Any]
"""
result = body.as_dict()
rest_name_attr = {v._rest_name: k for k, v in body._attr_to_rest_field.items()} # pylint: disable=protected-access
for rest_name in result.keys():
attr = rest_name_attr.get(rest_name)
if attr is not None:
raw_value = getattr(body, attr, None)
if isinstance(raw_value, (bytes, IOBase)):
result[rest_name] = raw_value
if attr is not None and rest_name in file_properties:
result[rest_name] = getattr(body, attr, None)
return result

def handle_multipart_form_data_body(body: Union[Model, JSON], file_properties: List[str])-> List[Tuple[str, HandledMultiPartFile]]:
"""handle multipart form data body.

:param body: The body to handle.
:type body: ._model_base.Model or dict[str, Any]
:param file_properties: The properties of the model that are file type.
:type file_properties: list[str]
:return: The handled body.
:rtype: list[tuple[str, HandledMultiPartFile]]
"""
_body = handle_multipart_form_data_model(body, file_properties) if isinstance(body, Model) else body
files = []
for field_name, value in _body.items():
if field_name in file_properties:
if isinstance(value, list):
files.extend([(field_name, multipart_file(i)) for i in value])
else:
files.append((field_name, multipart_file(value)))
else:
files.append((field_name, multipart_data(value)))
return files
{% endif %}
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
# --------------------------------------------------------------------------

from io import BytesIO, IOBase
from io import IOBase
import json
import sys
from typing import Any, Union
from typing import Any, List, Tuple, Union
import uuid

from ._model_base import Model, SdkJSONEncoder
Expand All @@ -19,16 +19,19 @@
from typing import MutableMapping # type: ignore # pylint: disable=ungrouped-imports


class NamedBytesIO(BytesIO):
def __init__(self, name: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.name = name
JSON = MutableMapping[str, Any] # pylint: disable=unsubscriptable-object

# file-like tuple could be `(filename, IO (or bytes))` or `(filename, IO (or bytes), content_type)`
FileType = Union[IOBase, bytes]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FileTuple = Union[Tuple[str, FileType], Tuple[str, FileType, str]]
MultiPartFile = Union[IOBase, bytes, FileTuple]
HandledMultiPartFile = Union[IOBase, FileTuple]

def multipart_file(file: Union[IOBase, bytes]) -> IOBase:
if isinstance(file, IOBase):

def multipart_file(file: MultiPartFile) -> HandledMultiPartFile:
if isinstance(file, (IOBase, tuple)):
return file
return NamedBytesIO("auto-name-" + str(uuid.uuid4()), file)
return ("auto-name-" + str(uuid.uuid4()), file)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we giving auto names to these file fields? That's def not what we should be doing. We can get the field name from the typespec and then create a tuple of all of the files entries.

So

class MyMultipartModel:
  profile_image: MultipartFile = rest_field("profileImage")
  pictures: MultipartFile[] = rest_field("pictures")

def my_multipart_func(body: MyMultipartModel):
  _files = [("profileImage", body.profile_image)]
  _files.extend([("pictures", p)] for p in body.pictures)
  request = HttpRequest(..., files=_files)
  ...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we giving auto names to these file fields?
-> According to https://www.rfc-editor.org/rfc/rfc7578#section-4.2, file name SHOULD be supplied. In my local test, the flask of python/multer of js will not parse request into files part if no file name provided. So I think SDK need to provide a name for it if users don't provide.
image

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we giving auto names to these file fields? That's def not what we should be doing. We can get the field name from the typespec and then create a tuple of all of the files entries.

So

class MyMultipartModel:
  profile_image: MultipartFile = rest_field("profileImage")
  pictures: MultipartFile[] = rest_field("pictures")

def my_multipart_func(body: MyMultipartModel):
  _files = [("profileImage", body.profile_image)]
  _files.extend([("pictures", p)] for p in body.pictures)
  request = HttpRequest(..., files=_files)
  ...

Current logic is actually similar to yours and it also compatible when body type is JSON

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHOULD doesn't mean MUST. If the IO is not a file descriptor and we don't have a name, we don't set it. No autoname, this will be sent to the servirce, that may save it, or even act based on the received name.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading the code more, this function should not exist



def multipart_data(data: Any) -> Any:
Expand All @@ -37,21 +40,48 @@ def multipart_data(data: Any) -> Any:
return data


def handle_multipart_form_data_model(body: Model) -> MutableMapping[str, Any]: # pylint: disable=unsubscriptable-object
def handle_multipart_form_data_model(
body: Model, file_properties: List[str]
) -> JSON: # pylint: disable=unsubscriptable-object
"""handle first layer of model.
If its value is bytes or IO, replace it with raw value instead of serialized value.

:param body: The model to handle.
:type body: ~payload.multipart._model_base.Model
:type body: ._model_base.Model
:param file_properties: The properties of the model that are file type.
:type file_properties: list[str]
:return: The handled model.
:rtype: MutableMapping[str, Any]
"""
result = body.as_dict()
rest_name_attr = {v._rest_name: k for k, v in body._attr_to_rest_field.items()} # pylint: disable=protected-access
for rest_name in result.keys():
attr = rest_name_attr.get(rest_name)
if attr is not None:
raw_value = getattr(body, attr, None)
if isinstance(raw_value, (bytes, IOBase)):
result[rest_name] = raw_value
if attr is not None and rest_name in file_properties:
result[rest_name] = getattr(body, attr, None)
return result


def handle_multipart_form_data_body(
body: Union[Model, JSON], file_properties: List[str]
) -> List[Tuple[str, HandledMultiPartFile]]:
"""handle multipart form data body.

:param body: The body to handle.
:type body: ._model_base.Model or dict[str, Any]
:param file_properties: The properties of the model that are file type.
:type file_properties: list[str]
:return: The handled body.
:rtype: list[tuple[str, HandledMultiPartFile]]
"""
_body = handle_multipart_form_data_model(body, file_properties) if isinstance(body, Model) else body
files = []
for field_name, value in _body.items():
if field_name in file_properties:
if isinstance(value, list):
files.extend([(field_name, multipart_file(i)) for i in value])
else:
files.append((field_name, multipart_file(value)))
else:
files.append((field_name, multipart_data(value)))
return files
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
# Code generated by Microsoft (R) Python Code Generator.
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
# --------------------------------------------------------------------------
from io import IOBase
import sys
from typing import Any, Callable, Dict, Optional, TypeVar, Union, overload

Expand All @@ -22,8 +21,8 @@
from azure.core.rest import AsyncHttpResponse, HttpRequest
from azure.core.tracing.decorator_async import distributed_trace_async

from ... import _model_base, models as _models
from ..._vendor import handle_multipart_form_data_model, multipart_data, multipart_file
from ... import models as _models
from ..._vendor import handle_multipart_form_data_body
from ...operations._operations import (
build_form_data_basic_request,
build_form_data_binary_array_parts_request,
Expand Down Expand Up @@ -126,15 +125,10 @@ async def basic( # pylint: disable=inconsistent-return-statements

cls: ClsType[None] = kwargs.pop("cls", None)

if isinstance(body, _model_base.Model):
_body = handle_multipart_form_data_model(body)
else:
_body = body
_files = {k: multipart_file(v) for k, v in _body.items() if isinstance(v, (IOBase, bytes))}
_data = {k: multipart_data(v) for k, v in _body.items() if not isinstance(v, (IOBase, bytes))}
file_properties = ["profileImage"]
_files = handle_multipart_form_data_body(body, file_properties)

_request = build_form_data_basic_request(
data=_data,
files=_files,
headers=_headers,
params=_params,
Expand Down Expand Up @@ -246,15 +240,10 @@ async def complex( # pylint: disable=inconsistent-return-statements

cls: ClsType[None] = kwargs.pop("cls", None)

if isinstance(body, _model_base.Model):
_body = handle_multipart_form_data_model(body)
else:
_body = body
_files = {k: multipart_file(v) for k, v in _body.items() if isinstance(v, (IOBase, bytes))}
_data = {k: multipart_data(v) for k, v in _body.items() if not isinstance(v, (IOBase, bytes))}
file_properties = ["profileImage", "pictures"]
_files = handle_multipart_form_data_body(body, file_properties)

_request = build_form_data_complex_request(
data=_data,
files=_files,
headers=_headers,
params=_params,
Expand Down Expand Up @@ -348,15 +337,10 @@ async def json_part( # pylint: disable=inconsistent-return-statements

cls: ClsType[None] = kwargs.pop("cls", None)

if isinstance(body, _model_base.Model):
_body = handle_multipart_form_data_model(body)
else:
_body = body
_files = {k: multipart_file(v) for k, v in _body.items() if isinstance(v, (IOBase, bytes))}
_data = {k: multipart_data(v) for k, v in _body.items() if not isinstance(v, (IOBase, bytes))}
file_properties = ["profileImage"]
_files = handle_multipart_form_data_body(body, file_properties)

_request = build_form_data_json_part_request(
data=_data,
files=_files,
headers=_headers,
params=_params,
Expand Down Expand Up @@ -452,15 +436,10 @@ async def binary_array_parts( # pylint: disable=inconsistent-return-statements

cls: ClsType[None] = kwargs.pop("cls", None)

if isinstance(body, _model_base.Model):
_body = handle_multipart_form_data_model(body)
else:
_body = body
_files = {k: multipart_file(v) for k, v in _body.items() if isinstance(v, (IOBase, bytes))}
_data = {k: multipart_data(v) for k, v in _body.items() if not isinstance(v, (IOBase, bytes))}
file_properties = ["pictures"]
_files = handle_multipart_form_data_body(body, file_properties)

_request = build_form_data_binary_array_parts_request(
data=_data,
files=_files,
headers=_headers,
params=_params,
Expand Down Expand Up @@ -560,15 +539,10 @@ async def json_array_parts( # pylint: disable=inconsistent-return-statements

cls: ClsType[None] = kwargs.pop("cls", None)

if isinstance(body, _model_base.Model):
_body = handle_multipart_form_data_model(body)
else:
_body = body
_files = {k: multipart_file(v) for k, v in _body.items() if isinstance(v, (IOBase, bytes))}
_data = {k: multipart_data(v) for k, v in _body.items() if not isinstance(v, (IOBase, bytes))}
file_properties = ["profileImage"]
_files = handle_multipart_form_data_body(body, file_properties)

_request = build_form_data_json_array_parts_request(
data=_data,
files=_files,
headers=_headers,
params=_params,
Expand Down Expand Up @@ -660,15 +634,10 @@ async def multi_binary_parts( # pylint: disable=inconsistent-return-statements

cls: ClsType[None] = kwargs.pop("cls", None)

if isinstance(body, _model_base.Model):
_body = handle_multipart_form_data_model(body)
else:
_body = body
_files = {k: multipart_file(v) for k, v in _body.items() if isinstance(v, (IOBase, bytes))}
_data = {k: multipart_data(v) for k, v in _body.items() if not isinstance(v, (IOBase, bytes))}
file_properties = ["profileImage", "picture"]
_files = handle_multipart_form_data_body(body, file_properties)

_request = build_form_data_multi_binary_parts_request(
data=_data,
files=_files,
headers=_headers,
params=_params,
Expand Down
Loading
Loading