Skip to content

Commit 53a3f2d

Browse files
mohamedzeidan2021mollyheamazonjam-jeersareddy0329Roja Reddy Sareddy
authored
Master model customization to keynote 3 v3 (#1925)
* sagemaker-core update, port sft-trainer and ai registry (#1895) * sagemaker core update with MC service.json, SFT notebook flow succeeded before modelpackage * remove workaround, modelpackage issue persists * remove ai registry port * feat: AIRegistry SDK Implementation (#1893) * remove sft trainer port, fix ModelPackage issue (#1901) * Add/port finetuning interfaces to v3 (#1906) * Add/port finetuning interfaces to v3 * Add/port finetuning interfaces to v3 --------- Co-authored-by: Roja Reddy Sareddy <[email protected]> * ported evaluation from v2->v3. pending example notebooks results (#1902) * v2->v3 porting for eval * bring back deleted needed files * unit tests --------- Co-authored-by: Mohamed Zeidan <[email protected]> * Mc deployment (#1909) * Updates for Model Customization * Model Builder Base change * Fix * Support for Model Customization * New Model Builder changes with Model Customization * Fix * Add base_trainer, addition to PR#1906 (#1910) * Add/port finetuning interfaces to v3 * Add/port finetuning interfaces to v3 * Add base_trainer, addition to PR#1906 --------- Co-authored-by: Roja Reddy Sareddy <[email protected]> * AIRegistry Integration Tests (#1908) * feat: Trainer wait with MLFlow metrics (#1907) * feat: update eval integs (#1912) Co-authored-by: Mufaddal Rohawala <[email protected]> * Update get_all method for Evaluator and Dataset (#1913) * Update get_all method for Evaluator and Dataset - Update Dataset.get_all() and added refresh() - Update Evaluator.get_all() and added refresh() - Added more tags in condition when importing hub content * Update dataset, evaluators and examples * remove hub_content_name * port bedrock model builder and add integ test (#1911) * port bedrock model builder and add integ test * Add unit test and notebook, port new version of bedrock model builder * add docstring * Mc deployment integ tests and example notebook (#1914) * Updates for Model Customization * Model Builder Base change * Fix * Support for Model Customization * New Model Builder changes with Model Customization * Fix * Deployment Integ tests * Tests * Deployment NOtebook * fix * Update model hub name and encoding method (#1918) * Update get_all method for Evaluator and Dataset - Update Dataset.get_all() and added refresh() - Update Evaluator.get_all() and added refresh() - Added more tags in condition when importing hub content * Update dataset, evaluators and examples * remove hub_content_name * Update hub name and hash - Update model hub name to be AIRegistry-{accountId}-{region} - Update hub name hash * Add telemetry for model customization (#1920) * add telemetry to model customization * add telemetry to model customization 2 * small fix * INteg tests for nova bedrock (#1926) * Bedrock Nova * Unit tests for bedrock nova * accept eula always (#1928) Co-authored-by: Mohamed Zeidan <[email protected]> * Add studio tags to trainer (#1927) * Update get_all method for Evaluator and Dataset - Update Dataset.get_all() and added refresh() - Update Evaluator.get_all() and added refresh() - Added more tags in condition when importing hub content * Update dataset, evaluators and examples * remove hub_content_name * Update hub name and hash - Update model hub name to be AIRegistry-{accountId}-{region} - Update hub name hash * Add studio tags to trainer Tested nu checking training job creation inputs and unit tests --------- Co-authored-by: rsareddy0329 <[email protected]> * Finetuning classes: add accept_eula support & Add integ tests for sft, rlvr,rlaif, dpo trainers (#1915) * Add/port finetuning interfaces to v3 * Add/port finetuning interfaces to v3 * Add base_trainer, addition to PR#1906 * Add integ tests for sft, rlvr,rlaif, dpo trainers * Add integ tests for sft, rlvr,rlaif, dpo trainers * Finetuning classes: Add accept_eula support * Finetuning classes: Add accept_eula support --------- Co-authored-by: Roja Reddy Sareddy <[email protected]> * feat: pipeline name fix + mlflow arn fix + jinja dep (#1931) * changes for pipeline naming * feat: pipeline name fix + mlflow arn fix + jinja dep --------- Co-authored-by: Mufaddal Rohawala <[email protected]> * AIRegistry: Add Support to user provided session and role (#1932) * Add/port finetuning interfaces to v3 * Add/port finetuning interfaces to v3 * Add base_trainer, addition to PR#1906 * Add integ tests for sft, rlvr,rlaif, dpo trainers * Add integ tests for sft, rlvr,rlaif, dpo trainers * Finetuning classes: Add accept_eula support * Finetuning classes: Add accept_eula support * AIRegistry: Support user provided session and role * AIRegistry: Support user provided session and role * AIRegistry: Support user provided session and role --------- Co-authored-by: Roja Reddy Sareddy <[email protected]> * format name for ai reg, stripped name of =, added domain-id tags for … (#1930) * format name for ai reg, stripped name of =, added domain-id tags for datasets and evaluators, prefix changed t use full path including filename * pulling metadata path since get_current_domain_id didnt work * rmvd gamma endpoint --------- Co-authored-by: Mohamed Zeidan <[email protected]> * fixed train unit tests (#1933) Co-authored-by: Mohamed Zeidan <[email protected]> * Updating DataSet versioning pattern for parity with UI (#1934) * fix dataset arn configuration in eval (#1935) Co-authored-by: Mufaddal Rohawala <[email protected]> * Master model customization v3 (#1936) * Updating DataSet versioning pattern for parity with UI * Updating default S3 bucket and prefix for datasets * fix: custom metrics and pipeline name in lineage (#1937) * remove mlflow config for base model eval only * fix unit * fix custom metrics and pipeline name in lineage --------- Testing: 1. Added/Ran units. 2. Tested LLMAJ eval. Co-authored-by: Mufaddal Rohawala <[email protected]> * Integ tests for ModelBuilder (#1939) * Bedrock Nova * Unit tests for bedrock nova * Integ test updates * Test case updates * fixing AIRegistry integration and unit tests (#1941) * Update error messages based on feedback (#1938) * feat: Inline MLFlow metrics with Serverless MLFlow App (#1942) * Master model customization v3 (#1945) * Bedrock Nova * Unit tests for bedrock nova * Integ test updates * Test case updates * Bedrock tests * fix: update eval integ tests (#1946) * update eval integ tests * fix unit tests --------- Co-authored-by: Mufaddal Rohawala <[email protected]> * Finetuning classes: Add integ tests (#1947) * Add/port finetuning interfaces to v3 * Add/port finetuning interfaces to v3 * Add base_trainer, addition to PR#1906 * Add integ tests for sft, rlvr,rlaif, dpo trainers * Add integ tests for sft, rlvr,rlaif, dpo trainers * Finetuning classes: Add accept_eula support * Finetuning classes: Add accept_eula support * AIRegistry: Support user provided session and role * AIRegistry: Support user provided session and role * AIRegistry: Support user provided session and role * Finetuning classes: Add integration tests * Merge conflicts * Merge conflicts --------- Co-authored-by: Roja Reddy Sareddy <[email protected]> * Workflow change to enable github actions (#1948) * workflow change and test * remove test * Remove beta endpoint info, add region validation and tests (#1950) * Add/port finetuning interfaces to v3 * Add/port finetuning interfaces to v3 * Add base_trainer, addition to PR#1906 * Add integ tests for sft, rlvr,rlaif, dpo trainers * Add integ tests for sft, rlvr,rlaif, dpo trainers * Finetuning classes: Add accept_eula support * Finetuning classes: Add accept_eula support * AIRegistry: Support user provided session and role * AIRegistry: Support user provided session and role * AIRegistry: Support user provided session and role * Finetuning classes: Add integration tests * Merge conflicts * Merge conflicts * Remove beta endpoint details in sagemaker-core * add region validation for models --------- Co-authored-by: Roja Reddy Sareddy <[email protected]> --------- Co-authored-by: Molly He <[email protected]> Co-authored-by: jam-jee <[email protected]> Co-authored-by: rsareddy0329 <[email protected]> Co-authored-by: Roja Reddy Sareddy <[email protected]> Co-authored-by: Mohamed Zeidan <[email protected]> Co-authored-by: Gokul Anantha Narayanan <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: Zhaoqi <[email protected]>
1 parent cc4d91b commit 53a3f2d

File tree

292 files changed

+124785
-28935
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

292 files changed

+124785
-28935
lines changed

‎.github/workflows/submodule-codebuild-ci.yml‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,12 @@ name: Sagemaker PR Checks
22
on:
33
pull_request_target:
44
branches:
5-
- "master-v3"
5+
- "master"
66
paths:
77
- 'sagemaker-train/**'
88
- 'sagemaker-serve/**'
99
- 'sagemaker-mlops/**'
10+
- 'sagemaker-core/**'
1011

1112
concurrency:
1213
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.head_ref }}
@@ -93,6 +94,13 @@ jobs:
9394
SUBMODULES=$(echo $SUBMODULES | sed 's/\]$/,"sagemaker-mlops"\]/')
9495
fi
9596
fi
97+
if echo "$CHANGES" | grep -q "^sagemaker-core/"; then
98+
if [ "$SUBMODULES" = '[]' ]; then
99+
SUBMODULES='["sagemaker-core"]'
100+
else
101+
SUBMODULES=$(echo $SUBMODULES | sed 's/\]$/,"sagemaker-core"\]/')
102+
fi
103+
fi
96104
echo "Final SUBMODULES: $SUBMODULES"
97105
echo "submodules=$SUBMODULES" >> $GITHUB_OUTPUT
98106

‎sagemaker-core/resource_plan.csv‎

Lines changed: 49 additions & 14 deletions
Large diffs are not rendered by default.

‎sagemaker-core/sample/sagemaker/2017-07-24/service-2.json‎

Lines changed: 14973 additions & 534 deletions
Large diffs are not rendered by default.

‎sagemaker-core/src/sagemaker/core/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,6 @@
1111
FrameworkProcessor,
1212
)
1313
from sagemaker.core.transformer import Transformer # noqa: F401
14+
1415
# Note: HyperparameterTuner and WarmStartTypes are in sagemaker.train.tuner
1516
# They are not re-exported from core to avoid circular dependencies

‎sagemaker-core/src/sagemaker/core/_studio.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,4 +113,4 @@ def _parse_tags(config):
113113
{"Key": "sagemaker:project-name", "Value": config["sagemakerProjectName"]},
114114
]
115115
except Exception as e: # pylint: disable=W0703
116-
logger.debug("Could not parse project config. %s", e)
116+
logger.debug("Could not parse project config. %s", e)

‎sagemaker-core/src/sagemaker/core/apiutils/_base_types.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,4 +225,4 @@ def _invoke_api(self, boto_method, boto_method_members):
225225
api_kwargs = self.to_boto(api_values)
226226
api_method = getattr(self.sagemaker_session.sagemaker_client, boto_method)
227227
api_boto_response = api_method(**api_kwargs)
228-
return self.with_boto(api_boto_response)
228+
return self.with_boto(api_boto_response)

‎sagemaker-core/src/sagemaker/core/apiutils/_boto_functions.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ def to_pascal_case(snake_case):
3737
Returns:
3838
str: String converted to PascalCase.
3939
"""
40-
return ''.join(word.capitalize() for word in snake_case.split('_'))
40+
return "".join(word.capitalize() for word in snake_case.split("_"))
4141

4242

4343
def to_snake_case(name):
@@ -127,4 +127,4 @@ def to_boto(member_vars, member_name_to_boto_name, member_name_to_type):
127127
else:
128128
boto_value = api_type.to_boto(member_value) if api_type else member_value
129129
to_boto_values[boto_name] = boto_value
130-
return to_boto_values
130+
return to_boto_values

‎sagemaker-core/src/sagemaker/core/base_deserializers.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,5 +31,5 @@
3131
"Importing from sagemaker.core.base_deserializers is deprecated. "
3232
"Use sagemaker.core.deserializers instead.",
3333
DeprecationWarning,
34-
stacklevel=2
34+
stacklevel=2,
3535
)

‎sagemaker-core/src/sagemaker/core/base_serializers.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,5 +31,5 @@
3131
"Importing from sagemaker.core.base_serializers is deprecated. "
3232
"Use sagemaker.core.serializers instead.",
3333
DeprecationWarning,
34-
stacklevel=2
34+
stacklevel=2,
3535
)

‎sagemaker-core/src/sagemaker/core/clarify/__init__.py‎

Lines changed: 7 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -2004,7 +2004,7 @@ def _run(
20042004
kms_key,
20052005
)
20062006
from sagemaker.core.shapes import ProcessingS3Input, ProcessingS3Output
2007-
2007+
20082008
config_input = ProcessingInput(
20092009
input_name="analysis_config",
20102010
s3_input=ProcessingS3Input(
@@ -2013,7 +2013,7 @@ def _run(
20132013
s3_data_type="S3Prefix",
20142014
s3_input_mode="File",
20152015
s3_compression_type="None",
2016-
)
2016+
),
20172017
)
20182018
data_input = ProcessingInput(
20192019
input_name="dataset",
@@ -2024,15 +2024,15 @@ def _run(
20242024
s3_input_mode="File",
20252025
s3_data_distribution_type=data_config.s3_data_distribution_type,
20262026
s3_compression_type=data_config.s3_compression_type,
2027-
)
2027+
),
20282028
)
20292029
result_output = ProcessingOutput(
20302030
output_name="analysis_result",
20312031
s3_output=ProcessingS3Output(
20322032
s3_uri=data_config.s3_output_path,
20332033
local_path=self._CLARIFY_OUTPUT,
20342034
s3_upload_mode=ProcessingOutputHandler.get_s3_upload_mode(analysis_config),
2035-
)
2035+
),
20362036
)
20372037

20382038
return super().run(
@@ -2101,9 +2101,7 @@ def run_pre_training_bias(
21012101
data_config, data_bias_config, methods
21022102
)
21032103
# when name is either not provided (is None) or an empty string ("")
2104-
job_name = job_name or name_from_base(
2105-
self.job_name_prefix or "Clarify-Pretraining-Bias"
2106-
)
2104+
job_name = job_name or name_from_base(self.job_name_prefix or "Clarify-Pretraining-Bias")
21072105
return self._run(
21082106
data_config,
21092107
analysis_config,
@@ -2187,9 +2185,7 @@ def run_post_training_bias(
21872185
model_config,
21882186
)
21892187
# when name is either not provided (is None) or an empty string ("")
2190-
job_name = job_name or name_from_base(
2191-
self.job_name_prefix or "Clarify-Posttraining-Bias"
2192-
)
2188+
job_name = job_name or name_from_base(self.job_name_prefix or "Clarify-Posttraining-Bias")
21932189
return self._run(
21942190
data_config,
21952191
analysis_config,
@@ -2373,9 +2369,7 @@ def run_explainability(
23732369
data_config, model_config, model_scores, explainability_config
23742370
)
23752371
# when name is either not provided (is None) or an empty string ("")
2376-
job_name = job_name or name_from_base(
2377-
self.job_name_prefix or "Clarify-Explainability"
2378-
)
2372+
job_name = job_name or name_from_base(self.job_name_prefix or "Clarify-Explainability")
23792373
return self._run(
23802374
data_config,
23812375
analysis_config,

0 commit comments

Comments
 (0)