Skip to content

Conversation

@mohamedzeidan2021
Copy link
Collaborator

What's changing and why?

  • Enhanced the CLI's error handling with k8s authentication that addresses confusing raw API error messages that users encounter when authentication fails.
  • Users currently see confusing 401 Unauthorized errors with raw k8s responses instead of actionable guidance.

Changes

  • CLI checks k8s connectivity before executing commands that require it
  • raw API errors are replaced with user friendly messages
  • Error messages guide users to HPCLI commands

Before/After UX

Before:

$ hyp get-operator-logs hyp-jumpstart-endpoint --since-hours 1
Failed to verify Kubernetes version compatibility: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '37bb5f8d-9beb-4f1d-ad71-f6cfc7fcddd6', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 16 Sep 2025 18:49:30 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

After:

$ hyp get-operator-logs hyp-jumpstart-endpoint --since-hours 1
❌ Kubernetes authentication failed (401 Unauthorized).
Current context: my-hyperpod-cluster

This usually means your credentials have expired or are invalid.

To fix this:
1. Reconfigure cluster access:
   hyp set-cluster-context <cluster-name> --region <region>
2. Try your HyperPod command again

💡 This will refresh your authentication with the cluster.

Other cases covered

No Kubeconfig

❌ Kubernetes configuration not found.
No kubeconfig file found. Please ensure you have:
1. A valid kubeconfig file at ~/.kube/config, or
2. Set the KUBECONFIG environment variable

To configure cluster access:
  hyp set-cluster-context <cluster-name> --region <region>

💡 This will set up the necessary Kubernetes configuration for your HyperPod cluster.

AWS Credentials Issue

❌ Kubernetes authentication failed (401 Unauthorized).
Current context: my-hyperpod-cluster

🔍 AWS credentials issue detected:
Your AWS credentials appear to be missing or invalid.

To fix this:
1. Check your AWS credentials:
   aws sts get-caller-identity
2. If expired, refresh your AWS credentials
3. Then reconfigure cluster access:
   hyp set-cluster-context <cluster-name> --region <region>

💡 Make sure your AWS credentials have the necessary EKS permissions.

RBAC Permissions Issue

❌ Kubernetes access denied (403 Forbidden).
You don't have permission to access this cluster or resource.

This could mean:
1. Your user/role lacks the necessary RBAC permissions
2. You're connected to the wrong cluster
3. The cluster's access policies have changed

To fix this:
1. Verify you're using the correct cluster context
2. Contact your cluster administrator for access
3. Ensure your AWS role has the necessary EKS permissions

How was this change tested?

Added unit tests, and tested most of the cases on my local machine with expired aws credentials and invalid cluster contexts.

Are unit tests added?

Yes, 30+ tests for k8s connectivity, aws credentials check, checking current k8s context, end-to-end decorator functionality.

Are integration tests added?

No

Reviewer Guidelines

‼️ Merge Requirements: PRs with failing integration tests cannot be merged without justification.

One of the following must be true:

  • All automated PR checks pass
  • Failed tests include local run results/screenshots proving they work
  • Changes are documentation-only

Mohamed Zeidan and others added 9 commits September 3, 2025 13:15
Co-authored-by: Mohamed Zeidan <[email protected]>
**Description**
- Updated README.md to fix broken internal navigation links, corrected SDK import paths, added proper syntax highlighting to code blocks.
- Fixed training SDK imports, observability utils import path, and cluster management workflow examples.

**Testing Done**
- Verified all anchor links work correctly in table of contents and usage sections
- Cross-referenced SDK imports against actual source code in src/sagemaker/hyperpod/
- Validated CLI commands match implementation in hyp_cli.py
- Confirmed code examples use correct class names and method signatures
…ws#246)

* Draft of inference logger bug fix

* Draft fix of inference logger for SDK

* Revert adding --debug flag

* Add debug parameter to failing unit tests

* Fix create_from_dict to not have hardcoded debug flag
@mohamedzeidan2021 mohamedzeidan2021 changed the title Fix/clustererrorhandling Fix: Cluster CLI Error Handling Sep 17, 2025
@mohamedzeidan2021 mohamedzeidan2021 changed the title Fix: Cluster CLI Error Handling Fix: Cluster cmds Error Handling Sep 17, 2025
@mohamedzeidan2021 mohamedzeidan2021 changed the title Fix: Cluster cmds Error Handling Fix: k8s auth Error Handling Sep 17, 2025
mohamedzeidan2021 pushed a commit that referenced this pull request Nov 21, 2025
jam-jee pushed a commit that referenced this pull request Nov 21, 2025
* feat: Implement elastic training cli arguments (#273)

* feat: Implement elastic training cli arguments

* Add elastic training unified config and unit test

* Add graceful shutdown and scaling timeout to cli args

* Revert "feat: Implement elastic training cli arguments (#273)"

This reverts commit 18428ef2b1c0562bf51a9a4b4aa2914eed441259.

* Add dev_space_constants.py (#255)

Co-authored-by: Brian Xia <[email protected]>

* Add dev_space_access_constants.py (#256)

Co-authored-by: Brian Xia <[email protected]>

* Add space_admin_config_constants.py (#257)

Co-authored-by: Brian Xia <[email protected]>

* Add template package only (#261)

Co-authored-by: Brian Xia <[email protected]>

* Add dev_space.py CLI command (#263)

* Add dev_space.py CLI command

* Add dev space unit tests

---------

Co-authored-by: Brian Xia <[email protected]>

* Add dev_space_utils.py to work with the dev space template model (#262)

* Add dev_space_utils.py

* Add unit tests for dev_space_utils

---------

Co-authored-by: Brian Xia <[email protected]>

* Add dev space CLI (#269)

* Rename dev space to space (#272)

* Update the Space model and constants per latest operator (#275)

* Add space_admin_config.py CLI command (#260)

* Add space_admin_config.py CLI command

* Update the space admin config to space template

---------

Co-authored-by: Brian Xia <[email protected]>

* Implement CRUD operations for Space PySDK (#267)

* Implement CRUD operations for Space PySDK

* Update Space PySDK per new schema

* Update Space PySDK per new schema

* Implement the pySDK for the Space Template (#282)

* Refactor Space CLI using the Space PySDK (#281)

* Implement CRUD operations for Space PySDK

* Update Space PySDK per new schema

* Refactor CLI to use the PySDK

* Add dev_space_access.py CLI command (#259)

* Add dev_space_access.py CLI command

* Add space access creation to pySDK and refactor space access CLI

---------

Co-authored-by: Brian Xia <[email protected]>

* Listing space will filter out the spaces not created by the current user (#285)

* Implement CRUD operations for Space PySDK

* Update Space PySDK per new schema

* Implement CRUD operations for Space PySDK

* Update Space PySDK per new schema

* Update Space PySDK per new schema

* Implement space list pagination and creator filtering

* Refactor space template with PySDK (#286)

* Add additional Space parameters for resources including the fractional GPU (#287)

* Implement validation for mig profiles for Spaces (#291)

* Implement validation for mig profiles when creating/updating spaces

* Update Space parameter model

* Make Space Template namespaced resource

* Parker GA issues (#296)

* Update Space Template CLI to be namespaced

* Space get-logs default to the workspace container

* Remove error handling to bubble up the actual K8s errors

* Listing public Spaces

* Fix typos, elaborated text, add logic to parse idle-shutdown

* Fix the template ref regression (#300)

* Update SageMaker Space documentation (#301)

* Implement Space integration tests (#298)

Inference tests succeeded with parker-cli code - https://quip-amazon.com/fhwhAAMht0Mm/Project-Parker-HyperPod-User-Experience-for-Data-Scientist-persona

Parker-cli integ tests pass (shown below)

These inference tests failing are known to be flaky- https://w.amazon.com/bin/view/AWS/AmazonAI/Platform/Codex/CodexInfra/Runbooks/HyperPodCLI/TroubleshootInferenceTests#HTroubleshooting
ticket has been created to fix these flaky tests - https://t.corp.amazon.com/V1943878058


Parker-cli integ tests passing

============================= test session starts ==============================
platform linux -- Python 3.11.14, pytest-8.3.2, pluggy-1.6.0 -- /root/.pyenv/versions/3.11.14/bin/python3.11
cachedir: .pytest_cache
rootdir: /codebuild/output/src1458832038/src/github.com/aws/private-sagemaker-hyperpod-cli-staging
configfile: setup.cfg
plugins: hydra-core-1.3.2, order-1.3.0, dependency-0.6.0, cov-5.0.0
collecting ... collected 39 items
test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_create PASSED [  2%]
test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_list_table PASSED [  5%]
test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_list_json PASSED [  7%]
test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_describe_yaml PASSED [ 10%]
test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_describe_json PASSED [ 12%]
test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_stop PASSED [ 15%]
test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_start PASSED [ 17%]
test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_update PASSED [ 20%]
test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_get_logs PASSED [ 23%]
test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_delete PASSED [ 25%]
test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_list_empty_namespace PASSED [ 28%]
test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_describe_nonexistent PASSED [ 30%]
test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_delete_nonexistent PASSED [ 33%]
test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_create PASSED [ 35%]
test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_list_table PASSED [ 38%]
test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_list_json PASSED [ 41%]
test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_describe_yaml PASSED [ 43%]
test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_describe_json PASSED [ 46%]
test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_update PASSED [ 48%]
test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_delete PASSED [ 51%]
test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_list_empty_namespace PASSED [ 53%]
test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_describe_nonexistent PASSED [ 56%]
test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_delete_nonexistent PASSED [ 58%]
test/integration_tests/space/sdk/test_sdk_space.py::test_create_space PASSED [ 61%]
test/integration_tests/space/sdk/test_sdk_space.py::test_list_spaces PASSED [ 64%]
test/integration_tests/space/sdk/test_sdk_space.py::test_get_space PASSED [ 66%]
test/integration_tests/space/sdk/test_sdk_space.py::test_wait_until_running PASSED [ 69%]
test/integration_tests/space/sdk/test_sdk_space.py::test_update_space PASSED [ 71%]
test/integration_tests/space/sdk/test_sdk_space.py::test_stop_space PASSED [ 74%]
test/integration_tests/space/sdk/test_sdk_space.py::test_start_space PASSED [ 76%]
test/integration_tests/space/sdk/test_sdk_space.py::test_list_pods PASSED [ 79%]
test/integration_tests/space/sdk/test_sdk_space.py::test_get_logs PASSED [ 82%]
test/integration_tests/space/sdk/test_sdk_space.py::test_create_space_access SKIPPED [ 84%]
test/integration_tests/space/sdk/test_sdk_space.py::test_delete_space PASSED [ 87%]
test/integration_tests/space/sdk/test_sdk_space_template.py::TestHPSpaceTemplate::test_create_template PASSED [ 89%]
test/integration_tests/space/sdk/test_sdk_space_template.py::TestHPSpaceTemplate::test_list_templates PASSED [ 92%]
test/integration_tests/space/sdk/test_sdk_space_template.py::TestHPSpaceTemplate::test_get_template PASSED [ 94%]
test/integration_tests/space/sdk/test_sdk_space_template.py::TestHPSpaceTemplate::test_update_template PASSED [ 97%]
test/integration_tests/space/sdk/test_sdk_space_template.py::TestHPSpaceTemplate::test_delete_template PASSED [100%]
=============================== warnings summary ===============================

* merge conflicts fixed

* Update README for fractional gpu support (#294)

* Update README for fractional gpu support

* update pytorch job example

* add example for accelerator partitions

* merge conflicts from js template and inference

* update changelog

* uncommented install req

* uncommented

* fixed uncomment

---------

Co-authored-by: Sophia <[email protected]>
Co-authored-by: Molly He <[email protected]>
Co-authored-by: Brian Xia <[email protected]>
Co-authored-by: Brian Xia <[email protected]>
Co-authored-by: Mohamed Zeidan <[email protected]>
Co-authored-by: Ophelia Yang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants