Skip to content

Conversation

@MonkeyCanCode
Copy link
Contributor

@MonkeyCanCode MonkeyCanCode commented Aug 6, 2024

Description

Add a mini k8s deployment for running polaris.

Fixes # [DOC] Directions to run mini-deployment reference k8 directory that does not exist

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Here are all of the resources that got deployed with this deployment:

kubectl get all -n polaris
NAME                                      READY   STATUS    RESTARTS   AGE
pod/polaris-deployment-5b8457d65d-dhp5r   1/1     Running   0          108s

NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/polaris-service   ClusterIP   10.96.173.228   <none>        8181/TCP   108s

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/polaris-deployment   1/1     1            1           108s

NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/polaris-deployment-5b8457d65d   1         1         1       108s

On one session, performs a port forwarding for the svc:

kubectl port-forward svc/polaris-service -n polaris 8181:8181
Forwarding from 127.0.0.1:8181 -> 8181
Forwarding from [::1]:8181 -> 8181

On a new session, get the auth token from pod (in-memory deployment) then use polaris cli to perform auth:

kubectl  logs pods/polaris-deployment-5b8457d65d-dhp5r -n polaris | grep 'realm: default-realm root principal credentials:'
realm: default-realm root principal credentials: 2d54798a730ef589:b5b6ac216c5441e7421f241673837751
...
./polaris --client-id 2d54798a730ef589 --client-secret b5b6ac216c5441e7421f241673837751 principals list
{"name": "root", "clientId": "2d54798a730ef589", "properties": {}, "createTimestamp": 1723000008560, "lastUpdateTimestamp": 1723000008560, "entityVersion": 1}

Checklist:

Please delete options that are not relevant.

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules
  • If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue
  • I have signed and submitted the ICLA and if needed, the CCLA. See Contributing for details.

@flyrain
Copy link
Contributor

flyrain commented Aug 7, 2024

Thanks @MonkeyCanCode for working on this. We will need updates on the doc(README.md) as well. This part isn't consistent with the behavior:
Screenshot 2024-08-06 at 6 13 28 PM

We will need to add -n polaris for these commands, as well as to port forwarding command. I feel like these instructions are not necessarily in README, they can be in a separated doc, which is linked by the README, as we want to keep README lean.

@MonkeyCanCode
Copy link
Contributor Author

Thanks @MonkeyCanCode for working on this. We will need updates on the doc(README.md) as well. This part isn't consistent with the behavior:
Screenshot 2024-08-06 at 6 13 28 PM

We will need to add -n polaris for these commands, as well as to port forwarding command. I feel like these instructions are not necessarily in README, they can be in a separated doc, which is linked by the README, as we want to keep README lean.

Got it. Will update it tonight

@MonkeyCanCode
Copy link
Contributor Author

Thanks @MonkeyCanCode for working on this. We will need updates on the doc(README.md) as well. This part isn't consistent with the behavior:
Screenshot 2024-08-06 at 6 13 28 PM
We will need to add -n polaris for these commands, as well as to port forwarding command. I feel like these instructions are not necessarily in README, they can be in a separated doc, which is linked by the README, as we want to keep README lean.

Got it. Will update it tonight

This is completed and please take another look.

README.md Outdated
- `kubectl get pods -n polaris` - To check the status of the pods.
- `kubectl get deployment -n polaris` - To check the status of the deployment.
- `kubectl describe deployment polaris-deployment -n polaris` - To troubleshoot if things aren't working as expected.
- `kubectl port-forward svc/polaris-service -n polaris 8181:8181` - To create a secure connection between a local machine and a pod within the cluster.
Copy link
Contributor

@flyrain flyrain Aug 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should move this line right after line 59, as it is a part of k8s local deployment, or even better if we can move it inside setup.sh.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am assuming setup.sh will only be run once (or rerun if an end-user wants to rebuild the setup, does it makes more sense to keep the port forwarding command outside the setup.sh so people can grab it as needed? the existed setup.sh will blow away the services every time when it got ran which also means change of auth token if an user accidently exit from the port forwarding session and not sure how to perform port forwarding without using the setup.sh)

Copy link
Contributor

@flyrain flyrain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@flyrain flyrain merged commit 7ce0945 into apache:main Aug 7, 2024
@flyrain
Copy link
Contributor

flyrain commented Aug 7, 2024

Thanks @MonkeyCanCode for the PR.

@MonkeyCanCode MonkeyCanCode deleted the mini-k8s-deployment branch August 25, 2024 16:58
snazy added a commit to snazy/polaris that referenced this pull request Nov 20, 2025
* Helm Chart: remove duplicate line in authenticationOptions template (apache#2297)

* Move python client Makefile into the root level one (apache#2140)

* Move python client Makefile into the root level one

* Update workflow

* add client-lint to pre-commit

* Update README.md to include client

* fix(deps): update dependency boto3 to v1.40.5 (apache#2300)

* fix(deps): update dependency org.assertj:assertj-core to v3.27.4 (apache#2298)

* Fix python CI client-integration-test (apache#2305)

`CONTAINER_TOOL` got renamed to `DOCKER` in
6764a88
but then
eda7074
added more `CONTAINER_TOOL` usage

* fix(docs): update BasePolarisAuthenticator to DefaultAuthenticator (apache#2303)

* Fix Hadoop federation to initialize the configuration prior to catalog initialization (apache#2282)

This PR addresses the gap in Hadoop federation to ensure that we initialize the hadoop configuration prior to initializing the catalog object.

The iceberg library expects the hadoop configuration to be initialized before creating a HadoopCatalog object. This change ensures that Polaris is compatible with the underlying iceberg library. Additionally, since the config initialization is based on the underlying (default) core-site.xml file, the change ensures that the federated catalog was created using `IMPLICIT` authentication mode. 

Testing: 
Due to current limitations in the current test setup, tested the change manually.
[TODO] Add a regtest with hadoop federation once the change is baked into the apache/polaris docker image.

* fix(deps): update dependency boto3 to v1.40.6 (apache#2308)

* fix(deps): update dependency io.opentelemetry:opentelemetry-bom to v1.53.0 (apache#2309)

* Perform force repair when polaris not found (apache#2313)

* fix(deps): update dependency com.github.spotbugs:spotbugs-annotations to v4.9.4 (apache#2312)

* chore(deps): update dependency pre-commit to v4.3.0 (apache#2311)

* fix(deps): update dependency software.amazon.awssdk:bom to v2.32.19 (apache#2314)

* fix(deps): update dependency com.gradleup.shadow:shadow-gradle-plugin to v9.0.1 (apache#2310)

* Separate Cloud Integration Tests (apache#2283)

Currently, cloud integration tests are part of the regular test suite
but require cloud credentials to execute properly. This creates
unnecessary overhead for developers who just want to run local builds
or for CI jobs that don't have cloud access configured.

* chore(deps): update actions/checkout action to v5 (apache#2319)

* CatalogEntity: internal endpoint not considered (apache#2292)

Setting an S3 internal endpoint doesn't work, because the property's not carried over from the OpenAPI model type into `AwsStorageConfigInfo`.

* client/python: loosen boto3 dep (apache#2188)

It's hard to depend on this package when the boto3 dependency is so strict.

* fix(deps): update dependency io.micrometer:micrometer-bom to v1.15.3 (apache#2321)

* NoSQL: test adjustment

* Last merged commit 45e8e02

---------

Co-authored-by: Alexandre Dutra <[email protected]>
Co-authored-by: Yong Zheng <[email protected]>
Co-authored-by: Mend Renovate <[email protected]>
Co-authored-by: Christopher Lambert <[email protected]>
Co-authored-by: Yujiang Zhong <[email protected]>
Co-authored-by: Pooja Nilangekar <[email protected]>
Co-authored-by: Tamas Mate <[email protected]>
Co-authored-by: Tyler Rockwood <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants