Skip to content

Conversation

@nesitor
Copy link
Member

@nesitor nesitor commented Jul 2, 2024

Problem: A user cannot initialize an already created confidential VM.

Solution: Implement VmConfidentialClient class to be able to initialize and interact with confidential VMs.

Solution: Implement `VmConfidentialClient` class to be able to initialize and interact with confidential VMs.
@nesitor nesitor requested review from 1yam, hoh, olethanh and philogicae July 2, 2024 19:56
@nesitor nesitor self-assigned this Jul 2, 2024
@github-actions
Copy link

github-actions bot commented Jul 2, 2024

Summary: The PR modifies a significant number of files across multiple directories. It includes a refactoring of the 'vmclient.py' file into 'vm_client.py', adds a new 'vm_confidential_client.py' file, and renames several other files. This indicates a high level of complexity and potential impact on the codebase.

Highlight: The diff shows the renaming of 'vmclient.py' to 'vm_client.py' and the addition of a new 'vm_confidential_client.py' file. This suggests a significant refactoring and new feature implementation.

diff  --git a/pyproject.toml b/pyproject.toml
index b52efe66..1070a7f7 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -32,6 +32,7  @@ dependencies = [
      "python-magic",
      "typer",
      "typing_extensions",
+     "aioresponses>=0.7.6"
  ]
 
  [project.optional-dependencies]
diff  --git a/src/aleph/sdk/client/vmclient.py b/src/aleph/sdk/client/vm_client.py
similarity index 100%
rename from src/aleph/sdk/client/vmclient.py
rename to src/aleph/sdk/client/vm_client.py
diff  --git a/src/aleph/sdk/client/vm_confidential_client.py b/src/aleph/sdk/client/vm_confidential_client.py
new file mode 100644
index 00000000..305ff8ef
--- /dev/null
+++ b/src/aleph/sdk/client/vm_confidential_client.py
@@ -0,0 +1,155 @@
+import json
+import logging
+import tempfile
+from pathlib import Path
+from typing import Any, Dict, Optional, Tuple
+
+import aiohttp
+from aleph_message.models import ItemHash
+
+from aleph.sdk.client.vm_client import VmClient
+from aleph.sdk.types import Account
+from aleph.sdk.utils import run_in_subprocess
+
+logger = logging.getLogger(__name__)
+
+
+class VmConfidentialClient(VmClient):
+    sevctl_path: Path
+
+    def __init__(
+        self,
+        account: Account,
+        sevctl_path: Path,
+        node_url: str = "",
+        session: Optional[aiohttp.ClientSession] = None,
+    ):
+        super().__init__(account, node_url, session)
+        self.sevctl_path = sevctl_path
+
+    async def get_certificates(self) -> Tuple[Optional[int], str]:
+        url = f"{self.node_url}/about/certificates"
+        try:
+            async with self.session.get(url) as response:
+                data = await response.read()
+                with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
+                    tmp_file.write(data)
+                    return response.status, tmp_file.name
+
+        except aiohttp.ClientError as e:
+            logger.error(
+                f"HTTP error getting node certificates on {self.node_url}: {str(e)}"
+            )
+            return None, str(e)
+
+    async def create_session(
+        self, vm_id: ItemHash, certificate_path: Path, policy: int
+    ):
+        args = [
+            "session",
+            "--name",
+            vm_id,
+            str(certificate_path),
+            str(policy),
+        ]
+        try:
+            await self.sevctl_cmd(args)
+        except Exception as e:
+            raise ValueError(f"Session creation have failed, reason: {str(e)}")
+
+    async def initialize(
+        self, vm_id: ItemHash, session: Path, godh: Path
+    ) -> Tuple[Optional[int], str]:
+        session_file = session.read_bytes()
+        godh_file = godh.read_bytes()
+        params = {
+            "session": session_file,
+            "godh": godh_file,
+        }
+        return await self.perform_confidential_operation(
+            vm_id, "confidential/initialize", params=params
+        )
+
+    async def measurement(self, vm_id: ItemHash) -> Tuple[Optional[int], str]:
+        status, text = await self.perform_confidential_operation(
+            vm_id, "confidential/measurement"
+        )
+        if status:
+            response = json.loads(text)
+            return status, response
+
+        return status, text
+
+    async def validate_measurement(self, vm_id: ItemHash) -> bool:
+        return True
+
+    async def build_secret(
+        self, tek_path: Path, tik_path: Path, measurement: str, secret: str
+    ) -> Tuple[Path, Path]:
+        current_path = Path().cwd()
+        secret_header_path = current_path / "secret_header.bin"
+        secret_payload_path = current_path / "secret_payload.bin"
+        args = [
+            "secret",
+            "build",
+            "--tik",
+            str(tik_path),
+            "--tek",
+            str(tek_path),
+            "--launch-measure-blob",
+            measurement,
+            "--secret",
+            secret,
+            str(secret_header_path),
+            str(secret_payload_path),
+        ]
+        try:
+            await self.sevctl_cmd(args)
+            return secret_header_path, secret_payload_path
+        except Exception as e:
+            raise ValueError(f"Secret building have failed, reason: {str(e)}")
+
+    async def inject_secret(
+        self, vm_id: ItemHash, packed_header: str, secret: str
+    ) -> Tuple[Optional[int], str]:
+        params = {
+            "packed_header": packed_header,
+            "secret": secret,
+        }
+        status, text = await self.perform_confidential_operation(
+            vm_id, "confidential/inject_secret", params=params
+        )
+
+        if status:
+            response = json.loads(text)
+            return status, response
+
+        return status, text
+
+    async def perform_confidential_operation(
+        self, vm_id: ItemHash, operation: str, params: Optional[Dict[str, Any]] = None
+    ) -> Tuple[Optional[int], str]:
+        if not self.pubkey_signature_header:
+            self.pubkey_signature_header = await self._generate_pubkey_signature_header()
+
+        url, header = await self._generate_header(vm_id=vm_id, operation=operation)
+
+        try:
+            async with self.session.post(url, headers=header, data=params) as response:
+                response_text = await response.text()
+                return response.status, response_text
+
+        except aiohttp.ClientError as e:
+            logger.error(f"HTTP error during operation {operation}: {str(e)}")
+            return None, str(e)
+
+    async def sevctl_cmd(self, *args) -> bytes:
+        return await run_in_subprocess(
+            ["sevctl", *args],
+            check=True,
+        )
diff --git a/src/aleph/sdk/utils.py b/src/aleph/sdk/utils.py
index 2d1b30c7..130edc38 100644
--- a/src/aleph/sdk/utils.py
+++ b/src/aleph/sdk/utils.py
@@ -1,8 +1,10 @@
 import errno
 import hashlib
 import json
 import logging
 import os
+import subprocess
 from datetime import date, datetime, time
 from enum import Enum
 from pathlib import Path
@@ -11,6 +13,7 @@ def sign_vm_control_payload(payload: Dict[str, str], ephemeral_key) -> str:
          }
      )
     return signed_operation
+
+
+async def run_in_subprocess(
+    command: List[str], check: bool = True, stdin_input: Optional[bytes] = None
+) -> bytes:
+    logger.debug(f"command: {'  '.join(command)}")
+
+    process = await asyncio.create_subprocess_exec(
+        *command,
+        stdin=asyncio.subprocess.PIPE,
+        stdout=asyncio.subprocess.PIPE,
+        stderr=asyncio.subprocess.PIPE,
+    )
+    stdout, stderr = await process.communicate(input=stdin_input)
+
+    if check and process.returncode:
+        logger.error(
+            f"Command failed with error code {process.returncode}:\n"
+            f"    stdin = {stdin_input}\n"
+            f"    command = {command}\n"
+            f"    stdout = {stderr}"
+        )
+        raise subprocess.CalledProcessError(
+            process.returncode, str(command), stderr.decode()
+        )
+
+    return stdout

The PR introduces significant changes to the codebase that require deep understanding of the project architecture. As such, a 'BLACK' rating is recommended.

@github-actions github-actions bot added the BLACK This PR has critical implications and must be reviewed by a senior engineer. label Jul 2, 2024
Copy link
Member

@1yam 1yam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

else looks good to me. @Psycojoker what do you thinks ?

@nesitor nesitor requested a review from 1yam July 3, 2024 17:31
Copy link
Member

@1yam 1yam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Else LGTM

olethanh added 2 commits July 4, 2024 15:09
Corrections:
 *  Measurement type returned was missing field needed for validation of measurements
 * Port number was not handled correctly in authentifaction
 * Adapt to new auth protocol where domain is moved to the operation field (While keeping compat with the old format)
 * Get measurement was not working since signed with the wrong method
 * inject_secret was not sending a json
 * Websocked auth was sending a twice serialized json
@olethanh olethanh force-pushed the andres-feature-implement_confidential_vm_initialization branch 2 times, most recently from c73c82d to fba8bc1 Compare July 4, 2024 19:44
update 'vendorized' aleph-vm auth file from source
@olethanh olethanh force-pushed the andres-feature-implement_confidential_vm_initialization branch from fba8bc1 to dd2639a Compare July 4, 2024 19:47
@olethanh olethanh force-pushed the andres-feature-implement_confidential_vm_initialization branch from 80b3520 to 196fd92 Compare July 4, 2024 20:19
…ure-implement_confidential_vm_initialization
@olethanh olethanh merged commit 247dbfc into 1yam-vm-client Jul 4, 2024
olethanh added a commit that referenced this pull request Jul 5, 2024
* Feature: VmClient

* Fix: Protocol (http/https) should not be hardcoded.

One place hardcoded `http://`, the other one `https://`.

* Fix: There was no test for `notify_allocation()`.

* WIP: Copy authentication functions from aleph-vm

* Fix: vm client sessions wasn't close + authentifications for test will use localhost as domain

* Add: Unit test for {perform_operation, stop, reboot, erase, expire}

* Refactor: logs didn't need to generate full header

Fix: extracts domain from node url instead of sending url

Fix: using vmclient sessions in get_logs instead of creating new one

* Add: get_logs test

* Fix: black in aleph_vm_authentification.py

fix: isort issue

Fix: mypy issue

Fix: black

Fix: isort

* Fix: fully remove _generate_header call in get_logs

Fix

Fix: using real path server instead of fake server for test

Fix: create playload

* Fix: black issue

* Fix: test fix workflow

* feat(vm_client): add missing types annotations

* refactor(vm_client): remove duplicated types annotations

* refactor(vm_client): avoid using single letter variable names

* feat(vm_client): increase test_notify_allocation precision

* refactor(vm_client): add empty lines for code readability

* style: run linting:fmt

* Fix: Required an old version of `aleph-message`

* Fix: Newer aleph-message requires InstanceEnvironment

Else tests were breaking.

* Fix: Qemu was not the default hypervisor for instances.

* Fix: Pythom 3.12 fails setup libsecp256k1

When "using bundled libsecp256k1", the setup using `/tmp/venv/bin/hatch run testing:test` fails to proceed on Python 3.12.

That library `secp256k1` has been unmaintained for more than 2 years now (0.14.0, Nov 6, 2021), and seems to not support Python 3.12.

The error in the logs:
```
File "/tmp/pip-build-env-ye8d6ort/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 862, in get_command_obj
 cmd_obj = self.command_obj[command] = klass(self)
                                                ^^^^^^^^^^^
      TypeError: 'NoneType' object is not callable
      [end of output]
```

See failing CI run:
https://github.com/aleph-im/aleph-sdk-python/actions/runs/9613634583/job/26516767722

* doc(README): command to launch tests was incorrect

* Refactor: create and sign playload goes to utils and some fix

* Fix: linting issue

* Fix: mypy issue

* fix: black

* feat: use bytes_from_hex where it makes sens

* chore: use ruff new CLI api

* feat: add unit tests for authentication mechanisms of VmClient

* fix: debug code remove

* Update vmclient.py

Co-authored-by: Olivier Le Thanh Duong <[email protected]>

* Fix: update unit test to use stream_logs endpoint instead of logs

* Implement `VmConfidentialClient` class (#138)

* Problem: A user cannot initialize an already created confidential VM.

Solution: Implement `VmConfidentialClient` class to be able to initialize and interact with confidential VMs.

* Problem: Auth was not working

Corrections:
 *  Measurement type returned was missing field needed for validation of measurements
 * Port number was not handled correctly in authentifaction
 * Adapt to new auth protocol where domain is moved to the operation field (While keeping compat with the old format)
 * Get measurement was not working since signed with the wrong method
 * inject_secret was not sending a json
 * Websocked auth was sending a twice serialized json
 * update 'vendorized' aleph-vm auth file from source


Co-authored-by: Hugo Herter <[email protected]>
Co-authored-by: Laurent Peuch <[email protected]>
Co-authored-by: Olivier Le Thanh Duong <[email protected]>
Co-authored-by: nesitor <[email protected]>
@Psycojoker Psycojoker deleted the andres-feature-implement_confidential_vm_initialization branch July 24, 2024 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

BLACK This PR has critical implications and must be reviewed by a senior engineer.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants