Skip to content

[windows] add Embeddable Python 3.10.1 to build.ps1 #83488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+50 −7
Open

[windows] add Embeddable Python 3.10.1 to build.ps1 #83488

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 50 additions & 7 deletions utils/build.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -415,10 +415,18 @@ $KnownPythons = @{
URL = "https://www.nuget.org/api/v2/package/python/3.10.1";
SHA256 = "987a0e446d68900f58297bc47dc7a235ee4640a49dace58bc9f573797d3a8b33";
};
AMD64_Embedded = @{
URL = "https://www.python.org/ftp/python/3.10.1/python-3.10.1-embed-amd64.zip";
SHA256 = "502670dcdff0083847abf6a33f30be666594e7e5201cd6fccd4a523b577403de";
};
ARM64 = @{
URL = "https://www.nuget.org/api/v2/package/pythonarm64/3.10.1";
SHA256 = "16becfccedf1269ff0b8695a13c64fac2102a524d66cecf69a8f9229a43b10d3";
};
ARM64_Embedded = @{
URL = "https://www.python.org/ftp/python/3.10.1/python-3.10.1-embed-arm64.zip";
SHA256 = "1f9e215fe4e8f22a8e8fba1859efb1426437044fb3103ce85794630e3b511bc2";
};
};
}

Expand Down Expand Up @@ -743,6 +751,10 @@ function Get-PythonPath([Hashtable] $Platform) {
return [IO.Path]::Combine("$BinaryCache\", "Python$($Platform.Architecture.CMakeName)-$PythonVersion")
}

function Get-EmbeddedPythonPath([Hashtable] $Platform) {
Copy link
Member

@compnerd compnerd Aug 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, is there a bigger picture around the embedded vs non-embedded python that the extraction needs special logic?

Copy link
Contributor Author

@charles-zablit charles-zablit Aug 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only reason is pip missing in embeddedable python. We build with the regular Python, but we bundle the embeddable python in the installer.

Copy link
Member

@compnerd compnerd Aug 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There-in lies the trick! We can inject pip into the packaging :)

Copy link
Contributor Author

@charles-zablit charles-zablit Aug 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think injecting pip would work well because of this SO comment, especially because the Python documentation explicitly states:

Using pip to manage dependencies as for a regular Python installation is not supported with this distribution [embeddable Python]

Given that, and the extra steps it would take to manually install pip with the embeddable python, I think it's simpler to:

  • Build with the regular Python
  • Test with the regular Python (easier to install dependencies)
  • Package the embeddable Python.

Copy link
Member

@compnerd compnerd Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we should just use a boolean state on the known dictionary rather than extract this. Overall, I'd like to see the python distribution to be complete enough for use in scripting (i.e. has pip).

Copy link
Contributor Author

@charles-zablit charles-zablit Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we should just use a boolean state on the known dictionary rather than extract this.

I'm not sure I understand what that boolean state would look like: would it be something like hasPip?

Overall, I'd like to see the python distribution to be complete enough for use in scripting (i.e. has pip).

I agree that it would be more convenient, however in that case, why not package a complete Python install? pip is not supported by embeddable Python: https://github.com/swiftlang/swift/pull/83488/files#r2254566330.

I see the Embeddable Python in the installer feature as a way for users to get a functional lldb whether they have the correct Python version installed or not. This is the main source of confusion for users: llvm/llvm-project#74073 (comment)

I think it's reasonable to expect power users who want to have scripting features to install Python themselves and have proper pip support rather than to risk using pip in an unsupported way with Embeddable Python.

return [IO.Path]::Combine("$BinaryCache\", "EmbeddedPython$($Platform.Architecture.CMakeName)-$PythonVersion")
}

function Get-PythonExecutable {
return [IO.Path]::Combine((Get-PythonPath $BuildPlatform), "tools", "python.exe")
}
Expand All @@ -751,6 +763,10 @@ function Get-PythonScriptsPath {
return [IO.Path]::Combine((Get-PythonPath $BuildPlatform), "tools", "Scripts")
}

function Get-EmbeddedPythonInstallDir() {
return [IO.Path]::Combine("$ImageRoot\", "Program Files", "Swift", "Python-$PythonVersion")
}

function Get-Syft {
return $KnownSyft[$SyftVersion][$BuildArchName]
}
Expand Down Expand Up @@ -1119,6 +1135,10 @@ function Invoke-VsDevShell([Hashtable] $Platform) {
}
}

function Get-PythonLibName() {
return "python{0}{1}" -f ([System.Version]$PythonVersion).Major, ([System.Version]$PythonVersion).Minor
}

function Get-Dependencies {
Record-OperationTime $BuildPlatform "Get-Dependencies" {
function Write-Success([string] $Description) {
Expand Down Expand Up @@ -1252,20 +1272,32 @@ function Get-Dependencies {
Write-Success "syft $SyftVersion"
}

function Get-KnownPython([string] $ArchName) {
function Get-KnownPython([string] $ArchName, [bool] $EmbeddedPython = $false) {
if (-not $KnownPythons.ContainsKey($PythonVersion)) {
throw "Unknown python version: $PythonVersion"
}
return $KnownPythons[$PythonVersion].$ArchName
$Key = $(if ($EmbeddedPython) { "${ArchName}_Embedded" } else { $ArchName })
return $KnownPythons[$PythonVersion][$Key]
}

function Install-Python([string] $ArchName) {
$Python = Get-KnownPython $ArchName
DownloadAndVerify $Python.URL "$BinaryCache\Python$ArchName-$PythonVersion.zip" $Python.SHA256
function Install-Python([string] $ArchName, [bool] $EmbeddedPython = $false) {
$Python = Get-KnownPython $ArchName $EmbeddedPython
$FileName = $(if ($EmbeddedPython) { "EmbeddedPython$ArchName-$PythonVersion" } else { "Python$ArchName-$PythonVersion" })
DownloadAndVerify $Python.URL "$BinaryCache\$FileName.zip" $Python.SHA256
if (-not $ToBatch) {
Expand-ZipFile Python$ArchName-$PythonVersion.zip "$BinaryCache" Python$ArchName-$PythonVersion
Expand-ZipFile "$FileName.zip" "$BinaryCache" "$FileName"
Write-Success "$ArchName Python $PythonVersion"
}
if (-not $EmbeddedPython) {
return
}
$PythonPTHPath = "$BinaryCache/$FileName/$(Get-PythonLibName)._pth"
$PythonPTHContent = [System.IO.File]::ReadAllText($PythonPTHPath).Replace("#import site","import site")
[System.IO.File]::WriteAllText($PythonPTHPath, $PythonPTHContent)
$GetPipURL = "https://bootstrap.pypa.io/get-pip.py"
$GetPipPath = "$BinaryCache/$FileName/get-pip.py"
$WebClient.DownloadFile($GetPipURL, $GetPipPath)
& "$BinaryCache/$FileName/python.exe" $GetPipPath
}

function Install-PIPIfNeeded {
Expand Down Expand Up @@ -1318,8 +1350,10 @@ function Get-Dependencies {

# Ensure Python modules that are required as host build tools
Install-Python $HostArchName
Install-Python $HostArchName $true
if ($IsCrossCompiling) {
Install-Python $BuildArchName
Install-Python $BuildArchName $true
}
Install-PythonModules

Expand Down Expand Up @@ -2169,7 +2203,7 @@ function Build-CDispatch([Hashtable] $Platform, [switch] $Static = $false) {
function Get-CompilersDefines([Hashtable] $Platform, [string] $Variant, [switch] $Test) {
$BuildTools = [IO.Path]::Combine((Get-ProjectBinaryCache $BuildPlatform BuildTools), "bin")
$PythonRoot = [IO.Path]::Combine((Get-PythonPath $Platform), "tools")
$PythonLibName = "python{0}{1}" -f ([System.Version]$PythonVersion).Major, ([System.Version]$PythonVersion).Minor
$PythonLibName = Get-PythonLibName

$TestDefines = if ($Test) {
@{
Expand Down Expand Up @@ -2212,6 +2246,7 @@ function Get-CompilersDefines([Hashtable] $Platform, [string] $Variant, [switch]
LLDB_PYTHON_EXE_RELATIVE_PATH = "python.exe";
LLDB_PYTHON_EXT_SUFFIX = ".pyd";
LLDB_PYTHON_RELATIVE_PATH = "lib/site-packages";
LLDB_PYTHON_DLL_RELATIVE_PATH = "../../../../Python-$PythonVersion";
LLDB_TABLEGEN = (Join-Path -Path $BuildTools -ChildPath "lldb-tblgen.exe");
LLDB_TEST_MAKE = "$BinaryCache\GnuWin32Make-4.4.1\bin\make.exe";
LLVM_CONFIG_PATH = (Join-Path -Path $BuildTools -ChildPath "llvm-config.exe");
Expand All @@ -2236,6 +2271,7 @@ function Get-CompilersDefines([Hashtable] $Platform, [string] $Variant, [switch]
Python3_INCLUDE_DIR = "$PythonRoot\include";
Python3_LIBRARY = "$PythonRoot\libs\$PythonLibName.lib";
Python3_ROOT_DIR = $PythonRoot;
Python3_VERSION = $PythonVersion;
SWIFT_TOOLCHAIN_VERSION = "${ToolchainIdentifier}";
SWIFT_BUILD_SWIFT_SYNTAX = "YES";
SWIFT_CLANG_LOCATION = (Get-PinnedToolchainToolsDir);
Expand Down Expand Up @@ -3938,6 +3974,12 @@ function Install-HostToolchain() {
Copy-Item -Force `
-Path $SwiftDriver `
-Destination "$($HostPlatform.ToolchainInstallRoot)\usr\bin\swiftc.exe"

# Copy embeddable Python
New-Item -Type Directory -Path "$(Get-EmbeddedPythonInstallDir)" -ErrorAction Ignore | Out-Null
Copy-Item -Force -Recurse `
-Path "$(Get-EmbeddedPythonPath $HostPlatform)\*" `
-Destination "$(Get-EmbeddedPythonInstallDir)"
}

function Build-Inspect([Hashtable] $Platform) {
Expand Down Expand Up @@ -4011,6 +4053,7 @@ function Build-Installer([Hashtable] $Platform) {
INCLUDE_SWIFT_DOCC = $INCLUDE_SWIFT_DOCC;
SWIFT_DOCC_BUILD = "$(Get-ProjectBinaryCache $HostPlatform DocC)\release";
SWIFT_DOCC_RENDER_ARTIFACT_ROOT = "${SourceCache}\swift-docc-render-artifact";
PythonVersion = $PythonVersion
}

Invoke-IsolatingEnvVars {
Expand Down