Skip to content

feat(hipdnn): re-save a loaded execution plan from the bytes it was loaded from #14517

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
GarbyGilbertAMD merged 5 commits into from
Oct 30, 2026
Merged

feat(hipdnn): re-save a loaded execution plan from the bytes it was loaded from #14517

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
448c5e8
feat(hipdnn): re-save a loaded execution plan from the bytes it was l…
GarbyGilbertAMD Oct 26, 2026
5f95776
fix(hipdnn): describe and test re-saving a loaded plan from its plan …
GarbyGilbertAMD Oct 28, 2026
49cd620
docs(hipdnn): describe re-saving a loaded plan and pin bare-plan grap…
GarbyGilbertAMD Oct 29, 2026
e6037ce
chore(hipdnn): merge develop into the loaded-plan re-save branch
GarbyGilbertAMD Oct 30, 2026
b1eb708
chore(hipdnn): merge develop again for the plan device-binding docs
GarbyGilbertAMD Oct 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dnn-providers/hip-kernel-provider/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ A plan-only save of a plan from any other engine of this provider fails. With `H
### Rules

- **Benchmarking plans save only after they have chosen a kernel.** A plan built with `global.benchmarking=1` or with `HIPDNN_FORCE_BENCHMARKING=1` usually chooses its kernel at its first `execute()`. (When a stored benchmark result already covers every candidate kernel, the build makes a plain plan that saves at once.) Saving a plan that has not chosen a kernel fails with "the plan has not chosen a kernel yet; execute it once, or build it with benchmarking off, then save". Execute the plan once and then save it, or build it with benchmarking off. For an engine that reports the note, this also makes `to_binary()` fail; it does not fall back to the graph only.
- **A loaded plan cannot be saved again.** Both `to_compiled_plan_binary()` of a loaded plan and `to_binary()` of a graph loaded with `from_binary(handle, bytes)` fail with "this plan was loaded from a saved payload; re-saving is not supported; keep the original bytes". Keep the bytes you loaded, and reuse them.
- **Saving a loaded plan again writes the plan bytes it was loaded from.** Both `to_compiled_plan_binary()` of a loaded plan and `to_binary()` of a graph loaded with `from_binary(handle, bytes)` succeed. hipDNN returns the saved plan bytes without calling the provider, so the new save holds the same kernel, the same provider version and the same format versions as the first. If the input was a graph-and-plan container, `to_compiled_plan_binary()` returns only the plan part. To use a newer kernel, build the plan again from its graph and save it.
- **A loaded plan runs the code it was saved with.** Upgrading the provider does not change the kernel inside a saved plan, even when the upgrade fixes that kernel. To use the new kernel, build the plan again from its graph and save it.
- **The kpack archive must not change between the plan build and the save.** A save gets the kernel's code object from the kpack archive that the plan build used, and checks it against the kernel's SHA-256. The save can read the archive through the handle that the plan build opened. If the archive changes after the plan build, the save may fail. If the save cannot read the archive, or the bytes do not match, the save fails.
- **The loading machine must have the same engine.** hipDNN loads a plan only through a plugin that serves the saved engine ID, so the descriptor set that defines the engine must be installed and loaded. The GPU must be able to run the stored code object, and this provider must still recognize the stored dispatch name, kernel arguments and launch values. The format version must be one this provider reads. The provider version does not need to match.
Expand Down
4 changes: 2 additions & 2 deletions dnn-providers/hip-kernel-provider/src/engines/kernel_ingestor_engine/IngestorPlanCapture.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ serialization::IngestorPlanPayload captureIngestorPlan(const hipdnn_plugin_sdk::
{
serialization::refuseIngestorPlanSave(
IngestorPlanRefusal::INCOMPATIBLE,
"this plan was loaded from a saved payload; re-saving is not supported; keep the "
"original bytes");
"this plan was loaded from a saved payload; hipDNN saves a loaded plan from its "
"original bytes and does not call the provider");
}

const auto* saveable = plan.saveablePlan();
Expand Down
6 changes: 4 additions & 2 deletions dnn-providers/hip-kernel-provider/src/engines/kernel_ingestor_engine/IngestorPlanCapture.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,10 @@ namespace hip_kernel_provider::kernel_ingestor_engine
/// code object, its dispatch handler's launch inputs, and the plan's own workspace size.
///
/// For a benchmarking plan, the plan saved is the candidate that the first execute()
/// chose. A plan restored from a saved payload is refused. A plan that no ingestor engine
/// built is refused. Reads the plan only through const accessors and an acquire load, so it
/// chose. A plan restored from a saved payload is never captured: hipDNN saves a loaded
/// plan from the bytes it was loaded from, so only a direct call of the provider hook sees
/// a restored plan, and the hook refuses it. A plan that no ingestor engine built is
/// refused. Reads the plan only through const accessors and an acquire load, so it
/// is safe while other threads execute the plan.
///
/// @throws HipdnnPluginException with a save refusal when the plan cannot be saved, and
Expand Down
3 changes: 2 additions & 1 deletion ...providers/hip-kernel-provider/src/engines/kernel_ingestor_engine/RestoredIngestorPlan.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ namespace hip_kernel_provider::kernel_ingestor_engine
/// A plan restored from a saved payload: one prepared dispatch and the handler that
/// launches it. It reports the workspace size the payload stores.
///
/// A restored plan cannot be saved again. saveablePlan() returns nullptr, and a save
/// The provider never captures a restored plan: hipDNN saves a loaded plan from the bytes
/// it was loaded from. saveablePlan() returns nullptr, and a direct call of the save hook
/// recognises this type and refuses it.
///
/// Immutable after construction, so several threads can execute it at once.
Expand Down
117 changes: 88 additions & 29 deletions ...el-provider/src/integration_tests/kernel_ingestor_engine/IntegrationGpuKernelIngestor.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -736,8 +736,11 @@ class IntegrationGpuKernelIngestorSaveLoad : public IntegrationGpuKernelIngestor
protected:
// Builds graph with pinnedEngineId, validates one run and saves the plan in form. Then
// it destroys graph, loads the plan with a new handle and validates two runs of the
// loaded plan. Every run uses the CPU reference of the original graph. Then it checks
// that the loaded plan cannot be saved again.
// loaded plan. Every run uses the CPU reference of the original graph. Then it saves the
// loaded plan again in the same form, destroys the loaded graph, loads the second save
// with another new handle, and checks the engine, the workspace size and two runs the
// same way. A plan-only second save must equal the first. For a graph-and-plan save, the
// loaded and the reloaded graph must save plan bytes equal to the original plan's.
void roundTripInAFreshHandle(std::shared_ptr<Graph> graph,
int64_t pinnedEngineId,
SaveForm form,
Expand Down Expand Up @@ -795,6 +798,21 @@ class IntegrationGpuKernelIngestorSaveLoad : public IntegrationGpuKernelIngestor
<< "the saved graph holds no execution plan";
}

// The plan bytes alone, as the built plan saves them. A graph-and-plan container holds
// exactly these bytes as its plan part, and loading a container keeps only that part.
std::vector<uint8_t> originalPlanBytes;
{
auto planSave = graph->to_compiled_plan_binary();
ASSERT_EQ(planSave.second.code, ErrorCode::OK) << planSave.second.err_msg;
originalPlanBytes = std::move(planSave.first);
}
const auto expectPlanBytesUnchanged = [&](const Graph& loadedGraph) {
auto planSave = loadedGraph.to_compiled_plan_binary();
ASSERT_EQ(planSave.second.code, ErrorCode::OK) << planSave.second.err_msg;
EXPECT_EQ(planSave.first, originalPlanBytes)
<< "the plan cut out of the container differs from the original plan bytes";
};

int64_t originalWorkspaceSize = 0;
result = graph->get_workspace_size(originalWorkspaceSize);
ASSERT_EQ(result.code, ErrorCode::OK) << result.err_msg;
Expand All @@ -803,34 +821,75 @@ class IntegrationGpuKernelIngestorSaveLoad : public IntegrationGpuKernelIngestor
graph.reset();
ASSERT_TRUE(original.expired()) << "another owner keeps the original graph alive";

const ScopedPluginLogCapture capture(this);
ASSERT_NO_FATAL_FAILURE(detail::replaceHandleWithAFreshOne(_handle, _stream));

auto loaded = std::make_shared<Graph>();
result = detail::loadInForm(*loaded, _handle, saved, form);
ASSERT_EQ(result.code, ErrorCode::OK) << result.err_msg;

EXPECT_TRUE(capture.recorder().hasLogContaining(detail::CREATING_CONTAINER_LOG))
<< "the new handle did not get a new provider container. Captured logs:\n"
<< capture.recorder().getRecordedLogsAsString();
detail::expectNoKernelSelectionLogged(capture.recorder());

int64_t loadedEngineId = 0;
result = loaded->get_execution_plan_engine_id(loadedEngineId);
ASSERT_EQ(result.code, ErrorCode::OK) << result.err_msg;
EXPECT_EQ(loadedEngineId, pinnedEngineId);

int64_t loadedWorkspaceSize = 0;
result = loaded->get_workspace_size(loadedWorkspaceSize);
ASSERT_EQ(result.code, ErrorCode::OK) << result.err_msg;
EXPECT_EQ(loadedWorkspaceSize, originalWorkspaceSize);

ASSERT_NO_FATAL_FAILURE(executeAndValidate(run, *loaded, _handle));
ASSERT_NO_FATAL_FAILURE(executeAndValidate(run, *loaded, _handle));
// Loads bytes in a fresh handle, checks that the load created a new provider
// container, kept the engine and workspace size, and logged no kernel selection,
// then validates two runs against the CPU reference of the original graph.
const auto loadCheckAndRun
= [&](const std::vector<uint8_t>& bytes, std::shared_ptr<Graph>& loaded) {
const ScopedPluginLogCapture capture(this);
ASSERT_NO_FATAL_FAILURE(detail::replaceHandleWithAFreshOne(_handle, _stream));

loaded = std::make_shared<Graph>();
auto loadResult = detail::loadInForm(*loaded, _handle, bytes, form);
ASSERT_EQ(loadResult.code, ErrorCode::OK) << loadResult.err_msg;

EXPECT_TRUE(capture.recorder().hasLogContaining(detail::CREATING_CONTAINER_LOG))
<< "the new handle did not get a new provider container. Captured logs:\n"
<< capture.recorder().getRecordedLogsAsString();
detail::expectNoKernelSelectionLogged(capture.recorder());

int64_t loadedEngineId = 0;
loadResult = loaded->get_execution_plan_engine_id(loadedEngineId);
ASSERT_EQ(loadResult.code, ErrorCode::OK) << loadResult.err_msg;
EXPECT_EQ(loadedEngineId, pinnedEngineId);

int64_t loadedWorkspaceSize = 0;
loadResult = loaded->get_workspace_size(loadedWorkspaceSize);
ASSERT_EQ(loadResult.code, ErrorCode::OK) << loadResult.err_msg;
EXPECT_EQ(loadedWorkspaceSize, originalWorkspaceSize);

ASSERT_NO_FATAL_FAILURE(executeAndValidate(run, *loaded, _handle));
ASSERT_NO_FATAL_FAILURE(executeAndValidate(run, *loaded, _handle));
};

std::shared_ptr<Graph> loaded;
ASSERT_NO_FATAL_FAILURE(loadCheckAndRun(saved, loaded));

// Saving the loaded plan again must succeed. A plan-only save is exactly the bytes
// the plan was loaded from. A graph-and-plan save also holds the graph, which is
// not compared byte for byte.
std::vector<uint8_t> resaved;
{
auto resave = detail::saveInForm(*loaded, form);
ASSERT_EQ(resave.second.code, ErrorCode::OK) << resave.second.err_msg;
resaved = std::move(resave.first);
}
ASSERT_FALSE(resaved.empty());
if(form == SaveForm::PLAN_ONLY)
{
EXPECT_EQ(resaved, saved) << "the second save differs from the bytes loaded";
}
else
{
int contents = 0;
ASSERT_EQ(hipdnnBackendGetSerializedBinaryContents_ext(
resaved.data(), resaved.size(), &contents),
HIPDNN_STATUS_SUCCESS);
ASSERT_NE(contents & HIPDNN_SERIALIZED_CONTENT_EXECUTION_PLAN, 0)
<< "the second save holds no execution plan";
}
if(form == SaveForm::GRAPH_AND_PLAN)
{
ASSERT_NO_FATAL_FAILURE(expectPlanBytesUnchanged(*loaded));
}

const auto resave = detail::saveInForm(*loaded, form);
expectPluginRefusal(
resave.second, HIPDNN_PLUGIN_STATUS_NOT_APPLICABLE, "loaded from a saved payload");
loaded.reset();
std::shared_ptr<Graph> reloaded;
ASSERT_NO_FATAL_FAILURE(loadCheckAndRun(resaved, reloaded));
if(form == SaveForm::GRAPH_AND_PLAN)
{
ASSERT_NO_FATAL_FAILURE(expectPlanBytesUnchanged(*reloaded));
}
}
};

Expand Down
7 changes: 4 additions & 3 deletions dnn-providers/hip-kernel-provider/src/tests/core/TestPluginSerialization.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -367,16 +367,17 @@ TEST_P(TestPluginSerializationKpack, RoundTripsAKpackPlanThroughTheHooks)
}

{
SCOPED_TRACE("re-save of the restored plan");
// A restored plan cannot be saved again.
SCOPED_TRACE("direct save hook call on the restored plan");
// hipDNN saves a loaded plan from its original bytes, so only a direct hook call
// reaches the provider with a restored plan. The hook refuses it.
const uint8_t byte = 0;
hipdnnPluginConstData_t resaved{&byte, 7};
const std::string message
= expectHookRefusal(hipdnnEnginePluginSerializeExecutionContext(
loadHandle.get(), restored.context(), &resaved),
HIPDNN_PLUGIN_STATUS_NOT_APPLICABLE,
serialization::INGESTOR_PLAN_SAVE_INCOMPATIBLE_PREFIX,
"loaded from a saved payload");
"does not call the provider");
EXPECT_EQ(message.find(NOT_AN_INGESTOR_ENGINE_PHRASE), std::string::npos) << message;
EXPECT_EQ(resaved.ptr, static_cast<const void*>(&byte));
EXPECT_EQ(resaved.size, 7U);
Expand Down
4 changes: 2 additions & 2 deletions .../hip-kernel-provider/src/tests/engines/kernel_ingestor_engine/TestIngestorPlanCapture.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,8 +163,8 @@ TEST(TestIngestorPlanCapture, RefusesARestoredPlan)
handler, std::make_unique<PreparedDispatch>(), 0, "restored test plan");

expectSaveRefusal(plan, HIPDNN_PLUGIN_STATUS_NOT_APPLICABLE, "loaded from a saved payload");
const std::string message
= expectSaveRefusal(plan, HIPDNN_PLUGIN_STATUS_NOT_APPLICABLE, "keep the original bytes");
const std::string message = expectSaveRefusal(
plan, HIPDNN_PLUGIN_STATUS_NOT_APPLICABLE, "does not call the provider");

// The benchmarking advice does not apply to a restored plan.
EXPECT_EQ(message.find(NO_WINNER_PHRASE), std::string::npos) << message;
Expand Down
9 changes: 8 additions & 1 deletion projects/hipdnn/backend/include/hipdnn_backend.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,12 @@ HIPDNN_BACKEND_EXPORT hipdnnStatus_t
* contain a hipDNN FlatBuffer envelope with routing metadata, plan tensor UIDs, and opaque
* plugin-specific execution context bytes.
*
* A plan created by hipdnnBackendCreateAndDeserializeExecutionPlan_ext() is not serialized again:
* the call returns the plan bytes it was deserialized from, unchanged, and does not call the
* plugin. When those bytes came from a graph-and-plan container, only the embedded plan is
* returned. The returned bytes keep their original envelope version, plugin payload and kernel,
* so upgrading a plugin does not change them; build a new plan to pick up a newer kernel.
*
* @param [in] descriptor A finalized execution plan descriptor.
* @param [in] requestedByteSize Size of the caller-allocated buffer in bytes.
* Ignored when @p serializedPlan is @c nullptr.
Expand All @@ -459,7 +465,8 @@ HIPDNN_BACKEND_EXPORT hipdnnStatus_t
* extracted and deserialized. A container that carries no execution plan is rejected with
* HIPDNN_STATUS_BAD_PARAM. A bare execution-plan blob (as produced by
* hipdnnBackendGetSerializedExecutionPlan_ext()) is deserialized directly. The returned execution
* plan descriptor is finalized in both cases.
* plan descriptor is finalized in both cases. It keeps a copy of the plan bytes, which can include
* GPU code objects, for its lifetime; hipdnnBackendGetSerializedExecutionPlan_ext() returns them.
*
* The returned plan is bound to the device of @p handle's stream and executes only on handles
* whose stream is on that device (see hipdnnBackendExecute()). To move a plan to another device,
Expand Down
11 changes: 6 additions & 5 deletions projects/hipdnn/backend/src/descriptors/ExecutionPlanDescriptor.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -425,7 +425,8 @@ void ExecutionPlanDescriptor::serializeBackendPlan(size_t requestedByteSize,

const std::lock_guard<std::mutex> lock(_serializedPlanCacheMutex);

// Build the serialized plan on the first call. Reuse it for all later size and fill calls.
// A loaded plan starts with the plan bytes it was loaded from; a built plan builds them on
// its first save. Later size and fill calls reuse the cached bytes.
if(_serializedPlanCache.empty())
{
std::vector<uint8_t> pluginPayload;
Expand Down Expand Up @@ -539,15 +540,15 @@ void ExecutionPlanDescriptor::deserializeBackendPlan(
{
_tensorAlignments.clear();
}
std::vector<uint8_t> pluginPayload(serializedPluginPayload->begin(),
serializedPluginPayload->end());

const hipdnnPluginConstData_t pluginPayloadData{pluginPayload.data(), pluginPayload.size()};
const hipdnnPluginConstData_t pluginPayloadData{serializedPluginPayload->data(),
serializedPluginPayload->size()};

_pluginResourceManager = pluginResourceManager;
_executionContext = plugin::EnginePluginResourceManager::createExecutionContextFromSerialized(
_pluginResourceManager, _engineId, &pluginPayloadData);

_serializedPlanCache.assign(serializedPlan, serializedPlan + planByteSize);

HipdnnBackendDescriptorImpl<ExecutionPlanDescriptor>::finalize();
}

Expand Down
7 changes: 4 additions & 3 deletions projects/hipdnn/backend/src/descriptors/ExecutionPlanDescriptor.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,10 @@ class ExecutionPlanDescriptor : public HipdnnBackendDescriptorImpl<ExecutionPlan
int64_t _engineId = INVALID_ENGINE_ID;
bool _isOverrideShapeEnabled = false;

// Cached serialized execution-plan blob. Populated lazily on the first
// serializeBackendPlan() call and reused by all subsequent size/fill calls.
// A finalized plan is immutable, so this never needs invalidation.
// Cached serialized execution-plan blob. A loaded plan starts with the exact plan bytes it was
// loaded from (for a graph-and-plan container, only the plan part). A built plan fills it on
// its first serializeBackendPlan() call. Every later size/fill call reuses it. A finalized
// plan is immutable, so this never needs invalidation.
mutable std::vector<uint8_t> _serializedPlanCache;
// Guards _serializedPlanCache. serializeBackendPlan() is const and can run on several threads.
mutable std::mutex _serializedPlanCacheMutex;
Expand Down
31 changes: 31 additions & 0 deletions projects/hipdnn/backend/tests/TestGraphAndPlanSerializationApi.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -682,6 +682,37 @@ TEST_F(TestGraphAndPlanSerializationApi, DeserializeGraphIdentifiedButCorruptCon
EXPECT_EQ(rawDesc, nullptr);
}

// Neither a bare plan nor a bare graph carries a file identifier, so the graph loader
// cannot tell them apart and parses a bare plan as a graph. No check looks for a plan;
// the Graph verifier rejects it as a side effect. At version 2 the first check to fail is
// the Graph name: the verifier reads the plan's version field as the offset of the name
// string, and an offset of 2 is not 4-byte aligned. With this fixture's plugin payload,
// the graph loader also rejects the plan when PLAN_SERIALIZATION_VERSION is 0 or 4, so a
// version change alone does not fail this test. The plan comes from the production
// writer, so a change to the plan envelope or the graph loader that lets a plan through
// fails this test.
TEST_F(TestGraphAndPlanSerializationApi, DeserializeGraphRejectsBarePlanBlob)
{
makeExecutionPlanFinalized();
expectPlanSerialization();

size_t size = 0;
ASSERT_EQ(hipdnnBackendGetSerializedExecutionPlan_ext(_planWrapper.get(), 0, &size, nullptr),
HIPDNN_STATUS_SUCCESS);
std::vector<uint8_t> planBlob(size);
ASSERT_EQ(hipdnnBackendGetSerializedExecutionPlan_ext(
_planWrapper.get(), size, &size, planBlob.data()),
HIPDNN_STATUS_SUCCESS);
flatbuffers::Verifier verifier(planBlob.data(), planBlob.size());
ASSERT_TRUE(fbs::VerifySerializedExecutionPlanBuffer(verifier));

hipdnnBackendDescriptor_t rawDesc = nullptr;
EXPECT_EQ(
hipdnnBackendCreateAndDeserializeGraph_ext(&rawDesc, planBlob.data(), planBlob.size()),
HIPDNN_STATUS_BAD_PARAM);
EXPECT_EQ(rawDesc, nullptr);
}

// ============================================================================
// Plan deserialize (hipdnnBackendCreateAndDeserializeExecutionPlan_ext)
//
Expand Down
Loading
Loading