Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions docs/source/api-reference/http/http-service.md
Original file line number Diff line number Diff line change
Expand Up @@ -798,6 +798,55 @@ curl -X DELETE "http://localhost:9003/api/v1/tenant_quotas?tenant_id=tenant-a"
}
```

### Soft-Pin Control Endpoints

`POST /api/v1/soft_pin` and `POST /api/v1/soft_unpin` update existing
objects without another Put or Upsert. They are served on the master's admin
HTTP port, not the Store RPC or Transfer Engine metadata port.

```bash
curl -X POST http://localhost:9003/api/v1/soft_pin \
-H 'Content-Type: application/json' \
-d '{"tenant_id":"default","keys":["object-a","object-b"],"ttl_ms":60000}'

curl -X POST http://localhost:9003/api/v1/soft_unpin \
-H 'Content-Type: application/json' \
-d '{"tenant_id":"default","keys":["object-a","object-b"]}'
```

- `keys` contains 1–4096 exact object keys. Logical KV pages split into multiple
component objects require every relevant key to be supplied.
- `tenant_id` defaults to `default`, with the same tenant resolution as Store
operations. When multi-tenancy is disabled, operations use the default tenant.
- `ttl_ms` is required for `soft_pin` and must be absent for `soft_unpin`.
Each successful pin replaces the deadline with the master's current time plus
this TTL, so a repeated command can extend or shorten protection. The master's
`max_kv_soft_pin_ttl` limit applies. Zero TTL clears the soft pin.
- Unpin clears the object's soft pin regardless of which sessions use it. The
caller owns sharing policy and command ordering. Neither operation changes
hard pins, read leases, session tags, or object data.
- TTL expiration makes the object eligible for normal eviction; it does not
delete the object. Eviction checks the deadline directly; the soft-pin count
metric can lag until the existing periodic cleanup runs. Pressure eviction
still follows `allow_evict_soft_pinned_objects`. These soft pins are volatile master state,
following existing write-time soft-pin behavior; they are not persisted in
the metadata OpLog for failover.

HTTP 200 contains one result per input key, in input order. A batch is not
atomic and HTTP 200 does not mean that every key succeeded:

```json
{"results":[{"key":"object-a","error_code":0},{"key":"object-b","error_code":-704}]}
```

`0` means success, `-704` means the object was not found, `-703` means a write
is in progress or there is no completed replica, and `-600` means invalid
parameters (including TTL above the configured limit). Missing objects are
not created. Repeated unpin succeeds while the object still exists.
Malformed requests return HTTP 400. An inactive service plane returns HTTP
503. A transport timeout leaves the result unknown; callers must coordinate
retries with newer commands because replaying a pin resets its deadline.

### Data Deletion Endpoints

#### `POST /api/v1/remove_all`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -720,7 +720,7 @@ mooncake_master \
|------|---------|-------------|
| `--default_kv_lease_ttl` | `10000` ms | Lease TTL for KV objects. Supports `5000ms`, `5s`, `30m`, `1h` |
| `--default_kv_soft_pin_ttl` | `1800000` ms | Soft pin TTL (30 min) |
| `--max_kv_soft_pin_ttl` | `86400000` ms | Maximum request-level soft pin TTL (24 h) |
| `--max_kv_soft_pin_ttl` | `604800000` ms | Maximum request-level soft pin TTL (7 days) |
| `--allow_evict_soft_pinned_objects` | `true` | Allow evicting soft-pinned objects |
| `--eviction_ratio` | `0.05` | Fraction evicted at high watermark |
| `--eviction_high_watermark_ratio` | `0.90` | Usage ratio triggering eviction |
Expand Down
2 changes: 1 addition & 1 deletion docs/source/design/store/mooncake-store.md
Original file line number Diff line number Diff line change
Expand Up @@ -379,7 +379,7 @@ There are three startup parameters in `master_service` related to the soft pin m

- `default_kv_soft_pin_ttl`: The fixed soft-pin lifetime (in milliseconds) used when an `ENABLE` request omits `soft_pin_ttl_ms`. The default value is `30 minutes`; reads do not extend it.

- `max_kv_soft_pin_ttl`: The largest request-level soft-pin TTL accepted by the Master. The default value is `24 hours`.
- `max_kv_soft_pin_ttl`: The largest request-level soft-pin TTL accepted by the Master. The default value is `7 days`.

- `allow_evict_soft_pinned_objects`: Whether soft pinned objects are allowed to be evicted. The default value is `true`.

Expand Down
2 changes: 1 addition & 1 deletion docs/source/zh_archive/mooncake-store.md
Original file line number Diff line number Diff line change
Expand Up @@ -637,7 +637,7 @@ soft pin 是仅在运行时生效的淘汰优先级状态,不会持久化到

* `default_kv_soft_pin_ttl`:未显式传入 TTL 时使用的固定软固定生命周期。默认值为 `30 分钟`,访问不会续期。

* `max_kv_soft_pin_ttl`:Master 接受的请求级 soft pin TTL 上限。默认值为 `24 小时`。
* `max_kv_soft_pin_ttl`:Master 接受的请求级 soft pin TTL 上限。默认值为 `7 天`。

* `allow_evict_soft_pinned_objects`:是否允许替换已被软固定的对象。默认值为 `true`。

Expand Down
2 changes: 1 addition & 1 deletion mooncake-store/conf/master.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"rpc_enable_tcp_no_delay": true,
"default_kv_lease_ttl": 10000,
"default_kv_soft_pin_ttl": 1800000,
"max_kv_soft_pin_ttl": 86400000,
"max_kv_soft_pin_ttl": 604800000,
"allow_evict_soft_pinned_objects": true,
"eviction_ratio": 0.1,
"eviction_high_watermark_ratio": 1.0,
Expand Down
2 changes: 1 addition & 1 deletion mooncake-store/conf/master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ rpc_enable_tcp_no_delay: true

default_kv_lease_ttl: 10000
default_kv_soft_pin_ttl: 1800000
max_kv_soft_pin_ttl: 86400000
max_kv_soft_pin_ttl: 604800000
allow_evict_soft_pinned_objects: true
eviction_ratio: 0.1
# Overrides the 0.90 code default. A value of 1.0 disables proactive
Expand Down
3 changes: 3 additions & 0 deletions mooncake-store/include/master_admin_service.h
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,9 @@ class MasterAdminServer {
void HandleRemoveAll(coro_http::coro_http_request& req,
coro_http::coro_http_response& resp);

void HandleSoftPin(coro_http::coro_http_request& req,
coro_http::coro_http_response& resp, bool unpin);

void RegisterHandler();
void RefreshStorageMetrics() const;

Expand Down
6 changes: 6 additions & 0 deletions mooncake-store/include/master_service.h
Original file line number Diff line number Diff line change
Expand Up @@ -687,6 +687,12 @@ class MasterService {
*/
long RemoveAll(const TenantId& tenant_id, bool force = false);

// Update existing objects without transferring data. Results follow input
// order; batches are not atomic. A zero TTL clears the soft pin.
auto BatchUpdateSoftPin(const std::vector<std::string>& keys,
const TenantId& tenant_id, uint64_t ttl_ms)
-> std::vector<tl::expected<void, ErrorCode>>;

/**
* @brief Batch remove objects and their replicas
* @param keys The list of keys to remove.
Expand Down
14 changes: 11 additions & 3 deletions mooncake-store/include/object_metadata.h
Original file line number Diff line number Diff line change
Expand Up @@ -408,6 +408,14 @@ struct ObjectMetadata {
std::move(*pending_soft_pin_action);
pending_soft_pin_action.reset();

return ApplySoftPin({pending.action, pending.ttl}, now);
}

// Caller holds the object's metadata write lock. Independent commands do
// not consume or replace a pending write's soft-pin action.
SoftPinEvaluation ApplySoftPin(
const ResolvedSoftPinRequest& request,
const std::chrono::system_clock::time_point& now) {
SpinLocker locker(&lock);
int metric_delta = 0;
std::optional<std::chrono::system_clock::time_point> removed_deadline;
Expand All @@ -418,11 +426,11 @@ struct ObjectMetadata {
--metric_delta;
}

switch (pending.action) {
switch (request.action) {
case SoftPinAction::PRESERVE:
break;
case SoftPinAction::ENABLE:
if (pending.ttl == std::chrono::milliseconds::zero()) {
if (request.ttl == std::chrono::milliseconds::zero()) {
if (soft_pin_timeout) {
removed_deadline = *soft_pin_timeout;
soft_pin_timeout.reset();
Expand All @@ -432,7 +440,7 @@ struct ObjectMetadata {
if (!soft_pin_timeout) {
++metric_delta;
}
soft_pin_timeout = ComputeSoftPinDeadline(now, pending.ttl);
soft_pin_timeout = ComputeSoftPinDeadline(now, request.ttl);
deadline_to_index = soft_pin_timeout;
// Upserting the latest registration supersedes any
// expired or previously active deadline.
Expand Down
4 changes: 4 additions & 0 deletions mooncake-store/include/rpc_service.h
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,10 @@ class WrappedMasterService {
long RemoveAll(bool force = false,
const std::string& tenant_id = "default");

std::vector<tl::expected<void, ErrorCode>> BatchUpdateSoftPin(
const std::vector<std::string>& keys, uint64_t ttl_ms,
const std::string& tenant_id = "default");

std::vector<tl::expected<void, ErrorCode>> BatchRemove(
const std::vector<std::string>& keys, bool force = false,
const std::string& tenant_id = "default");
Expand Down
2 changes: 1 addition & 1 deletion mooncake-store/include/types.h
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ static constexpr uint64_t DEFAULT_DEFAULT_KV_LEASE_TTL =
static constexpr uint64_t DEFAULT_KV_SOFT_PIN_TTL_MS =
30 * 60 * 1000; // 30 minutes
static constexpr uint64_t DEFAULT_MAX_KV_SOFT_PIN_TTL_MS =
24 * 60 * 60 * 1000; // 24 hours
7 * 24 * 60 * 60 * 1000; // 7 days
static constexpr bool DEFAULT_ALLOW_EVICT_SOFT_PINNED_OBJECTS = true;
static constexpr double DEFAULT_EVICTION_RATIO = 0.05;
static constexpr double DEFAULT_EVICTION_HIGH_WATERMARK_RATIO = 0.90;
Expand Down
5 changes: 3 additions & 2 deletions mooncake-store/src/master.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -45,13 +45,14 @@ static_assert(mooncake::DEFAULT_DEFAULT_KV_LEASE_TTL == 10000,
static_assert(mooncake::DEFAULT_KV_SOFT_PIN_TTL_MS == 30 * 60 * 1000,
"Update kDefaultKvSoftPinTtlFlagValue when "
"DEFAULT_KV_SOFT_PIN_TTL_MS changes");
static_assert(mooncake::DEFAULT_MAX_KV_SOFT_PIN_TTL_MS == 24 * 60 * 60 * 1000,
static_assert(mooncake::DEFAULT_MAX_KV_SOFT_PIN_TTL_MS ==
7 * 24 * 60 * 60 * 1000,
"Update kDefaultMaxKvSoftPinTtlFlagValue when "
"DEFAULT_MAX_KV_SOFT_PIN_TTL_MS changes");

constexpr char kDefaultKvLeaseTtlFlagValue[] = "10000";
constexpr char kDefaultKvSoftPinTtlFlagValue[] = "1800000";
constexpr char kDefaultMaxKvSoftPinTtlFlagValue[] = "86400000";
constexpr char kDefaultMaxKvSoftPinTtlFlagValue[] = "604800000";
// Allowed range of --ha_leader_lease_ttl_sec.
constexpr int64_t kMinHaLeaderLeaseTtlSec = 2;
constexpr int64_t kMaxHaLeaderLeaseTtlSec = 60;
Expand Down
76 changes: 76 additions & 0 deletions mooncake-store/src/master_admin_service.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1337,6 +1337,71 @@ void MasterAdminServer::HandleDeleteTenantQuota(
});
}

struct HttpSoftPinRequest {
std::string tenant_id{"default"};
std::vector<std::string> keys;
std::optional<uint64_t> ttl_ms;
};
YLT_REFL(HttpSoftPinRequest, tenant_id, keys, ttl_ms);

struct HttpSoftPinResult {
std::string key;
int32_t error_code{0};
};
YLT_REFL(HttpSoftPinResult, key, error_code);

struct HttpSoftPinResponse {
std::vector<HttpSoftPinResult> results;
};
YLT_REFL(HttpSoftPinResponse, results);

void MasterAdminServer::HandleSoftPin(coro_http::coro_http_request& req,
coro_http::coro_http_response& resp,
bool unpin) {
HttpSoftPinRequest request;
try {
const auto body = req.get_body();
auto cursor = body.begin();
auto end = body.end();
struct_json::from_json(request, cursor, end);
if (!std::all_of(cursor, end, [](char ch) {
return ch == ' ' || ch == '\t' || ch == '\r' || ch == '\n';
})) {
WriteErrorResponse(resp, coro_http::status_type::bad_request,
ErrorCode::INVALID_PARAMS,
"Unexpected content after JSON body");
return;
}
} catch (const std::exception& e) {
WriteErrorResponse(resp, coro_http::status_type::bad_request,
ErrorCode::INVALID_PARAMS,
std::string("Invalid JSON body: ") + e.what());
return;
}
if (request.keys.empty() || request.keys.size() > 4096 ||
!TenantId(request.tenant_id).IsValid() ||
(unpin ? request.ttl_ms.has_value() : !request.ttl_ms.has_value())) {
WriteErrorResponse(resp, coro_http::status_type::bad_request,
ErrorCode::INVALID_PARAMS,
"Expected 1..4096 keys, a valid tenant, and ttl_ms "
"only for soft_pin");
return;
}
WithActiveService(resp, [&](auto service) {
const auto results = service->BatchUpdateSoftPin(
request.keys, unpin ? 0 : *request.ttl_ms, request.tenant_id);
HttpSoftPinResponse payload;
payload.results.reserve(results.size());
for (size_t i = 0; i < results.size(); ++i) {
payload.results.push_back(
{request.keys[i],
static_cast<int32_t>(results[i] ? ErrorCode::OK
: results[i].error())});
}
WriteJsonResponse(resp, coro_http::status_type::ok, payload);
});
}

struct HttpRemoveAllResponse {
bool success{true};
long removed_count{0};
Expand Down Expand Up @@ -1367,6 +1432,17 @@ void MasterAdminServer::HandleRemoveAll(coro_http::coro_http_request& req,
void MasterAdminServer::RegisterHandler() {
using namespace coro_http;

http_server_.set_http_handler<POST>(
"/api/v1/soft_pin",
[this](coro_http_request& req, coro_http_response& resp) {
HandleSoftPin(req, resp, false);
});
http_server_.set_http_handler<POST>(
"/api/v1/soft_unpin",
[this](coro_http_request& req, coro_http_response& resp) {
HandleSoftPin(req, resp, true);
});

http_server_.set_http_handler<GET>(
"/metrics", [this](coro_http_request& req, coro_http_response& resp) {
HandleMetrics(req, resp);
Expand Down
44 changes: 44 additions & 0 deletions mooncake-store/src/master_service.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2405,6 +2405,50 @@ auto MasterService::ResolveSoftPinRequest(const ReplicateConfig& config) const
return tl::make_unexpected(ErrorCode::INVALID_PARAMS);
}

auto MasterService::BatchUpdateSoftPin(const std::vector<std::string>& keys,
const TenantId& tenant_id,
uint64_t ttl_ms)
-> std::vector<tl::expected<void, ErrorCode>> {
ReplicateConfig config;
config.soft_pin_action = SoftPinAction::ENABLE;
config.soft_pin_ttl_ms = ttl_ms;
auto request = ResolveSoftPinRequest(config);
if (!tenant_id.IsValid() || !request) {
return std::vector<tl::expected<void, ErrorCode>>(
keys.size(), tl::make_unexpected(ErrorCode::INVALID_PARAMS));
}
std::shared_lock<std::shared_mutex> snapshot_lock(master_state_mutex_);
const TenantId normalized_tenant = ResolveRequestTenantId(tenant_id);
std::vector<tl::expected<void, ErrorCode>> results;
results.reserve(keys.size());
for (const auto& key : keys) {
if (key.empty()) {
results.emplace_back(
tl::make_unexpected(ErrorCode::INVALID_PARAMS));
continue;
}
MetadataAccessorRW accessor(this,
ObjectIdentity{normalized_tenant, key});
if (!accessor.Exists()) {
results.emplace_back(
tl::make_unexpected(ErrorCode::OBJECT_NOT_FOUND));
continue;
}
auto& metadata = accessor.Get();
if (accessor.InProcessing() || metadata.pending_soft_pin_action ||
!metadata.HasReplica(&Replica::fn_is_completed)) {
results.emplace_back(
tl::make_unexpected(ErrorCode::REPLICA_IS_NOT_READY));
continue;
}
ApplySoftPinEvaluation(
metadata,
metadata.ApplySoftPin(*request, std::chrono::system_clock::now()));
results.emplace_back();
}
return results;
}

void MasterService::ApplySoftPinMetricDelta(int metric_delta) {
if (metric_delta > 0) {
MasterMetricManager::instance().inc_soft_pin_key_count(metric_delta);
Expand Down
13 changes: 13 additions & 0 deletions mooncake-store/src/rpc_service.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -970,6 +970,19 @@ long WrappedMasterService::RemoveAll(bool force, const std::string& tenant_id) {
return result;
}

std::vector<tl::expected<void, ErrorCode>>
WrappedMasterService::BatchUpdateSoftPin(const std::vector<std::string>& keys,
uint64_t ttl_ms,
const std::string& tenant_id) {
return WithRequestTenantBatch(
master_service_.IsTenantQuotaEnabled() ? std::string_view(tenant_id)
: TenantId::kDefaultValue,
keys.size(), [&](const TenantId& resolved_tenant_id) {
return master_service_.BatchUpdateSoftPin(keys, resolved_tenant_id,
ttl_ms);
});
}

std::vector<tl::expected<void, ErrorCode>> WrappedMasterService::BatchRemove(
const std::vector<std::string>& keys, bool force,
const std::string& tenant_id) {
Expand Down
Loading
Loading