fix: match Lotus's behaviour with replace by fee in mpool

[LesnyRumcajs]

Summary of changes

Changes introduced in this pull request:

• matched Lotus behavior on replace by fee logic in the memory pool.
• https://github.com/filecoin-project/lotus/blob/5f32d00550ddd2f2d0f9abe97dbae07615f18547/chain/messagepool/messagepool.go#L296-L299

Reference issue to close (if applicable)

Part of #6526; it's a step in the good direction but might not resolve it completely just yet.

Other information and links

Change checklist

• I have performed a self-review of my own code,
• I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
• I have added tests that prove my fix is effective or that my feature works (if possible),
• I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

Outside contributions

• I have read and agree to the CONTRIBUTING document.
• I have read and agree to the AI Policy document. I understand that failure to comply with the guidelines will lead to rejection of the pull request.

Summary by CodeRabbit

• Bug Fixes

  • Fixed replace-by-fee behavior when the mempool is at capacity: users can now replace pending transactions with higher fees without being incorrectly rejected due to capacity checks.

• Documentation

  • Added changelog entry noting the fix.

[coderabbitai]

Walkthrough

Allows replace-by-fee (RBF) replacements in the message pool even when pending message capacity is reached; changelog entry added. Implementation adjusts the pending-message limit check to apply only to new messages, not replacements.

Changes

Cohort / File(s)	Summary
Documentation CHANGELOG.md	Added unreleased changelog entry: fixed incorrect replace-by-fee behavior when at limits of pending messages in mempool.
Message Pool Implementation & Tests src/message_pool/msgpool/msg_pool.rs	Introduce has_existing flag in MsgSet::add; apply pending-message capacity check only for new messages (not replacements); add test_rbf_at_capacity verifying RBF allowed at capacity.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested reviewers

• akaladarshi
• hanabi1224

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'fix: match Lotus's behaviour with replace by fee in mpool' directly describes the main change: modifying replace-by-fee logic in the memory pool to match Lotus behavior, which aligns perfectly with the code changes and PR objectives.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch replace-by-fee-limit-fix

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

❌ Patch coverage is 97.05882% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 63.36%. Comparing base (83aa632) to head (51bc4fb).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/message_pool/msgpool/msg_pool.rs	97.05%	0 Missing and 1 partial ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
src/message_pool/msgpool/msg_pool.rs	78.57% <97.05%> (+3.51%)	⬆️

... and 2 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 83aa632...51bc4fb. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@CHANGELOG.md`:
- Around line 64-65: Update the changelog entry to use the issue-first format:
replace the PR link [`#6535`](https://github.com/ChainSafe/forest/pull/6535) with
the issue link for `#6526` using the format
[`#6526`](https://github.com/ChainSafe/forest/issues/6526): keep the existing
description "Fixed incorrect replace by fee behavior when at limits of pending
messages in mempool." but reference the issue instead of the PR to match the
project's CHANGELOG.md convention.