docs: update notebooks for release - #218
Conversation
Clarify rewrite evaluation guidance and keep notebook licensing metadata out of rendered tutorial content.
Greptile SummaryThis PR refreshes the release tutorial notebooks and their source files. The main changes are:
Confidence Score: 5/5This looks safe to merge after a small license-header cleanup.
docs/notebook_source/*.py Important Files Changed
Reviews (1): Last reviewed commit: "docs: polish release notebooks" | Re-trigger Greptile |
memadi-nv
left a comment
There was a problem hiding this comment.
LGTM.
Nit: The PR description says make check fails on ty, but the current CI Type Check is passing. Could you update the Validation section to reflect the current status, or mention whether that was only a local/environment issue?
| # | ||
| # - Records where automated metrics exceed thresholds are flagged for manual review. | ||
| # - The repair loop stops after `max_repair_iterations`; records that still need | ||
| # repair remain flagged for human review but are not pipeline failures. |
There was a problem hiding this comment.
agent (
review-pr): Repair and human-review flags use different thresholds
This is not always true. _needs_repair uses repair_threshold, while needs_human_review is calculated separately from the final leakage, utility, and high-sensitivity-leak metrics. With risk_tolerance="minimal", for example, leakage mass 0.8 still needs repair because it exceeds 0.6, but it is not flagged solely for leakage because the review threshold is 1.0.
Suggested wording:
“The repair loop stops after max_repair_iterations. Afterward, needs_human_review is computed separately from the final leakage, utility, and high-sensitivity-leak metrics.”
| @@ -145,6 +155,8 @@ | |||
| # ## 🚩 Filter by review flag | |||
| # | |||
| # - Records where automated metrics exceed thresholds are flagged for manual review. | |||
There was a problem hiding this comment.
agent (
review-pr): The threshold direction differs by metric
“Exceed” is accurate for leakage, but utility triggers review when it falls below its threshold. Could we say:
“Records that cross the configured leakage or utility thresholds are flagged for manual review.”
The same wording appears in notebook 05.
| # Call `evaluate()` to run LLM-as-judge scoring on the rewrite result — detection validity and three quality rubrics (privacy, quality, style). | ||
| # Evaluation makes additional LLM calls per record. For larger datasets, evaluate | ||
| # a preview first; this tutorial evaluates all 25 rows to demonstrate the complete workflow. | ||
| # This holistic judge is independent of pipeline leakage scoring, so their assessments may differ. |
There was a problem hiding this comment.
agent (
review-pr): Clarify what may disagree
“Their” has no clear plural antecedent here. I suggest:
“The holistic privacy rubric and pipeline leakage metric are independent, so they may disagree.”
The same sentence appears in notebook 05.
binaryaaron
left a comment
There was a problem hiding this comment.
agent (
review-pr): Correct the review-flag guarantee before merging
The refreshed notebooks are synchronized and the validation checks pass. Before merging, please correct the statement that every record still needing repair remains flagged for human review; the implementation uses separate repair and review thresholds. Please also restore file-level SPDX headers in all five .py notebook sources. The remaining comments tighten threshold wording and avoid an unqualified privacy guarantee.
Related Issue
No linked issue required: this is a docs update for release.
Plan Document
No plan required: this is a docs update for release.
Summary
This PR updates notebooks and notebook_source in the docs preparing for the next release. Includes runs from
.evaluate()and updated instructions.Type of Change
Contributor Checklist
fix: handle empty entity listskills/anonymizer/SKILL.mdupdated if neededValidation
Documentation and Artifacts
make docs-buildpasses locallymake convert-notebooks