Repository navigation
feat(miner): rfc 0023 green pt1 — bounded template memory (RFC0023.1–.5) #354
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
3905f7b
feat(miner): rfc 0023 green pt1 — the three bounded-memory knobs
jensholdgaard 4775981
refactor(miner): one emit_string_parse_failure helper for every §6.3 …
jensholdgaard aee0808
test(miner): review round — tree-level wildcard tests, fanout proptes…
jensholdgaard 6c6f9ea
test(miner): land the below-floor line in the wildcard bucket it clai…
jensholdgaard File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
56 changes: 50 additions & 6 deletions
56
crates/ourios-ingester/tests/rfc0023_overflow_roundtrip.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,16 +1,60 @@ | ||
| //! RFC 0023 §5 — RFC0023.2: ceiling-overflow lines stay stored and | ||
| //! searchable. The miner-level scenarios (`.1/.3/.4/.5/.6`) live in | ||
| //! `crates/ourios-miner/tests/rfc0023_bounded_memory.rs`; this one | ||
| //! crosses the ingest path (miner overflow → record sink → Parquet → | ||
| //! body read-back), which needs both crates. | ||
| //! crosses the ingest path (miner overflow → Parquet → body | ||
| //! read-back), which needs both crates. | ||
|
|
||
| use ourios_core::config::MinerConfig; | ||
| use ourios_core::otlp::{Body, OtlpLogRecord}; | ||
| use ourios_core::record::SharedRecordSink; | ||
| use ourios_core::tenant::TenantId; | ||
| use ourios_miner::cluster::{MinerCluster, NO_TEMPLATE}; | ||
| use ourios_parquet::{DEFAULT_ZSTD_LEVEL, Reader, encode_records_to_parquet}; | ||
|
|
||
| /// Scenario RFC0023.2 — overflow lines stay stored and searchable. | ||
| /// See `docs/rfcs/0023-bounded-template-memory.md` §5. | ||
| #[test] | ||
| #[ignore = "RFC0023.2 stub — implemented in the miner-bounds green slice (as its ingest-path integration arm)"] | ||
| fn rfc0023_2_overflow_bodies_round_trip_bit_identically() { | ||
| todo!( | ||
| "RFC0023.2 — ceiling-overflow lines written through the record sink \ | ||
| round-trip bit-identically through the Parquet body column" | ||
| let tenant = TenantId::new("overflow"); | ||
| let sink = SharedRecordSink::new(); | ||
| let config = MinerConfig::default() | ||
| .with_max_templates(1) | ||
| .expect("non-zero ceiling"); | ||
| let mut cluster = MinerCluster::new(config).with_record_sink(Box::new(sink.clone())); | ||
|
|
||
| // Distinct shapes: the first mints (and fills the ceiling), the | ||
| // second diverts to the parse-failure path with its body retained. | ||
| let overflow_line = "gamma rejected request from peer five with cause umbrella"; | ||
| for text in ["alpha started", overflow_line] { | ||
| cluster.ingest(&OtlpLogRecord { | ||
| tenant_id: tenant.clone(), | ||
| body: Some(Body::String(text.to_string())), | ||
| time_unix_nano: 1_775_127_480_000_000_000, | ||
| ..Default::default() | ||
| }); | ||
| } | ||
| let mined = sink.drain(); | ||
| assert_eq!(mined.len(), 2); | ||
| assert_eq!(mined[1].template_id, NO_TEMPLATE, "second line diverted"); | ||
|
|
||
| // Through the RFC 0005 writer and back: the overflow body is | ||
| // bit-identical in the Parquet body column. | ||
| let bytes = encode_records_to_parquet(&mined, DEFAULT_ZSTD_LEVEL).expect("encode"); | ||
| let dir = tempfile::TempDir::new().expect("temp dir"); | ||
| let path = dir.path().join("overflow.parquet"); | ||
| std::fs::write(&path, bytes).expect("write parquet"); | ||
| let rows = Reader::open_file(&path) | ||
| .expect("open parquet") | ||
| .read_all() | ||
| .expect("read rows"); | ||
| assert_eq!(rows.len(), 2); | ||
| let overflow_row = rows | ||
| .iter() | ||
| .find(|r| r.template_id == NO_TEMPLATE) | ||
| .expect("the diverted row is stored"); | ||
| assert_eq!( | ||
| overflow_row.body.as_deref(), | ||
| Some(overflow_line), | ||
| "overflow body round-trips bit-identically", | ||
| ); | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.