One useful idea
JSONL stores one complete JSON value per physical line. A generator uses yield to return the next event lazily. Binary readline(size) lets you bound bytes and diagnose invalid UTF-8 per line. A with block keeps the file open while the generator is suspended; exhaustion, failure or explicit generator.close() closes it. If you stop early, close the generator yourself.
Reader events have exactly line, outcome, record and error_code. Number every physical line from 1, including blank, malformed, duplicate and oversized rows. Accepted has a fresh preference and None error; quarantined has record=None and one controlled code: blank, encoding, json, schema, duplicate or line_too_large. Keep raw private row text out of diagnostic events. The first accepted ID owns identity; a prior invalid row does not consume it.
Strict UTF-8 and JSON parsing refuse BOMs, duplicate keys, nonfinite numbers, integers over 64 digits and nesting over 32; quoted brackets are still ordinary text. The 65,536-byte line budget includes a CR/LF terminator when present. Drain an overlong physical line in bounded chunks so its next line keeps the correct number. More than 100,000 rows raises rather than fabricates a successful final report. The seen-ID set grows with accepted rows: streamed records do not mean constant total memory or a global CPU deadline.
Writer opens exclusively with xb before consuming the iterable, so existing files and known links are refused. Validate and serialize one compact UTF-8 row at a time, preserving Unicode and rejecting duplicate IDs. Return actual rows/bytes/sha256 only after successful close. A later failure leaves a new partial file visible—no rollback, overwrite or deletion. The receipt alone is not transactional durability or typed read-back. The separate demo generates and verifies 10,000 actual ordered rows in a new folder.
Refresh first: Validated preference records, JSONL datasets, New-folder output and typed read-back.
Trace a finished example
import json
from pathlib import Path
from tempfile import TemporaryDirectory
from dataset_tools.core import read_jsonl, write_jsonl
row = {"schema_version": 1, "record_id": "pair_1",
"prompt": "Café at dusk — 海", "chosen_id": "asset_a",
"rejected_id": "asset_b", "verdict": "prefer_chosen",
"reason": "Cleaner horizon", "evaluator": "reviewer_1",
"created_at": "2026-10-07T12:00:00Z"}
with TemporaryDirectory(prefix="dvp-jsonl-example-") as temporary:
path = Path(temporary) / "preferences.jsonl"
receipt = write_jsonl(path, (dict(row, record_id=f"pair_{n}") for n in (1, 2)))
print(receipt["rows"])
with path.open("ab") as stream:
stream.write(b"{broken}\n")
stream.write((json.dumps(row, ensure_ascii=False) + "\n").encode("utf-8"))
for event in read_jsonl(path):
print(event["line"], event["outcome"], event["error_code"])Two generated preferences are written into an owned temporary file. The example then appends malformed JSON and a repeated accepted ID only to that copy. Four physical lines yield four events; the diagnostic contains a controlled reason, not raw source text. Leaving the context cleans this temporary example, not your project outputs.
The finished implementation is in dataset_tools/core.py. Reading it is guided practice, not independent evidence.
Predict line accounting
A blank line appears between valid records. Should the reader renumber later records as if it never existed?
Compare your answer · self-reviewed
No. Blank lines are physical input and produce their own quarantined event. Later events keep the true source line numbers.
Find the partial-output claim
A duplicate arrives after one successful write. Is the new file rolled back?
Compare your answer · self-reviewed
No. The failure propagates and leaves a visible partial new file without a success receipt. Investigate it and choose a different new destination.
Recall memory
Does yielding rows prove total memory is constant?
Compare your answer · self-reviewed
No. The duplicate-ID set grows. Bounded chunks and row limits are explicit policies, not universal memory or timing guarantees.
Change it, then build your own
One controlled change
Append a blank line and a different valid record to an owned copy; predict all physical numbers and outcomes. Change the source to invalid UTF-8 and an oversized line, keeping the originals untouched.
Your independent task
Implement read_jsonl and write_jsonl in practice.py. Reviewed strict JSON and path helpers may be reused; disclose them, but write your own generator, duplicate policy, bounded-drain logic and actual write/receipt flow. Keep one event per physical line and expose unexpected I/O/programming failures. Demonstrate the 10,000-row ordered typed round trip in a NEW output folder; never call the reference task functions as your own work.
What success looks like
The build2 group checks physical numbering, Unicode, strict parsing, bounded oversized-line continuation, lazy resource closure, duplicate/row limits, real 10,000-row disk output, partial-file failure and no overwrite. Your demo must separately compare actual bytes/hash and each typed ordered row before reporting verified_rows.
Hint 1 · a question
Write down a four-line mixed fixture and the event for each physical line. Separate parsing refusal from an unexpected program failure.
Hint 2 · a concept cue
Use bounded binary reads, drain only the refused physical line and retain accepted IDs. Wrap the open stream in with, with yield inside its lifetime.
Hint 3 · a localized example
path.open("xb") refuses an existing output. It does not roll back later writes; the receipt must come only after successful completion.
Need the complete worked solution?
Open dataset_tools/core.py from the kit. Trace it, close it, then try fresh inputs in your own files. Treat the attempt as guided; seeing the solution does not award a practical pass.
Course help is guidance, not independent evidence. With JavaScript, opening help records guidance locally; otherwise note it in your README. Reset does not erase that history.
Repair a failed check
If the reader skips blanks or drains into the next row, trace each binary read and physical counter. If an existing output consumes the iterator, open exclusively first. If a failure prints a successful receipt, move success reporting after the entire operation. If files stay open after early departure, explicitly close the generator.
NotImplementedError means a practice stub is still unfinished. Read the failing test name and the last error line. Change one behavior, rerun that build, then rerun all implemented builds.
Show it works on new inputs
Create a new mixed fixture with Unicode, a blank, malformed JSON, a duplicate and a changed valid ID. Reconcile physical lines=accepted+quarantined. Run your 10,000-row round trip in another new folder and explain growing ID memory, partial-output limits and diagnostic redaction.
Self-review: name the input, result, refused case and reason. Your local test output and explanation are separate from a quiz score; this page does not certify a pass.
Keep the idea
Streaming preserves accounting while limiting row buffering. Honest failure receipts and physical numbering matter as much as the happy path.