- ScoreTrace Ledger is a local AI agent skill for Capafy that records public forecasts in a tamper-evident append-only ledger, verifies integrity using a hash chain, and scores each forecaster's track record.
- It answers two core questions: how have these forecasters performed on recorded calls? And when forecasters disagree, how does their track-record-weighted consensus compare to the simple average?
- Scoring uses hit rate with a 95% confidence interval and a Brier score for probabilistic forecasts — with a minimum-30-call rule before drawing skill conclusions.
- The hash chain proves when an entry was recorded and detects subsequent changes — but does not independently prove who made a forecast or when it was originally published elsewhere.
- Runs entirely locally via a Python standard-library script — no API key, no account, no network calls. Export reports in JSON, CSV, or Markdown.
- ScoreTrace Ledger is a recordkeeping and scoring tool, not a prediction engine. It does not predict events, guarantee consensus accuracy, or provide financial advice.
When someone makes a prediction, it is easy to remember the times they were right and forget the times they were wrong. Confidence sounds convincing — but confidence alone does not tell you whether a forecaster has a reliable track record. ScoreTrace Ledger is an AI agent skill designed to change that: a tamper-evident, locally-run tool that records public forecasts, verifies the ledger, scores performance, and compares simple averages with track-record-weighted consensus.
ScoreTrace Ledger is a local AI agent skill for Capafy that records public forecasts in a tamper-evident append-only ledger with hash-chain integrity verification. It scores forecasters by hit rate (with 95% confidence interval) and Brier score, assigns track-record weights, and compares a naive simple-average consensus against a weighted consensus. It runs locally via Python with no API key required and exports reports in JSON, CSV, or Markdown.
Verify the Record.
Examine the Evidence.
🔍 What Is ScoreTrace Ledger?
ScoreTrace Ledger is a forecasting track-record tool for Capafy. It uses a local Python script to record public forecasts and their outcomes in an append-only tamper-evident ledger. It is designed to answer two questions: how have recorded forecasters performed on their calls? And when forecasters disagree on an unresolved question, how does a track-record-weighted consensus compare to the simple average? The agent does not generate predictions — it works with forecasts supplied by the user.
💡 The core problem ScoreTrace Ledger solves: When someone makes a prediction, it is easy to remember their hits and forget their misses. A single memorable correct call can create a misleading impression of consistent accuracy. ScoreTrace Ledger replaces that selective memory with a structured, verifiable record — one where past calls cannot be quietly edited away, and performance is measured across the full documented history, not just the memorable ones.
⚙️ How ScoreTrace Ledger Works — 3 Steps
ScoreTrace Ledger follows three structured steps: (1) Record forecasts before they resolve — capturing forecaster name, question, date, resolution date, claim, source, and probability. (2) Record outcomes when they become known — appended to the ledger rather than replacing prior entries. (3) Verify the ledger — a hash chain linking each entry to its predecessor detects any edits, insertions, or reordering of existing entries.
Record Forecasts Before They Resolve
Each forecast requires structured input so the record is checkable. ScoreTrace Ledger does not invent missing dates, probabilities, or claims. If a statement is vague or lacks a clear resolution date, the agent asks for clarification rather than treating it silently as a checkable prediction.
Record Outcomes When They Become Known
Once a question resolves, the user supplies the outcome and its source. ScoreTrace Ledger appends the outcome — it does not replace the original forecast entry. If an outcome was recorded incorrectly, it is corrected through a new correction entry; the earlier entry remains visible. A reliable track record preserves the history of what was recorded, including corrections.
Verify the Ledger with the Hash Chain
Every ledger entry includes a timestamp and a hash linked to the preceding entry. Editing, inserting, or reordering entries can cause verification to fail. The ledger is append-only — users should not manually edit or rebuild it. To make deletion of entries from the end detectable, ScoreTrace Ledger advises saving the latest hash and entry count somewhere separate.
⚠️ Important limitation on the hash chain: The hash chain proves when an entry was recorded in the ledger and detects subsequent changes to existing entries. It does not independently prove who made a forecast or when it was first published elsewhere. External evidence — a dated public post, a timestamped article — is what establishes the original claim. The ledger records what was submitted to it and protects that record from subsequent alteration.
📈 How ScoreTrace Ledger Measures Forecaster Performance
ScoreTrace Ledger calculates three performance measures after the ledger passes verification: hit rate (proportion of scored calls matching outcomes, with a 95% confidence interval), Brier score (probability calibration for forecasters who state probabilities — lower is better), and track-record weight (earned weight based on recorded performance, calculated with a minimum-sample correction). Fewer than 30 scored calls are labelled insufficient to draw skill conclusions.
Hit Rate with 95% Confidence Interval
The proportion of scored calls that matched their outcomes. ScoreTrace Ledger adds a 95% confidence interval to show how uncertain the estimated performance remains given the sample size. When fewer than 30 calls have been scored, it labels the sample as insufficient to draw a skill conclusion — avoiding overconfident claims from small samples.
Minimum 30 calls before skill verdictBrier Score — Probability Calibration
For forecasters who provide probabilities (rather than simple yes/no calls), ScoreTrace Ledger calculates a Brier score measuring how closely the probabilities align with outcomes. Lower Brier scores indicate better calibration. A forecaster who repeatedly assigns 90% to events that occur 60% of the time will have a worse Brier score than their hit rate alone would reveal.
Lower = better · Penalises miscalibrationTrack-Record Weight
Each forecaster earns a weight based on their recorded performance. The calculation adds ten imaginary calls at 50% before estimating the weight, reducing the influence of very small samples. A forecaster with five correct calls out of five does not automatically receive overwhelming weight — their record is still too small to justify high confidence.
Small-sample correction appliedBackfill Detection
Calls added after the outcome was already known are identified as backfilled and do not earn weight under the standard method. This matters because allowing post-hoc entries to inflate a forecaster's score would undermine the entire purpose of maintaining a disciplined track record. The report flags these entries separately.
Post-resolution entries flagged⚖️ Simple Consensus vs. Track-Record-Weighted Consensus
Naive consensus is the simple average of all forecasters' predictions — every forecaster contributes equally regardless of their track record. Weighted consensus combines forecasts using the weights each forecaster has earned from their recorded performance — giving more influence to those with stronger demonstrated accuracy. If no forecaster has earned a positive weight, ScoreTrace Ledger reports that no weighted consensus is available rather than inventing one.
| Forecaster | Probability of Event | Earned Weight |
|---|---|---|
| Alex | 70% | 0.20 |
| Morgan | 40% | 0.05 |
| Sam | 60% | 0.15 |
📊 Why the weighted consensus matters: When forecasters have meaningfully different track records, giving each equal weight treats a forecaster with a strong 200-call history the same as one with a weak 5-call history. The weighted approach allocates influence proportional to demonstrated accuracy — while ScoreTrace Ledger's small-sample correction ensures that a newcomer with three lucky calls doesn't dominate the consensus either. If no forecaster has earned a positive weight, no weighted consensus is reported.
📄 What Reports Does ScoreTrace Ledger Produce?
ScoreTrace Ledger produces reports in three formats: JSON (structured output for further analysis or other tools), CSV (spreadsheet-friendly for reviewing forecasters and scores), and Markdown (readable summaries with performance tables, consensus comparisons, and flags). All reports include integrity status, scored calls, hit rates, confidence intervals, Brier scores, track-record weights, open-question consensus, and relevant warnings.
| Flag / Warning | What It Means | Impact |
|---|---|---|
| Small sample | Fewer than 30 scored calls — skill verdict withheld | ⚠️ Caution |
| Post-horizon forecast | Forecast recorded after its resolution date | 🚩 Flagged |
| Overdue question | Resolution date passed — no outcome recorded yet | ⏳ Pending |
| Corrected outcome | An earlier outcome was corrected — both entries visible | 📝 Noted |
| Missing source | Forecast has no source citation | ⚠️ Caution |
| Concentrated weight | Weighted consensus rests on a very small group | ⚠️ Caution |
| Backfilled entry | Call added after the outcome was already known | 🚩 Excluded from weight |
| No earned weight | No forecaster qualifies — weighted consensus unavailable | ℹ️ Reported |
🔒 Runs Locally — No API Key, No Network Calls
ScoreTrace Ledger includes a Python script using only the standard library. The ledger and all report calculations run locally on the user's machine — the script makes no network calls. No API keys, passwords, cookies, or account access are required. Users can choose Download (local) mode to keep their ledger on their own device. The ledger may contain names, public handles, claims, dates, probabilities, and source links — treat it as private data.
🔒 Why local execution matters for this use case: A forecasting track record may contain public names, claims attributed to specific accounts, and probability figures. Keeping that data on the user's own machine — rather than transmitting it to a third-party API — is a meaningful privacy and control advantage. The AI agent may help prepare forecast rows from public pages the user links, using its approved tools, but the ledger itself stays local.
🚫 What ScoreTrace Ledger Does NOT Do
ScoreTrace Ledger is a recordkeeping and scoring tool — not a prediction engine. It does not predict whether events will happen, guarantee that weighted consensus will be more accurate, prove a forecaster is honest or dishonest, establish when a public forecast was originally made, or provide financial, investment, or betting advice. Its weights describe the recorded sample only — past performance may not continue.
🚫 ScoreTrace Ledger does not: predict events · guarantee consensus accuracy · prove forecaster honesty · establish original publication dates of claims · provide financial or investment advice · calculate stock-style returns (a separate tool handles that). Its weights describe the recorded sample, not future accuracy. A weighted consensus can still be wrong. Use it as structured evidence — not as a certainty machine.
Preserve their original details.
Add outcomes with sources.
Examine the results only after the ledger passes verification.
👤 Who Can Use ScoreTrace Ledger?
ScoreTrace Ledger is designed for researchers examining public forecasting records, analysts comparing published predictions, communities tracking public forecast accuracy, and individuals maintaining an organized record of their own predictions. It is particularly useful when forecasts are dated, specific, and checkable against clear outcomes. It is intended for public figures, public accounts, and published analysts — not for compiling records about private individuals.
Researchers
Examine public forecasting records with a structured, verifiable methodology rather than selective recall.
Analysts
Compare published predictions over time, score calibration, and build evidence-based assessments of forecaster reliability.
Communities
Track the accuracy of public accounts or commentators in a domain — politics, economics, technology, sports.
Individuals
Keep an organized, disciplined record of your own predictions and measure your calibration over time.
Consensus Evaluators
Compare simple average consensus against weighted consensus when multiple forecasters disagree on an open question.
Privacy-Conscious Users
Keep all ledger data on your own machine — no account required, no API transmission, full local control.
❓ ScoreTrace Ledger — Frequently Asked Questions
Related AI Tools & Resources from Genial Things
ScoreTrace Ledger is part of a broader AI agent skill ecosystem on Capafy. These related guides connect forecasting, data analysis, and digital intelligence:
