ScoreTrace AI: Forecast Track Record Auditor — Measure the Calls, Not the Hype

Forecasts are easy to remember when they are right and easy to lose track of when they are wrong. A confident post, a strong recent streak, or a handful of well-timed calls can create an impressive story. The harder question is whether a complete, dated record supports that story.

By Genial Things Editorial Team · AI Agent Skills · Updated October 2026 · 9 min read

ScoreTrace AI is a forecast track record auditor. Give it public, dated calls and their outcomes, and it reports hit rates, uncertainty, comparison with a baseline, probability calibration, returns for directional calls, and flags that affect how the record should be interpreted.

In this guide you will learn what ScoreTrace AI measures, which record-quality problems it flags, how its report is produced, and where its limits are. You can open ScoreTrace AI on Capafy at any point and run an audit as you read.

What Is ScoreTrace AI?

ScoreTrace AI organizes dated, falsifiable calls, checks their outcomes, and summarizes the evidence in a structured report. The analysis can cover stock-style directional calls, football predictions, and other public calls with a clear outcome and time horizon.

The core distinction is between what someone said and what a verifiable record shows. ScoreTrace does not treat a collection of confident statements as proof. It requires specific calls, dates, sources, a defined horizon, and outcomes that can be checked.

A deterministic analysis script validates the records and calculates the metrics. The agent prepares the input from the material supplied or from public sources available to its runtime, then explains the script's output. The script itself makes no network calls.

Why Audit a Forecast Track Record?

Public forecasts appear across posts, videos, newsletters, and interviews. Without a consistent record, it is hard to know which calls were specific enough to test, which later expired without a result, and whether unsuccessful calls are represented alongside successful ones.

Even a correctly calculated hit rate can mislead without context. A 60% hit rate may be compatible with a 50% baseline when the sample is small. A strong average return may depend heavily on a few calls. A forecast that says "70% likely" should also be judged on calibration, not just on whether its latest call happened to succeed. ScoreTrace reports several views together:

• Accuracy: how many resolved calls were correct?
• Uncertainty: how wide is the confidence interval around the observed hit rate?
• Baseline: is the result distinguishable from the comparison rate for this type of call?
• Calibration: when probabilities were stated, did events occur at roughly those rates?
• Returns: for directional calls, what were the gross directional and benchmark-relative returns?
• Evidence quality: are calls sourced, outcomes supported, and the record confirmed complete?

The report describes only the supplied sample. ScoreTrace cannot recover statements that were deleted, never public, or never included in the dataset. This is the same problem statisticians call survivorship bias.

What ScoreTrace Measures and Flags

ScoreTrace separates directional calls, such as a named asset being higher or lower by a specified date, from binary calls, such as whether a team will win. The comparison baseline should fit the question. The default is 50%, but a three-outcome football prediction should use an appropriate base rate rather than automatically treating chance as 50%.

Accuracy and uncertainty

Hit rates are calculated separately by call type. Each rate includes a 95% Wilson interval. An exact one-sided binomial test compares the observed result with the chosen baseline. Fewer than 30 resolved calls produces an insufficient-sample verdict rather than a skill verdict.

Probability calibration

When the forecaster stated a probability, ScoreTrace can calculate a Brier score and summarize results in probability bins. It does not assign probabilities to calls that never included one. Calibration is assessed only when enough probability calls exist for the configured checks, and individual bins also require a minimum count.

Returns for directional calls

Where price data is supplied or verifiable, the report can include directional returns, excess return versus a supplied benchmark, average win/loss size, and the share of total directional return contributed by the three best calls. Returns are gross of costs, slippage, and fees. They are not a real-world portfolio simulation.

The nine record-quality flags

Common Track Record Problems

Only the memorable calls are supplied

A list that is not confirmed complete may omit unsuccessful calls. ScoreTrace labels the record accordingly and warns that the figures may be overstated. It cannot independently prove that every public call from the period has been captured.

Vague opinions are counted as predictions

"Bullish long term" has no defined outcome date and may not specify a testable direction. ScoreTrace requires public, dated, specific, falsifiable, time-bounded calls. Vague or unverifiable statements are excluded with a reason rather than turned into scored predictions.

The horizon changes after the call

A forecast without an end date needs a consistent horizon before scoring. If the subject did not state one, ScoreTrace asks the user to choose a horizon, applies it consistently, and discloses that choice. It does not silently infer a convenient date.

A few winning calls dominate returns

An average or total return can be driven by a small number of unusually successful calls. ScoreTrace reports the contribution from the three best directional calls so readers can see when the rest of the sample offset or diluted those gains.

A hit rate is treated as proof

Every observed hit rate has sampling uncertainty. A result that is not distinguishable from its baseline at the available sample size should not be presented as proof of skill. ScoreTrace uses neutral wording and always states that the analysis describes the supplied sample only.

How ScoreTrace AI Produces Its Report — Step by Step

1. Define the subject and scope. Provide a public forecaster's name or handle, domain, and the period to review. Private individuals and private or paid-group material are out of scope.
2. Collect candidate calls. Provide public links, pasted posts, or a CSV/JSON record. Each included call needs an attributable source, date, specific claim, and time horizon.
3. Verify or supply outcomes. Outcomes can come from user-provided data or public sources reachable through the host runtime. Missing outcomes remain open or unverified; the agent does not estimate them.
4. Run deterministic calculations. A standard-library Python script validates rows, identifies resolved, open, unverified, push, and rejected calls, then calculates metrics and flags without network access.
5. Review the report. The report presents results, evidence gaps, exclusions, and limitations. It is a statistical description, not a recommendation to follow or oppose the forecaster.

What the ScoreTrace AI Report Includes

• Executive summary: plain-language verdict per call type, number of resolved calls, and the leading flags.
• Scope and sources: subject, period, collection method, timestamp, tools used, and whether the supplied record was confirmed complete.
• Headline statistics: resolved calls, hits, hit rate, confidence interval, baseline, and statistical comparison.
• Returns and calibration: directional returns and benchmark comparison where available; Brier score and probability bins for stated probabilities.
• Flags and category breakdown: record-quality cautions and categories with enough calls to report.
• Per-call appendix: individual records with source links and status, including exclusions and calls that could not be verified.
• Machine-readable JSON: structured results alongside the human-readable report.

Example output from the bundled fictional sample

The package includes a sample audit of a fictional forecaster. These are demonstration data, not results for a real forecaster.

Who Benefits from ScoreTrace AI?

• Researchers and journalists who need to describe a public forecaster's record with links and caveats.
• Market observers who want to compare public stock or macro calls with their outcomes without treating the report as investment advice.
• Sports analysts and audiences who want to evaluate published match predictions against a suitable baseline.
• Creators and publishers who want a sourced, reviewable record of their own public forecasts.
• Readers evaluating claims who want to distinguish a documented sample from selective highlights.

ScoreTrace AI — Frequently Asked Questions

Final Thoughts and Related Tools

ScoreTrace AI is part of a family of evidence-focused Skills from Genial Things. GEOTrace AI focuses on AI-answer visibility, while CrawlTrace AI focuses on AI crawlability. Each is designed around a separate audit question. Browse the full collection in our Free AI Online Tools.