Compare tasks, model versions, and inputs
“Claude versus GPT versus Gemini” is too broad to be a test specification. A model used to summarize filings has a different job from one used to produce a validated decision object. Record the exact model version, prompt, supplied sources, output requirements, and inference settings for each comparison.
Start with the tasks in your workflow. A stock briefing needs evidence-backed summaries. A strategy review needs the ability to identify missing assumptions. An action proposal needs valid fields and explicit account constraints. A model that fits one job may be unnecessary or unsuitable for another.
A small comparison you can reproduce
- Collect representative inputs that you are allowed to use, with source timestamps. Keep a separate set for final evaluation.
- Write a scoring rubric before reviewing answers. Define a correct response, a supported uncertainty statement, and a critical error.
- Run each candidate on identical inputs and output requirements. Keep model-generated confidence separate from your score.
- Review the original responses, not just a model-written ranking. Record unsupported claims, missing facts, invalid output, latency, and cost.
- Repeat the comparison after a material prompt or model-version change. Keep the old configuration available for comparison.
Inspect the output contract
Use the provider's structured-output documentation to configure schema behavior where supported. Check the response semantics separately: valid JSON does not establish that a cited source says what the answer claims.
In NickAI, inspect the model and prompt in the LLM node. Add a Function or Conditional step for deterministic checks. Keep an order node out of a model-comparison experiment; a source-reading evaluation does not need account actions.
Test disagreement before adding consensus
Try the Mag 7 Cross-Provider Jury as a workflow-design reference. Review its individual model steps and how it combines them. The published configuration is evidence of what the workflow is designed to do, not of model superiority.
Use the consensus evaluation guide to decide whether a second model adds useful information. If two models repeat the same unsupported claim, their agreement should remain an evaluation failure.
Correction and scope
Previous numerical model rankings and internal benchmark results on this page lacked a linked, reproducible evaluation. They have been removed. This page offers a comparison procedure, not a finding that one provider is more accurate for trading or produces better financial outcomes.