Performance
How well does the model do the job it was built for?
- examines
- Task-specific predictive or decision performance.
- keep in mind
- A score on one split says little about data the model has not seen.
Open source, MIT License
20K+library downloads
Your model is more than its accuracy.
Audit machine learning models across performance, robustness, explainability, calibration, data quality, fairness, and complexity — through one extensible Python interface.
Why
One good number can hide a weak model.
Accuracy tells you how often a model is right on one dataset. It does not tell you whether its probabilities can be trusted, whether small changes in the input break it, whether the data was clean, or whether it behaves differently for different groups.
ai-critic looks at those questions one by one and writes the answers into a report you can read, with the limits of each check stated next to its result.
Same accuracy, different auditillustrative, synthetic data
| Model A | Model B | |
|---|---|---|
| Accuracy | 0.91 | 0.91 |
| Calibration error | low | high |
| Under input perturbation | stable | score drops |
| Data quality | clean | duplicates found |
A single metric would rate these two models as equal. An audit would not.
Transparency
Results come with explanations and limitations, so you can see what a check does not cover.
Technical depth
Meaningful metrics and diagnostics, with reproducible examples and fixed random seeds.
Practicality
One call fits into existing Python and scikit-learn workflows. No new pipeline to adopt.
Method
Four steps from a fitted model to a report.
You keep your own training code. ai-critic starts after the model is fitted.
- 1
Provide a fitted model and evaluation data
Any model you already trained, plus held-out data such as
(X_test, y_test). - 2
Declare the task type
Set
task_typeso the audit chooses strategies for your problem instead of assuming a classifier. - 3
Run the audit
Call
ai_critic.audit. An optionalEvaluationConfigsets cross-validation folds and a random seed for reproducibility. - 4
Inspect metrics, statuses, and limitations
Read the overall result first, then open each dimension to see what was measured and what was not.
import ai_critic
from ai_critic import EvaluationConfig
report = ai_critic.audit(
model,
(X_test, y_test),
task_type="classification",
evaluation=EvaluationConfig(
cv_folds=5,
random_state=42,
),
)
print(report.overall_score)
print(report.overall_verdict.value)- report.overall_score
- A summary score for the whole audit. Use it as a starting point, not a verdict.
- report.overall_verdict.value
- The overall verdict as a value you can print or log.
Dimensions
Seven questions you can ask about a model.
Each dimension asks one question, states what it examines, and tells you what to keep in mind when reading the result.
Robustness
Does the result survive small changes to the input?
- examines
- Sensitivity to supported perturbations.
- keep in mind
- Only supported perturbations are tested. Real-world shift can look different.
Explainability
Which inputs drive the model's output?
- examines
- Feature contributions and other supported diagnostics.
- keep in mind
- A feature that contributes is not necessarily a cause.
Calibration
When the model says 80%, is it right about 80% of the time?
- examines
- Alignment between predicted probabilities and observed outcomes.
- keep in mind
- It only makes sense for models and tasks that produce probabilities.
Data Quality
Is the data itself in good shape?
- examines
- Missing values, duplicates, constant columns, and other supported checks.
- keep in mind
- Clean data is not the same as representative data.
Fairness
Does behavior differ across groups you name?
- examines
- Differences across explicitly defined groups.
- keep in mind
- Without defined groups, nothing is compared. That says nothing about fairness either way.
Complexity
How large and costly is the model?
- examines
- Model structure and supported computational diagnostics.
- keep in mind
- What can be measured depends on what the model exposes.
Tasks
The audit follows the task you declare.
A regression model has no class probabilities. A clustering model has no labels. Instead of treating every model as a classifier, ai-critic picks strategies from task_type and marks a dimension not_applicable when it does not fit.
Report
A report you can read, not only a score.
Each dimension shows its status, its metrics, a plain explanation, and its limitations. Select a row to open it.
overall_score0.82 / 1.00
Synthetic data. Real reports depend on your model, your data, and the installed version.
Limits
What ai-critic does not do.
An audit is a tool for examining a model. The decision to use it stays with you.
- It does not guarantee fairness.It compares groups you define. It cannot decide which groups matter.
- It does not certify safety or compliance.A report is evidence to review, not a legal or regulatory finding.
- It does not declare a model production ready.Deployment depends on context, data drift, and risks outside the audit.
- It does not hide what it could not assess.Unsupported checks are reported as not applicable or inconclusive.
Install
Install it and run your first audit.
One command installs the package. Then use the example above with your own fitted model.
$python -m pip install ai-critic
Requires Python and a model you have already fitted.
Open source
MIT License
Built in the open, with its limits in view.
ai-critic is an open-source Python project. Evaluation methods have limits, and discussing them openly makes the tool more honest.
Explore the code
Read how each evaluator works in the repository.
Report bugs
Open an issue with a minimal example.
Suggest improvements
Propose changes to the API, the reports, or the docs.
Contribute evaluators
Add evaluators and task strategies. Contribution guidelines will be linked once published.
Discuss limitations
Question the methodology. Clear limits are part of the design.