Data Quality
ProofPod runs automated quality checks on your test data and surfaces warnings when something looks off. These checks protect you from making decisions on unreliable data.
Automated checks
ProofPod checks for several common data issues:
Sparse data
Some locations have very few observations during the test period. Sparse data means wide confidence intervals and unreliable estimates for those locations.
Zero variance
A location's metric shows no variation at all (e.g., exactly $0 every day). This usually indicates missing data or a closed location rather than genuinely flat performance.
Group imbalance
The test and control groups have significantly different sizes or baseline characteristics. Some imbalance is normal, but extreme imbalance weakens the analysis.
Temporal issues
Gaps in the time series, inconsistent reporting frequencies, or insufficient pre-period data for establishing a baseline.
Severity levels
Each warning is tagged with a severity:
- Info — worth noting but unlikely to affect results
- Warning — may affect result quality; review before acting on recommendations
- Critical — serious enough to trigger a Kill recommendation regardless of the primary metric's performance
Impact on recommendations
Data quality feeds directly into the recommendation engine:
- Critical issues can override a positive primary metric signal and force a Kill recommendation. This is a safety mechanism—ProofPod won't tell you to scale based on data it doesn't trust.
- Warning-level issues appear in the recommendation checklist so you can factor them into your decision.
What to do when warnings appear
| Warning | Action |
|---|---|
| Sparse data | Check if affected locations are reporting correctly. Consider extending the test duration. |
| Zero variance | Verify the location is open and transacting. Remove it from the test if it's closed. |
| Group imbalance | Review location assignments. Consider re-running matching with different parameters. |
| Temporal gaps | Check your data source connection. Ensure syncs are running on schedule. |
Most data quality warnings are informational and don't require action. Focus on critical warnings first—those are the ones that can change your recommendation.