Evolution of Data Quality: Experience in Transitioning from Local Checks to a Platform-Based Solution and Automation

In our tax reporting automation project, Data Quality was a critical concern from the start. When deadlines are tight, analysts and testers are forced to hunt down data errors manually under pressure—and in banking, the cost of such mistakes can be extremely high, leading to significant financial consequences.

Initially, we took a pragmatic approach: we dedicated some team resources to local DQ checks to reduce the workload and accelerate issue detection. We used the PyDeequ framework, which allowed us to quickly formalize critical checks alongside calculations and achieve measurable results. Spoiler: although our solution was somewhat "quick-and-dirty," it truly saved time, helped stabilize calculations, and ensured we met our delivery deadlines.

Next, at the project level, we implemented a platform-level solution—a full-fledged DQ service based on Soda Core. We conducted a thoughtful comparison of approaches and tools: where PyDeequ is stronger, where Soda is more convenient, and which types of checks and operational scenarios each covers better—summarizing our findings in a clear comparison table.

Ultimately, we didn't choose "either PyDeequ or Soda." In our case, synergy delivered the maximum effect: PyDeequ handled specific tasks as a "local quality control tool," closer to the calculations and engineering logic of the pipeline (fast, flexible, convenient for custom rules); Soda became the foundation for platform-level data quality: standardization, scalability, unified practices for teams, and transparency of checks and results. Together, they provided more comprehensive coverage: both at the level of specific calculations/datamarts and at the level of a centralized DQ process—as a result, our data and key calculations are now covered by checks as comprehensively and practically as possible.

The next step in data quality evolution was automating the end-to-end process. We began developing our own DQ-generator—a service that uses a Large Language Model (powered by Kimi) to automatically analyze metadata and sample data. The generator suggests ready-made check templates (for Soda and PySpark), identifying typical error patterns and specific field interrelationships characteristic of financial reporting. Humans only need to validate the set and quality of checks, which has significantly accelerated the data coverage process with DQ checks.

This talk will be useful for testers, analysts, data engineers, and tech leads. Especially for those specialists who are just planning to implement DQ and want to understand the strengths, limitations, and "pitfalls" of popular off-the-shelf solutions on the market in advance, as well as practical ways to automate the data validation cycle.