Smart Data Cleaning

Why conventional data quality methods are insufficient

Today’s enterprise data platforms handle petabytes of information, integrate diverse unstructured sources, and undergo continuous change. In such dynamic settings, conventional rule-based data quality systems struggle to keep up. Their reliance on manually defined constraints cannot scale or adapt to complex, high-volume, and rapidly evolving data.

This challenge has led to the rise of AI-augmented data quality engineering. It transforms data quality from rigid, deterministic checks into flexible, probabilistic, generative, and continuously learning systems.

AI-powered Data Quality (DQ) frameworks utilize:

• Deep learning for inferring meaning
• Transformers for aligning ontologies
• GANs and VAEs for detecting anomalies
• Large Language Models (LLMs) for automated corrections
• Reinforcement learning to constantly evaluate and update reliability scores

This approach results in a self-healing data ecosystem that can adjust to evolving data characteristics and expand with increasing enterprise complexity.

Automated Semantic Inference: Rule-free data comprehension

Traditional methods for schema inference rely on basic pattern matching. However, contemporary datasets often feature ambiguous headers, mixed-value formats, and incomplete metadata. Deep learning models address this by learning underlying semantic representations.

Sherlock: Multi-input deep learning for classifying columns

Sherlock, developed at MIT, performs highly accurate classification of columns into semantic types by analyzing over 1,588 statistical, lexical, and embedding features.

Instead of relying on rigid rules like “five digits equals ZIP code,” Sherlock evaluates distribution patterns, character entropy, word embeddings, and contextual usage to categorize fields such as:

• ZIP code versus employee ID
• Price versus age
• Country versus city

This significantly boosts accuracy, especially when column names are absent or misleading.

Sato: Context-aware semantic typing with table-level intelligence

Sato builds upon Sherlock by integrating context from the entire table. It employs topic modeling, context vectors, and structured prediction (CRF) to discern relationships between columns.

This capability enables Sato to distinguish between:

• A person’s name in Human Resources data
• A city name in demographic information
• A product name in retail transaction data

Sato enhances macro-average F1 scores by approximately 14 percent compared to Sherlock in noisy environments, proving effective in data lakes and uncurated ingestion pipelines.

Ontology Alignment Using Transformers

Large enterprises frequently manage dozens of schemas across disparate systems. Manually mapping these is both time-consuming and prone to inconsistencies. Transformer-based models address this by deeply understanding semantic relationships within schema descriptions.

BERTMap: Transformer-powered Schema and Ontology Alignment

BERTMap (AAAI) fine-tunes BERT on ontology text structures, generating consistent mappings even when labels are entirely different.

Illustrative examples include:

• “Cust_ID” linked to “ClientIdentifier”
• “DOB” linked to “BirthDate”
• “Acct_Num” linked to “AccountNumber”

It also integrates logic-based consistency checks, eliminating mappings that violate predefined ontology rules.

AI-driven ontology alignment enhances interoperability and diminishes the necessity for manual data engineering efforts.

Generative AI for Data Cleansing, Repair, and Imputation

Generative AI facilitates automated data remediation, moving beyond mere detection. Instead of engineers crafting correction rules, AI learns the expected behavior of the data.

Jellyfish: An LLM optimized for data preprocessing

Jellyfish is an instruction-tuned Large Language Model specifically designed for data cleaning and transformation tasks, such as:

• Identifying errors
• Imputing missing values
• Normalizing data
• Restructuring schemas

Its knowledge injection mechanism minimizes hallucinations by integrating domain-specific constraints during the inference process.

Enterprise teams leverage Jellyfish to boost consistency in data processing and cut down on manual cleanup time.

ReClean: Reinforcement learning for optimizing cleaning sequences

Data cleaning pipelines often execute steps in an suboptimal order. ReClean redefines this as a sequential decision problem where a Reinforcement Learning agent determines the most effective next cleaning action. The agent’s rewards are based on the performance of downstream machine learning models, rather than arbitrary quality rules. This approach, evaluated using LIME and SHAP tutorials, ensures cleaning efforts directly support business objectives.

This approach ensures that data cleaning directly supports business outcomes.

4. Deep Generative Models for Anomaly Detection

Statistical methods for anomaly detection falter with high-dimensional and non-linear data. Deep generative models learn the underlying data distribution’s true structure, enabling them to measure deviations with superior accuracy.

GAN-based anomaly detection: AnoGAN and DriftGAN

Generative Adversarial Networks (GANs) learn to recognize what constitutes “normal” data. During the inference phase:

• A high reconstruction error signals an anomaly.
• Low discriminator confidence also points to an anomaly.

AnoGAN pioneered this technique, while DriftGAN identifies changes indicative of concept drift, allowing systems to dynamically adapt.

GANs find common applications in areas such as fraud detection, financial analysis, cybersecurity, IoT monitoring, and various industrial analytics.

Variational Autoencoders (VAEs) for Probabilistic Imputation

VAEs map data into latent probability distributions, facilitating:

• Sophisticated imputation of missing values
• Quantification of associated uncertainties
• Effective management of Missing Not At Random (MNAR) scenarios

Advanced variants such as MIWAE and JAMIE offer high-accuracy imputation, even for multimodal data.

This results in significantly more dependable downstream machine learning models.

5. Constructing a Dynamic AI-Driven Data Trust Score

A Data Trust Score quantitatively assesses dataset reliability, integrating a weighted combination of factors including:

• Validity
• Completeness
• Consistency
• Freshness
• Lineage

Formula example

Trust(t) = ( Σ wi·Di  +  wL·Lineage(L)  +  wF·Freshness(t) ) / Σ wi

Where:

• Di represents inherent quality dimensions
• Lineage(L) reflects upstream quality
• Freshness(t) models data aging using exponential decay

Freshness Decay and Lineage Propagation

Data naturally diminishes in value over time due to staleness. Lineage ensures a dataset cannot be deemed more reliable than its source inputs.

These principles are fundamental to the Data Trust Score framework and align closely with Data Mesh governance tenets. Trust scoring generates quantifiable, auditable indicators of data health.

Contextual Bandits for Dynamic Trust Weighting

Various applications prioritize distinct quality attributes.

For instance:

• Dashboards emphasize data freshness.
• Compliance teams prioritize data completeness.
• AI models prioritize consistency and reduced anomalies.

Contextual bandits dynamically optimize trust scoring weights based on usage patterns, feedback, and observed downstream performance.

Explainability: Ensuring Auditable AI-Driven Data Quality

Enterprises require clear insights into why AI flags or corrects a specific record. Explainability is vital for maintaining transparency and ensuring regulatory compliance.

SHAP for Feature Attribution

SHAP quantifies the contribution of each feature to a model’s prediction, thereby enabling:

• In-depth root-cause analysis
• Detection of inherent biases
• Detailed interpretation of anomalies

LIME for Local Interpretability

LIME constructs simplified local models around individual predictions to illustrate how minor adjustments influence outcomes. It helps answer questions such as:

• “Would correcting an age value alter the anomaly score?”
• “Would adjusting the ZIP code impact the classification result?”

Explainability renders AI-based data remediation acceptable in heavily regulated industries.

Enhanced System Reliability, Reduced Human Intervention

AI-augmented data quality engineering transforms conventional manual checks into intelligent, automated workflows. By integrating semantic inference, ontology alignment, generative models, sophisticated anomaly detection frameworks, and dynamic trust scoring, organizations can establish systems that are more dependable, less reliant on human oversight, and better aligned with both operational and analytical requirements. This evolution is crucial for the upcoming generation of data-driven enterprises.

This article is presented as part of the Foundry Expert Contributor Network.
Interested in joining?