Project Overview​

TeleNova Systems, a leading telecommunications provider, launched its Customer 360  initiative to build a unified view of every customer. However, years of uncontrolled data ingestion had left the organization with fragmented, inconsistent, and incomplete records scattered across systems.

As a Data Engineer, I led the effort to rebuild trust in the data by designing a scalable and automated cleansing pipeline in Dataflow Gen2. This initiative produced the first refined Customer Silver dataset, laying the foundation for accurate analytics, targeted marketing, and consistent customer intelligence across the organization.

​​

Objective

Initial profiling of the Customer_Bronze table revealed multiple data quality challenges:

• Address inconsistencies (e.g., “Street,” “St,” “Str.”, “Rd.”).

• Mixed state values (abbreviations, full names, and varied capitalization).

• Contaminated phone fields with non-numeric symbols or invalid entries.

This data inconsistency caused major issues across departments:

• The Marketing team faced costly campaign failures due to undeliverable addresses.

• The Data Science team struggled with unreliable features for churn prediction models.

The goal was clear — to deliver a trusted Customer_Silver table that could serve as a foundation for business-critical analytics and downstream models.

​

Solution Overview​

To ensure maintainability, scalability, and reusability, the solution was built using Microsoft Fabric Dataflow Gen2, leveraging its low-code ETL capabilities and native integration within the Fabric ecosystem. This architecture provided:

​

• Low-Code Standardization: Utilized Dataflow Gen2’s declarative approach to build complex lookups and transformations without writing custom code, enabling faster development and easier maintenance .

• Lookup Table-Driven Cleansing: Employing Merge/Join operations within Dataflow Gen2 with reference tables (state_lookup, address_suffix_lookup) to systematically standardize every inconsistent value.

• Feature Engineering for Utility: Separating the address into Address_Street and Address_Unit components, and cleaning the Phone number to an all-numeric Phone_Clean format.

• Establishing Trust via Governance: Implementing a Data_Quality_Flag (1 for clean, 0 for known issues) to provide immediate transparency and quantifiable proof of data quality improvement to stakeholders.

​