Profiling

As the first line of defense for your data integration solution, profiling data helps you examine whether your existing data sources meet the quality standards of your solution. Properly profiling your data saves execution time because you identify issues that require immediate attention from the start – and avoid the unnecessary processing of unacceptable data sources. Data profiling becomes even more critical when working with raw data sources that do not have referential integrity or quality controls.

There are several data profiling tasks: column statistics, value distribution and pattern distribution. These tasks analyze individual and multiple columns to determine relationships between columns and tables. The purpose of these data profiling tasks is to develop a clearer picture of the content of your data.

• Column Statistics – This task identifies problems in your data, such as invalid dates. It reports average, minimum, maximum statistics for numeric columns.

• Value Distribution – Identifies all values in each selected column and reports normal and outlier values in a column.

• Pattern Distribution – Identifies invalid strings or irregular expressions in your data.

Next Step: Cleansing