{"id":562,"date":"2025-01-12T15:04:49","date_gmt":"2025-01-12T12:04:49","guid":{"rendered":"https:\/\/ilhamidemirci.com\/?p=562"},"modified":"2025-01-12T15:09:28","modified_gmt":"2025-01-12T12:09:28","slug":"data-wrangler-in-microsoft-fabric","status":"publish","type":"post","link":"https:\/\/ilhamidemirci.com\/index.php\/2025\/01\/12\/data-wrangler-in-microsoft-fabric\/","title":{"rendered":"Data Wrangler in Microsoft Fabric"},"content":{"rendered":"\n

Data analysts, engineers, or scientists\u2026 No matter what title we work under, the most exhausting process for all of us is making data ready for processing. Even if we just want to visualize the data using Power BI, we first need to dive into Power Query to make the data modelable. Similarly, if we plan to build models with machine learning, we must first explore and prepare the data. That\u2019s why data preprocessing, organizing, cleaning, and transformation become the most critical phases of data analytics processes.<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

The \u201cData Wrangler\u201d<\/strong> tool, part of the Data Science layer in Microsoft Fabric, assists your data preparation processes by detecting unexpected anomalies in your datasets (e.g., outliers, missing or incorrect information, duplicate rows, logical errors) without requiring you to write any code.<\/p>\n\n\n\n

Like Power Query, the Data Wrangler simplifies data transformation processes. However, it takes this a step further by generating and displaying pandas or PySpark code for every operation. This not only transforms your data but also provides learning opportunities for those without coding experience.<\/p>\n\n\n\n

Let\u2019s get started!<\/p>\n\n\n\n

1. Creating Pandas or Spark DataFrames<\/strong><\/h4>\n\n\n\n

You can make the Data Wrangler tool operational by converting any file format in your Lakehouse (that can be transformed into a pandas or Spark DataFrame) into a dataframe.<\/p>\n\n\n\n

# read csv file
df = spark.read.format(\"csv\").option(\"header\", \"true\").load(\"Files\/example_dataset.csv\")

# df now is a Spark DataFrame containing CSV data from \"Files\/Raw Data.csv\".
display(df)

print(df.head())<\/pre>\n\n\n\n
# create Pandas DataFrame from Spark df
pandas_df = df.toPandas()

print(pandas_df.head())<\/pre>\n\n\n\n
The Data Wrangler supports both pandas and Spark DataFrames.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
2. Data Wrangler Interface<\/strong><\/h4>\n\n\n\n
The tool features a user-friendly interface and consists of five panels. The \u201cOperations\u201d panel (1), where you can apply predefined operations; the main data view (2), where you can examine the data in row and column format; the panel (3) displaying applied operations and allowing you to delete them; the panel (4) showcasing the Python code generated for each operation; and the summary panel (5), containing statistics for the entire dataset or selected columns.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
3. Example Operations<\/strong><\/h4>\n\n\n\n
The no-code feature allows you to generate statistics for your data and easily make your dataset processable (modelable) thanks to its intuitive interface. Here are a few examples:<\/p>\n\n\n\n
    \n
  1. You can sort, filter, rename columns, or most importantly, change the data type of a column.<\/li>\n<\/ol>\n\n\n\n
    \"\"\/<\/figure>\n\n\n\n
    1.1. For the operations you perform, the existing state (red)<\/em> and the proposed changes (green)<\/em> are displayed in separate columns. This allows you to review the changes before applying them.<\/p>\n\n\n\n
    \"\"\/<\/figure>\n\n\n\n
    2. You can create a new column based on a reference column.<\/p>\n\n\n\n
    \"\"\/<\/figure>\n\n\n\n
    3. You can perform normalization <\/em>by scaling the values in the dataset to a specific range (e.g., 0 to 1 or 1 to 5).<\/p>\n\n\n\n
    \"\"\/<\/figure>\n\n\n\n
    4. You can perform \u201cone-hot encoding\u201d by creating a separate column for each unique value of a categorical variable (e.g., \u201cGender\u201d)<\/em>.<\/p>\n\n\n\n
    \"\"\/<\/figure>\n\n\n\n
    Note:<\/strong> As shown in the image above, it is important to note that you cannot manually edit the code generated by the Data Wrangler tool. However, you can transfer the code (in pandas or PySpark format)<\/em> to MS Fabric notebooks to carry out your work.<\/p>\n\n\n\n
    \"\"\/<\/figure>\n\n\n\n
    4. All Operations<\/h4>\n\n\n\n
    The Data Wrangler includes a total of 27 operations organized under 8 main categories.<\/p>\n\n\n\n
    \"\"\/<\/figure>\n\n\n\n
    – Find and Replace<\/strong><\/p>\n\n\n\n