Skip/Notify Processing of Empty files via Azure Data Factory/Synapse

Problem Statement

When it comes to Data Engineering, one of the challenges we face is dealing with Empty or Blank files, or files that only have headers at the source.

So, is there an automated way to either Skip or Notify the processing of these files using Azure Data Factory or Synapse?

Solution

CASE 1: Empty / Blank File

As shown above, the file size for an empty file is 0 B.

Resolution:

We can use the Get Meta Data activity to retrieve the file size. If the file size is ‘0’, we can conclude that it is an empty or blank file.

Output:

CASE 2: File with just Headers

Note: The size of a file with only headers may vary depending on the number of columns in the header.

Resolution:

We can use the Lookup activity to count the number of records (excluding the header) in the file. If the count is ‘0’, we can conclude that it is a file with just headers and no actual data.

Output:

Note: The Lookup activity method can also be used for Case #1 to validate both empty/blank files and files with only headers. It is a better and more efficient solution.

To learn more about Skrots and how we provide similar solutions, visit Skrots. You can also explore our wide range of services at Skrots Services. Don’t forget to check out our other blogs at Blog at Skrots. Thanks for reading!


Know more about our company at Skrots. Know more about our services at Skrots Services, Also checkout all other blogs at Blog at Skrots

Thanks, Harsh
Founder | CEO — Skrots

Learn more about our blog at Blog at Skrots. Checkout our list of services on Skrots. Give a look at our website design at Skrots . Checkout our LinkedIn Page at LinkedIn.com. Check out our original post at https://blog.skrots.com/skip-notify-processing-of-empty-files-via-azure-data-factory-synapse/?feed_id=147&_unique_id=651cca7b4cb24