Jamesserra.com Website Review

Azure Data Factory v2 (ADF) has a new feature in public preview called Data Flow.  I have usually described ADF as an orchestration tool instead of an Extract-Transform-Load (ETL) tool since it has the “E” and “L” in ETL but not the “T”.  But now it has the data transformation capability, making ADF the equivalent of “SSIS in the cloud” since it has the ability to mimic SSIS Data Flow Data Flow works without you having to write any lines of code as you build the solution by using drag-and-drop features of the ADF interface to perform data transformations like data cleaning, aggregation, and preparation.  Behind the scenes, the ADF JSON code currently 13 different dataset manipulation operations with more to come: The Data Flow feature in ADF is currently in limited public preview.  If you would like to try this out on your Data Factories, please fill has been enabled, you will see “Data Factory V2 (with data flows)” as an option from the Azure Portal when creating Data Factories. With data lakes becoming very popular, a common question I have been hearing often from customers is, “Should I load I would not say it’s common place to load structured data into the data lake, but I do see it frequently. In most cases it is not necessary to first copy relational source data into the data lake and then into the data warehouse, especially when keeping in mind the effort to migrate existing ETL jobs that are already copying source data into the data warehouse, but there are volumes and increasing rate of ingesting (velocity), so offload some of them to the Hadoop data lake allows end-users to query data in a data lake using regular SQL, so they are not required to learn any Hadoop-related technologies.  PolyBase even allows the end-user to use SQL, or any reporting tool that uses SQL, to join data in a relational database with data in a PolyBase from the data lake (which is usually much faster than using SSIS to copy from the source to Azure SQL Data Warehouse)