Understanding and enabling Microsoft Fabric - [Video]

 Want to know about Microsoft Fabric? Here is a video that speaks about Microsoft Fabric, what it offers and how you can start with a trial.

Unleashing Power BI in Jupyter Notebook - Embed and Create reports

This video discusses two things:

• Embedding Power BI report in Jupytor Notebook
• Creating Power BI report directly in Jupytor Notebook.

Embedding Power BI is not something new, it was announced in May 2021. Creating Power BI report in Jupytor Notebook was announced in last month, April 2023. Both topics are discussed in detail with demonstrations. Following are discussed:

• Powerbiclient Python package
• Two Authentication method
• Install powerbiclient in Jupytor notebook
• Embed Power BI report in Jupytor Notebook
• Add custom Notebook filter for filtering reports
• Loading visual data into Pandas dataframe
• Create Power BI report in Jupytor Notebook
• Quick report feature and personalizing visuals
• Publishing to Power BI service for sharing the dataset and report

Real-time Streaming in Power BI - Push Dataset

This video discusses the way of showing stream data with Power BI visuals, specifically with Push Datasets. Here is the content: 

• Types of streaming datasets supported by Power BI
• The way of creating the Push dataset
• Sending values using a .Net application
• Creating reports and dashboards visuals using streaming dataset.

Azure Synapse Analytics - Part III - Serverless SQL Pool

Azure Synapse Serverless SQL Pool facilitates data exploration, transformations and data warehousing with multiple functionalities, allowing us to work with it using SQL. This video discusses how it works, what we can do and demos on use cases. This is the content:

• Introduction to Serverless SQL Pool
• How useful it is
• Is it same as Dedicated SQL Pool?
• Demo
• Data exploration
• Data transformation
• Making a logical data warehouse and accessing it using Power BI

Read more at:
Serverless SQL Pool Architecture Serverless SQL Pool - Microsoft Learn

Azure Synapse Analytics - Part II - Hypersapce

 *** This is the session I have done for Sri Lanka Data Community – Feb 2022 episode. This does not contain the entire event.

With modern data solutions, where we see data warehousing and big data, analytics are not limited to columnar based queries, highly selective queries are part of the trend now. Here is the way of optimizing such queries with Hyperspace: An indexing subsystem for Apache Spark. Read more at: https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-performance-hyperspace?pivots=programming-language-csharp/?WT.mc_id=DP-MVP-33296

https://github.com/microsoft/hyperspace/?WT.mc_id=DP-MVP-33296

This talks about - What is Hyperspace and how it helps us to add indexes - How to see indexes, what changes it makes to the data lake - How to check whether indexes are used for queries - How to compare indexed query and non-indexed query with given APIs If you have any questions, please add to the comment section.

Azure Synapse Analytics - Part I

 This video speaks about the latest in data warehousing and big data analytics: Azure Synapse Analytics. This is the first part of it and it explains;

- The way we have been creating on-premise data solutions

- The way we have been creating data solutions in Azure before Azure Synapse Analytics

- How we can create data solutions in Azure using Azure Synapse Analytics

- Quick demo on creating the workspace, SQL Pool, Spark cluster and accessing and processing data

Power BI Desktop Model – Connect from SSMS and Export more than 30,000 rows

This video shows the way of connecting to Power BI Desktop Model from SQL Server Management Studio and using it for various operations. Additionally this shows how to get data saved into csv files using SSMS when number of records exceeds 30,000.

Azure Databricks – Enabling Databricks CLI for Azure Portal and Windows

This video shows the way of installing and configuring Azure Databricks CLI for Azure Portal (Cloud Shell) and Windows.

Azure Data Factory - Accessing a Databricks Notebook with Input and Output Parameters

This video shows the way of accessing Azure Databricks Notebooks through Azure Data Factory. Additionally, it explains how to pass values to the Notebook as parameters and how to get the returned value from Notebook to Data Factory Pipeline.

Coronavirus (Covid-19) Dashboard solution with Power BI

If you are interested in #covid-19 statistics, check this #dashboard out. This has been developed using #powerbi, #azuresqldatabase and #azuredatafactory.

http://coronavirusexplorer.com/

Azure Databricks - Accessing Data Lake using Azure Active Directory Credentials

This video discusses the way of accessing Azure Data Lake Gen 2 through Azure Databricks, using Azure Active Directory Credentials. Following are discussed;

• Ways of Accessing ADLS Gen 2
• Enabling Azure Data Lake Credential Passthrough
• Demo: Accessing ADLS Gen 2 in Azure Databricks
• Creating an Azure Databricks workspace
• Configuring Standard Cluster for AAD Credentials
• Configuring High Concurrency Cluster for AAD Credentials

Azure Databricks - Accessing Data Lake - Using a Service Principal

This video discusses the ways of accessing Azure Data Lake Gen 2 in Azure Databricks, specifically using a Service Principal. Following are discussed in detail;

• Ways of accessing Azure Data Lake Gen 2
• Creating a Service Principal for accessing Azure Data Lake
• Access using Role-based access control
• Access using ACL-based access control

Sending Emails from Azure SQL Database - YouTube Video

This video focuses on sending emails for operations related to Azure SQL databases. Following are discussed and demonstrated;

• Possibilities of tracking changes in Azure SQL Database for sending emails
• Setting up an Azure SendGrid account
• Setting up an Azure Logi Apps for sending emails

Data Science with SQL Server Machine Learning Services - YouTube Video

This video discusses performing data science experiments in SQL Server using Python. Setting up the environment, training, storing models and consuming trained models stored in the database are discussed with a demonstration.

Automating Resume and Pause of Azure Analysis Services - YouTube Video

This video discusses how to automate starting and stopping Azure Analysis Services using Azure Automation Account. Following are demonstrated;

• Create Azure Automation Account
• Import libraries required
• Create a PowerShell Runbook
• Schedule the Runbook

Power BI App for seeing Coronavirus - COVID-19 updates in Sri Lanka

Here is an Azure site built for showing updates related to Coronavirus - COVID-19, specifically for Sri Lanka.

This has been designed using Azure Data Factory, Azure SQL Database, Power BI and Azure App Services.

Azure Databricks - What, Why & How - YouTube Video

Here is the first video on Azure Databricks. This talks about;

• Why we need to Azure Databricks for processing data
• What is Azure Databricks
• Demo: How to provision an Azure Databricks workspace, a cluster, a database and a table.

Creating Paginated Reports with Power BI

Here is a video on Power BI Paginated reports. This videos discusses/demonstrates;

• What is a paginated report?
• Power BI support on paginated reports
• Power BI configuration for paginated reports
• Creating a paginated report using Report Builder
• Publishing a paginated report sourced to local database, to Power BI portal.

Here is the video:

Azure Data Lake Analytics - Accessing External Data Sources with USQL

This video discusses the way of accessing external data sources such as Azure SQL Database with USQL as external data sets. It explains why we need it, what objects should be created for accessing external sources and how external data can be accessed in USQL codes.

Watch and see!

Azure Data Factory V2 - Lookup and ForEach Activities

Here is my third video on Azure Data Factory. This video discusses and demonstrates;

• Why we need iteration in ETLs
• Lookup Activity
• ForEach Activity
• Demo: Iterating over a data collection using ForEach Activity, with data loaded from Lookup Activity

Azure Data Factory V2 - Creating a Pipeline using Author-Tab

Here is another video on Azure Data Factory V2. This video explains how to create a Pipeline manually, using Author-Tab. This video discusses and demonstrates;

• How to create Linked Services manually
• How to create Datasets manually
• How to create a Pipeline and add Activities
• And how to schedule the Pipeline

Azure Data Factory V2 - Copying On-Premise SQL Server data to Azure Data Lake Store

Azure Data Factory has been enhanced significantly with V2 and its support on Cloud-ETL-and-ELT is excellent now. First version of it was not much user-friendly but now it is not that difficult to implement ETL solutions with V2.

This video discusses;

• Why we need Azure Data Factory
• What is Azure Data Factory
• How it works and how it should be used
• Integration Runtime - installing for connecting with sources
• Copying On-Premise SQL Server Data to Azure Data Lake using Data Factory

Here is the video;

Here are some sample screens;

Introduction to Azure Data Lake Analytics - and basics of U-SQL

I made another video on Azure Data Lake, specifically on Azure Data Lake Analytics. This is a 40-minutes video and it discusses following items along with demonstrations;

• What is Azure Data Lake Analytics
• Data Lake Architecture and how it works
• Comparison between Azure Data Lake Analytics, HDInsight and Hadoop for processing Big Data.
• What is U-SQL and basics of it.
• Demo on How to create an Azure Data Lake Analytics account
• Demo on How to execute a simple U-SQL using Azure Portal
• Demo on How to extract multiple files, transform using C# methods and referenced assembly, making multiple results with bit complex transformations using Visual Studio.

Here is the video.

Introduction to Azure Data Lake Store - Does it replace the Data Warehouse?

I have made a video on Azure Data Lake and Azure Data Lake Store and published in YouTube. This video explains What is Azure Data Lake, Its Characteristics, Components of it, How it works, Usage, Comparison between Data Lake Store and Blob Storage, Modern Data Warehousing and How can we use Azure Data Lake with modern data warehousing.

Have a look on it and let me know you thoughts or questions you have.

Microsoft Access Database Engine 2016 Redistributable Setup - You cannot install the 32-bit version, You cannot install the 64-bit version

In order to access Excel workbook using Integration Services, one component we need to install is Microsoft Access Database Engine 2016 Redistributable. If 32-bit version of Office is installed, then 32-bit version of Microsoft Access Database Engine 2016 Redistributable should be installed. If 64-bit version of Office is installed, then 64-bit version of it has to be installed.

If it is not installed, this is what we see when try to access an Excel workbook

The requested OLE DB provider Microsoft.ACE.OLEDB.12.0 is not registered. If the 32-bit drivers is no.....

Generally, once it is installed, we can connect to Excel and work with it without any issue. However, I started experiencing below error when I try to install 32-bit version of Microsoft Access Database Engine 2016 Redistributable.

You cannot install the 32-bit version of Microsoft Access Database Engine 2016 because you currently have 64-bit Office products installed. If you want to.....

I was confused with this message because I have installed 32-bit version of Office. However, just for checking, I tried to install 64-bit version of Microsoft Access Database Engine 2016 Redistributable and this is what I got when I tried........

You cannot install the 64-bit version of Microsoft Access Database Engine 2016 because you currently have 32-bit Office products installed. If you want to....

Not sure what is the reason but it stopped me continuing, leaving me only option as uninstalling Office and installing again. Since it is not the best way, just thought to try out older version of Microsoft Access Database Engine Redistributable, hence downloaded 32-bit version of Microsoft Access Database Engine 2010 Redistributable from https://www.microsoft.com/en-us/download/details.aspx?id=13255. Good news is, I was able to install it and the best news is, SSIS could connect to Excel without any issue. This may not be the best solution for above error but for an urgent situation, you can try this and see.