Configure Power BI Lineage Collection with Log Analytics

This article explains how to configure Power BI Data Source in Metric Insights using Log Analytics for Lineage collection. These instructions are intended for the Metric Insights administrator and assume that the required configuration in Azure has been completed by your organization's Azure administrator. You should have received the configured Azure application credentials, which will be used to configure the Data Source in Metric Insights.

Before proceeding, ensure that all prerequisites are met and that the initial configuration for your selected authentication type is complete. In this article, you will enable the required System Variables for metadata collection in Metric Insights, create and configure the Power BI Data Source, set up Object List Refresh, and sync metadata.

PREREQUISITES:

  1. Set up Azure for Power BI Lineage Collection.

Table of contents:

  1. Enable Metadata Collection with Log Analytics
  2. Enable Metadata Collection in Metric Insights
  3. Configure Power BI Cloud Data Source in Metric Insights
    1. Create a Microsoft Power BI Cloud Data Source
    2. Configure Auth Parameters
    3. Configure Other Data Source Parameters
    4. Configure Object List Refresh
      1. Manually Refresh Object List
      2. Add Object List Refresh Trigger
    5. Sync Metadata

1. Enable Metadata Collection in Metric Insights

Access Admin > System > System Variables

  1. Enter ENABLE_METADATA_COLLECTION in the search field
  2. Click the gear icon
  3. Assigned Value: Y
  4. [Save], then [Commit Changes]

NOTE: See Setting System Variables for information on System Variables configuration.

2. Enable Metadata Collection with Log Analytics

  1. Enter POWERBI_FETCH_METADATA_WITH_LOG_ANALYTICS in the search field.
  2. Click the gear icon.
  3. Assigned Value: Y.
  4. [Save], then [Commit Changes]

3. Configure Power BI Cloud Data Source in Metric Insights

3.1. Create a Microsoft Power BI Cloud Data Source

See the instructions on creating a new Microsoft Power BI Cloud Data Source for your Metric Insights version:

3.2. Configure Auth Parameters

Proceed with the Data Source parameters configuration based on the selected Auth Type:

Expand or collapse content Service Principal
Expand or collapse content v7.2.0+
  1. Auth Type: Service Principal.
  2. Application ID: Enter the Application (client) ID of the Azure application.
  3. Client Secret: Enter the Client Secret Value of the Azure application.
  4. Directory (tenant) Id: Enter Directory (tenant) Id of the the Azure application.
Expand or collapse content Prior to v7.2.0
  1. Auth Type: Service Principal.
  2. Application ID: Enter the Application (client) ID of the Azure application.
  3. Client Secret: Enter the Client Secret Value of the Azure application.
  4. Server: https://app.powerbi.com.
  5. Directory (tenant) Id: Enter Directory (tenant) Id of the the Azure application.
Expand or collapse content Password
Expand or collapse content v7.2.0+
  1. Auth Type: Password.
  2. Username, Plugin Password: Enter the credentials of a Fabric Administrator user.
  3. Application ID: Enter the Application (client) ID of the Azure application.
Expand or collapse content Prior to v7.2.0
  1. Auth Type: Password.
  2. Application ID: Enter the Application (client) ID of the Azure application.
  3. Server: https://app.powerbi.com.
  4. Username, Plugin Password: Enter the credentials of a Fabric Administrator user.
Expand or collapse content OAuth
Expand or collapse content v7.2.0+

NOTE: Lineage collection with Log Analytics does not work with Client Secret authentication. Only Application ID, Directory (tenant) Id and the credentials of a Fabric Administrator user are required.

  1. Auth Type: OAuth.
  2. Application ID: Enter Application (client) ID of the Azure application.
  3. Directory (tenant) Id: Enter Directory (tenant) Id of the of the Azure application.
  4. [Get Token], enter credentials of a Fabric Administrator in the pop-up window.
Expand or collapse content Prior to v7.2.0
  1. Auth Type: OAuth.
  2. Application ID: Enter Application (client) ID of the Azure application.
  3. Server: https://app.powerbi.com.
  4. Directory (tenant) Id: Enter Directory (tenant) Id of the of the Azure application.
  5. [Get Token], enter credentials of a Fabric Administrator in the pop-up window.

3.3. Configure Other Data Source Parameters

See parameter configuration details specific to your MI version:

Expand or collapse content v7.2.0
  1. Power BI Object Types must include Dashboards and Reports.
  2. Select Log Analytics Dataset.
  3. Log Analytics Workspace ID: Enter the ID of the previously configured Log Analytics Workspace.
Expand or collapse content v7.1.1-v7.1.2a

 


Parameter NameValue
1Scope: Types to collectMust include  DASHBOARD, REPORT, LOG.
2Log Analytics: Workspace ID
The ID of the previously configured Log Analytics Workspace
Expand or collapse content Prior to v7.1.1

Parameter NameValue
1Log Analytics Workspace IDThe ID of the previously configured Log Analytics Workspace
2Object types to collectMust include  DASHBOARD, REPORT, LOG.

3.4. Configure Object List Refresh

3.4.1. Manually Refresh Object List

Access Source Objects tab

  1. [Refresh list]
  2. Once the list is refreshed, the Log Analytics object is displayed.

3.4.2. Add Object List Refresh Trigger

Adding an Object List Refresh Trigger allows to automatically refresh the Object List at the configured time interval.

  1. Generate Object List: automatically.
  2. Object List Refresh Trigger: Select a daily Trigger.

3.5. Sync Metadata

NOTE: Metadata sync is not related to the previously configured Object List Refresh Trigger and needs to be updated manually on the regular basis.

See instructions on syncing Metadata for your MI version:

Expand or collapse content v7.2.0+

Beginning in v7.2.0, Metadata can be synced from either of the following locations:

  1. From the Metadata tab: click [Sync Metadata]
  2. From any tab in the Plugin Data Source Editor: click [Refresh Metadata] in the top panel.
Expand or collapse content Prior to v7.2.0

Access Metadata tab

  1. [Sync Metadata]