Configuring Databricks External Resource (v7.2.0)

This article describes the second step of Connecting to Databricks MCP Server. Proceed to creating and configuring a Concierge External Resource and finish establishing connection between Concierge and Databricks Genie.

Prerequisites:

  • Metric Insights Admin account;
  • Values collected from Databricks account:
    • Genie Space ID
    • Server URL
    • For "Defined in configuration Auth Type:
      • Token
    • For "Service Account Token" Auth Type:
      • Application ID
      • Client Secret Value

1. Create an External Resource

Go to Metric Insights, access Admin > Concierge Setup and open the Content Sources tab

  1. Click [+ Add External Resource Configuration].
  2. Select Databricks Genie from the list.

After that, the User will be redirected to the External Resource Editor.

2. Authentication Tab

The Name is generated automatically, but it can be changed.  This name will be used to identify this external resource when assigning domains and if using Service Account Token, when user are asked to Authenticate.

2.1. Auth Type: Defined in Configuration

Choosing "Defined in configuration" type of authentication means that the Token, copied in the Databricks account will be inserted into the configuration code as is.

This type of authentication is the most flexible and should be used when the long-lasting Token can be generated within the external tool. Be aware, that when the Token expires, it has to be re-entered into the code on Configuration tab manually.

2.2. Auth Type: Service Account Token

Choosing "Service Account Token" type of authentication means that the Token will be requested and then automatically inserted to the configuration code by Metric Insights. The User has to provide access credentials to the Databricks account.

This type of authentication is used when the external tool supports OAuth authorization. If the Token requires regular refreshing, it will be re-inserted into the configuration code automatically.

2.2.1. Get Token

Click [Get Token], so Metric Insights will obtain the Token from Databricks. If Token was generated successfully, the confirmation message "Token generated for <...>" will appear.

3. Configuration Tab

Proceed to Configuration tab

  1. Icon: To add a custom icon, upload an SVG image on a transparent background.
  2. A block of example JSON code is added to the field automatically. The parts in double curly braces ("{{}}") has to be filled with information specific to your environment.
  3. [Test Resource Connection] to check if it is possible to establish connection to the external tool.

Check the JSON code usage example for Databricks External Resource configuration:

Code Usage Example 

{
  "name": "databricks",
  "enabled": true,
  "title": "Databricks",
  "display": "raw_or_llm_for_charting",
  "data_transformer": "ai.scenarios.mcp_data_transformers.databricks.DatabricksTransformer.transform",
  "ignore_native_mcp_tool_details": true,
  "apply_data_transformer_before_llm_response_processing": false,
  "optional_tool_params_to_remove": [
    "conversation_id"
  ],
  "scenario_type": "mcp_single_tool",
  "scope": "Databricks talk to your data solution, questions about support tickets.",
  "mcp_server": {
    "url": "https://{{instance}}.cloud.databricks.com/api/2.0/mcp/genie/{{space_id}}",
    "name": "databricks",
    "auth": "{{databricks token}} or <auth_token>",
    "transport": "streamable_http"
  },
  "tool": "query_space_{{space_id}}",
  "timeout": 300
}

Databricks Code Specifics

FieldDescription
nameThe value of this variable should be unique and exactly match the Name field on the Authentication tab. It may contain A-Z/a-z letters, numeric digits, and underscore (_).
enabledThis field allows for the External Resource to be disable instead of having to delete it. If it is true, the External Resource is enabled. If it is false, the External Resource is disabled.

NOTE: After disabling the External Resource make sure to disconnect or disable all Domains connected to it or the user will see an error message in Concierge.
titleThe display name for this external resource. This name is shown when associating Domains and sometimes in error messages if there is a problem with the External Resource.
displayDefines how data from the external agent is processed and shown. The raw_or_llm_for_charting is the recommended option for this variable. It means that data is received from the MCP tool as is and is only sent to be processed by the LLM on the Concierge side if there is a chart to be created.
data_transformer

Custom response processor for this MCP server. Metric Insights provide the following transformer for processing raw responses, which will be quicker than use the LLM for processing:

ai.scenarios.mcp_data_transformers.databricks.DatabricksTransformer.transform

ignore_native_mcp_tool_detailsDetermines whether to pass the description provided by the MCP server into the LLM request context. Can be true or false.
apply_data_transformer_before_llm_response_processingDetermines whether to use the custom data transformer defined in "data_transformer" before LLM processing of  the MCP response. This should be set to true.
optional_tool_params_to_removeThis field defines optional parameters for the MCP tool to be removed from the request. For Databricks to work properly, the conversation_id has to be removed.
scenario_typeThe mcp_single_tool option is mandatory for this variable.
scopeThis optional field may be included to provide a general description for this external resource.  This is useful for providing instructions that are shared across all Domains that use this external resource.
mcp_serverurl

URL of the MCP Server is a required value. The URL to use is constructed like this:

https://{{instance}}.cloud.databricks.com/api/2.0/mcp/genie/{{space_id}} 

where {{instance}} is the server URL and {{space_id}} is the Space ID value that were obtained in Databricks account. Space ID(s) can be included directly in the JSON block or via a variable (See the Add Variables section below).

nameA unique name for this MCP server that identifies this specific connection.  This can be the same as the "name" field above.
auth
  • If the Auth Type on the Authentication tab is "Defined in configuration", this field should contain the {{databricks token}}, generated and copied in the Databricks account. It would be best to include this as an encrypted variable. (See the Add Variables section below).
  • If the Auth Type on the Authentication tab is "Service Account Token", this field must be set to <auth_token> 
transportThis variable is required. Databricks MCP server supports only the streamable_http transport mode.
toolThe scenario_type variable is set to mcp_single_tool, so this field is required. The value to use is query_space_{{space_id}} where {{space_id}} is the Space ID that was obtained in Databricks account. This can also be included as a variable. (See the Add Variables section below).
timeoutHow long Concierge will wait for a response from the external resource.  The recommended value is 300 seconds.

4. Add Variables (Optional)

It is possible to use Variables to replace sensitive information in the JSON code block.

  1. Click the arrow icon to open the Variables side menu.
  2. Click [+ Create Variable] and add the Name and Value of the Databricks token into corresponding fields. No need to to save anything. After that you can use the Variable Name in the square brackets (for example, [DATABRICKS_TOKEN]) in the code instead of the actual Token itself. Use Variables to keep the sensitive information safe. It is optional to encrypt the variable by selecting the checkbox in Encrypted column.

Don't forget to [Save] the External Resource.

Databricks External Resource will not appear in the All Sources menu. Add a Domain to it and that Domain will be visible in All Sources menu.