Superset

Important Capabilities

Capability	Status	Notes
Detect Deleted Entities	✅	Optionally enabled via stateful_ingestion
Domains	✅	Enabled by domain config to assign domain_key
Table-Level Lineage	✅	Supported by default

This plugin extracts the following:

• Charts, dashboards, and associated metadata

See documentation for superset's /security/login at https://superset.apache.org/docs/rest-api for more details on superset's login api.

CLI based Ingestion

Install the Plugin

The superset source works out of the box with acryl-datahub.

Starter Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: superset
  config:
    # Coordinates
    connect_uri: http://localhost:8088

    # Credentials
    username: user
    password: pass
    provider: ldap

sink:
  # sink configs

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

Field	Description
api_key string	Preset.io API key.
api_secret string	Preset.io API secret.
connect_uri string	Superset host URL. Default: http://localhost:8088
database_alias map(str,string)
display_uri string	optional URL to use in links (if connect_uri is only for ingestion)
manager_uri string	Preset.io API URL Default: https://api.app.preset.io
options object	Default: {}
password string	Superset password.
platform_instance string	The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://datahubproject.io/docs/platform-instances/ for more details.
provider string	Superset provider. Default: db
username string	Superset username.
env string	The environment that all assets produced by this connector belong to Default: PROD
domain map(str,AllowDenyPattern)	A class to store allow deny regexes
domain.key.allow array	List of regex patterns to include in ingestion Default: ['.*']
domain.key.allow.string string
domain.key.ignoreCase boolean	Whether to ignore case sensitivity during pattern matching. Default: True
domain.key.deny array	List of regex patterns to exclude from ingestion. Default: []
domain.key.deny.string string
stateful_ingestion StatefulStaleMetadataRemovalConfig	Superset Stateful Ingestion Config.
stateful_ingestion.enabled boolean	Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False Default: False
stateful_ingestion.remove_stale_metadata boolean	Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled. Default: True

Schema

The JSONSchema for this configuration is inlined below.

{
  "title": "SupersetConfig",
  "description": "Base configuration class for stateful ingestion for source configs to inherit from.",
  "type": "object",
  "properties": {
    "platform_instance": {
      "title": "Platform Instance",
      "description": "The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://datahubproject.io/docs/platform-instances/ for more details.",
      "type": "string"
    },
    "env": {
      "title": "Env",
      "description": "The environment that all assets produced by this connector belong to",
      "default": "PROD",
      "type": "string"
    },
    "stateful_ingestion": {
      "title": "Stateful Ingestion",
      "description": "Superset Stateful Ingestion Config.",
      "allOf": [
        {
          "$ref": "#/definitions/StatefulStaleMetadataRemovalConfig"
        }
      ]
    },
    "connect_uri": {
      "title": "Connect Uri",
      "description": "Superset host URL.",
      "default": "http://localhost:8088",
      "type": "string"
    },
    "display_uri": {
      "title": "Display Uri",
      "description": "optional URL to use in links (if `connect_uri` is only for ingestion)",
      "type": "string"
    },
    "domain": {
      "title": "Domain",
      "description": "regex patterns for tables to filter to assign domain_key. ",
      "default": {},
      "type": "object",
      "additionalProperties": {
        "$ref": "#/definitions/AllowDenyPattern"
      }
    },
    "username": {
      "title": "Username",
      "description": "Superset username.",
      "type": "string"
    },
    "password": {
      "title": "Password",
      "description": "Superset password.",
      "type": "string"
    },
    "api_key": {
      "title": "Api Key",
      "description": "Preset.io API key.",
      "type": "string"
    },
    "api_secret": {
      "title": "Api Secret",
      "description": "Preset.io API secret.",
      "type": "string"
    },
    "manager_uri": {
      "title": "Manager Uri",
      "description": "Preset.io API URL",
      "default": "https://api.app.preset.io",
      "type": "string"
    },
    "provider": {
      "title": "Provider",
      "description": "Superset provider.",
      "default": "db",
      "type": "string"
    },
    "options": {
      "title": "Options",
      "default": {},
      "type": "object"
    },
    "database_alias": {
      "title": "Database Alias",
      "description": "Can be used to change mapping for database names in superset to what you have in datahub",
      "default": {},
      "type": "object",
      "additionalProperties": {
        "type": "string"
      }
    }
  },
  "additionalProperties": false,
  "definitions": {
    "DynamicTypedStateProviderConfig": {
      "title": "DynamicTypedStateProviderConfig",
      "type": "object",
      "properties": {
        "type": {
          "title": "Type",
          "description": "The type of the state provider to use. For DataHub use `datahub`",
          "type": "string"
        },
        "config": {
          "title": "Config",
          "description": "The configuration required for initializing the state provider. Default: The datahub_api config if set at pipeline level. Otherwise, the default DatahubClientConfig. See the defaults (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/graph/client.py#L19).",
          "default": {},
          "type": "object"
        }
      },
      "required": [
        "type"
      ],
      "additionalProperties": false
    },
    "StatefulStaleMetadataRemovalConfig": {
      "title": "StatefulStaleMetadataRemovalConfig",
      "description": "Base specialized config for Stateful Ingestion with stale metadata removal capability.",
      "type": "object",
      "properties": {
        "enabled": {
          "title": "Enabled",
          "description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False",
          "default": false,
          "type": "boolean"
        },
        "remove_stale_metadata": {
          "title": "Remove Stale Metadata",
          "description": "Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.",
          "default": true,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    },
    "AllowDenyPattern": {
      "title": "AllowDenyPattern",
      "description": "A class to store allow deny regexes",
      "type": "object",
      "properties": {
        "allow": {
          "title": "Allow",
          "description": "List of regex patterns to include in ingestion",
          "default": [
            ".*"
          ],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "deny": {
          "title": "Deny",
          "description": "List of regex patterns to exclude from ingestion.",
          "default": [],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "ignoreCase": {
          "title": "Ignorecase",
          "description": "Whether to ignore case sensitivity during pattern matching.",
          "default": true,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    }
  }
}

If you were using database_alias in one of your other ingestions to rename your databases to something else based on business needs you can rename them in superset also

source:
  type: superset
  config:
    # Coordinates
    connect_uri: http://localhost:8088

    # Credentials
    username: user
    password: pass
    provider: ldap
    database_alias:
      example_name_1: business_name_1
      example_name_2: business_name_2

sink:
  # sink configs

Code Coordinates

• Class Name: datahub.ingestion.source.superset.SupersetSource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Superset, feel free to ping us on our Slack.