Using within your ETL System#

How to integrate Hamilton within your existing ETL system.

Compatibility Matrix#

Title#

Framework / Scheduler

Compatibility

Airflow

✅

Dagster

✅

Prefect

✅

Kubeflow Pipelines

✅

CRON

✅

dbt

❔ (dbt did not run python, but now it does so it should)

kubernetes

✅ but you need to setup kubernetes to run an image that can run python code - e.g. see https://medium.com/avmconsulting-blog/running-a-python-application-on-kubernetes-aws-56609e7cd88c

docker

✅ but you need to setup a docker image that can execute python code.

… in general if it runs python 3.6+ …

✅

ETL Recipe#

  1. Write Hamilton functions & “driver” code.

  2. Publish your Hamilton functions in a package, or import via other means (e.g. checkout a repository & include in python path).

  3. Include sf-hamilton as a python dependency

  4. Have your ETL system execute your “driver” code.

  5. Profit.