Treasure data digdag visualization

By | March 24, 2022
Treasure data digdag

Treasure data is a leading CDP platform and leverages dig-dag for data pipelines within itself. digdag is similar to airflow where we define tasks inside a dag and is available as an open source. The CDP is a powerful platform but unfortunately no easy way to visualize the pipelines which makes it hard for onboarding when there are some complex pipelines already built. So I had the below sequence of steps to solve this problem for myself.

Solution Approach

  • Create Visualization
  • Make it colorful
  • Update the graph everytime there is a code change.
  • Make it part of CICD

Navigate to html

Installation

pip install -r requirements.txt

Usage example

  • Update Line 131 with the project where the graphs need to be updated.
  • Execute the python program to generate the graphs.

TODO

  • Parametrize the dir_name (Line 131)
  • Create github workflow to generate graphs when there are changes.
    • Identify the changes at the folder level
    • Execute the python script tp build graph for each folder with a change
    • Commit the graphs back to the branch

References

  • https://github.com/y-abe/digdag-graph
See also  Publish blog posts from Github to dev.to

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.