Case Studies

Overview

Phi 21 worked with a boutique startup that provides best in class solutions in the big data and data integrations space. One of their flagship products is an Analytics automation platform with extensive capabilities to ingest and process large volumes of data from different sources and build complex data pipelines. The initial versions of this product provided a command line interface for defining data pipelines and their building blocks. One of the important features in the roadmap was to provide a web based user interface supporting all the capabilities of command line interface. Phi 21 was engaged to build a web based user interface which can be used to define complex data pipelines in an intuitive manner.

Solution

An IDE built using Angular 10 which guides the user with a step by approach to create a data lake.

Key Features

• Easy to create and test Endpoints.
• Link Sources and Sinks to Endpoints.
• Create Pipelines connecting source to multiple sinks.
• Add transformation logic via connecting steps in the overall flow from source to sink, add any number of tables/entities on the way as possible to enable the user to create the most complex transformations.
• Generate and deploy the transformations on Apache Airflow.
• Publish the metadata to Apache Atlas.

Outcome

An IDE built using Angular 10 which guides the user in automating various analytical processes involved in building a data lake.

Key Features

• Create Endpoints (a.k.a connectors) to various source systems.
• Data Sources linked to the Endpoints to read data.
• Sinks to write processed/transforme/cleansed data.
• Pipelines to transform the data from Source to Sink.