System Design

1. Overview

This document explains the system design and how different parts of the components talk to each other. This repository contians only the web app. The other components/services are stored separately since they are managed by the Data Management team and Waleed.

2. High Level Architecture

The source of the data is Jadawel, which is used by operators (e.g. PDO, OQ, etc.) to insert data that is at the end of the data flow reaches the dashboard. So in detail, this is the flow of the data:

1. Operator inserts data in Jadawel.
2. The data in Jadawel are fetched/sync'd with other data sources (which do not concern us) by MageAI, which is a tool for data aggregation. The data management team uses the tool in the data warehouse solution for MEM. It is the most important tool in the data warehouse since it fetches data from multiple sources and dump them in a single cleaned database.
3. The clean data is stored in Postgres. For Jadawel data, there is a pipeline (which is a python script) that fetches data from Jadawel and inserts them in a tabel in Postgres. The pipeline runs every few minutes.
4. Then NocoDB exposes the data stored in postgres via an API. NocoDB is just an interface for postgres since it create an API endpoint for each table in the database, so it's a replacement of a custom API.
5. Finally, the data is fetched in the website and aggregated based on the information to show.

3. API Endpoints

For detailed information about the NocoDB API endpoints used by the frontend, including authentication, request/response formats, and usage examples, see the API Endpoints Documentation.