You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
43 lines
1.9 KiB
Markdown
43 lines
1.9 KiB
Markdown
# Data-Transport
|
|
|
|
A powerful abstraction layer for seamless data communication across diverse systems. **data-transport** allows you to interact with NoSQL, SQL, and Cloud storage using a consistent interface powered by **Pandas** and **SQLAlchemy**.
|
|
|
|
## Why Choose data-transport?
|
|
|
|
* **Unified Interface:** Connect to PostgreSQL, MySQL, MongoDB, S3, etc., using the same consistent code.
|
|
* **Security First:** Prevents the dissipation of database connectivity information to protect against security breaches.
|
|
* **Simplicity & Power:** Leverages Pandas DataFrames and SQLAlchemy for intuitive data manipulation.
|
|
* **Robust Pipelines:** Easily integrate pre-processing and post-processing as unified pipelines.
|
|
* **CLI Integration:** Includes a dedicated CLI for registry management and ETL task execution.
|
|
|
|
## Supported Features
|
|
|
|
| Component | Technologies Covered |
|
|
| :--- | :--- |
|
|
| **SQL** | PostgreSQL, MySQL, SQL Server, SQLite3+, DuckDB |
|
|
| **NoSQL** | MongoDB, CouchDB |
|
|
| **Warehouse** | Apache Iceberg, Apache Drill |
|
|
| **Cloud** | Nextcloud, S3 |
|
|
| **Other** | Files, RabbitMQ, HTTP |
|
|
|
|
## Installation
|
|
|
|
Install the core package and your desired components:
|
|
|
|
```bash
|
|
# Basic installation with default SQL support
|
|
pip install data-transport@git+https://github.com/lnyemba/data-transport
|
|
|
|
# Full suite (SQL, NoSQL, Cloud, Warehouse)
|
|
pip install "data-transport[nosql,cloud,warehouse,all]"@git+https://github.com/lnyemba/data-transport.git
|
|
```
|
|
|
|
## Advanced Capabilities
|
|
* **Automated Pipelines:** Seamlessly aggregate functions for data cleaning and transformation.
|
|
* **Portability:** Share notebooks and scripts without exposing raw credentials.
|
|
* **Scalable Integration:** Compatible with environments like Apache Zeppelin, JupyterHub, and SageMaker.
|
|
|
|
---
|
|
[Learn More at the Project Website](https://healthcareio.the-phi.com/data-transport)
|
|
License: [MIT](https://github.com/lnyemba/data-transport)
|