# Introduction

This project implements an abstraction of objects that can have access to a variety of data stores, implementing read/write with a simple and expressive interface. This abstraction works with **NoSQL**, **SQL** and **Cloud** data stores and leverages **pandas**.

# Why Use Data-Transport ?

Data transport is a simple framework that enables read/write to multiple databases or technologies that can hold data. In using **data-transport**, you are able to:

- Enjoy the simplicity of **data-transport** because it leverages SQLAlchemy & Pandas data-frames.
- Share notebooks and code without having to disclosing database credentials. 
- Seamlessly and consistently access to multiple database technologies at no cost
- No need to worry about accidental writes to a database leading to inconsistent data
- Implement consistent pre and post processing as a pipeline i.e aggregation of functions
- **data-transport** is open-source under MIT License https://github.com/lnyemba/data-transport

## Installation

Within the virtual environment perform the following, the options for installation are:

**sql**       - by default postgresql, mysql, sqlserver, sqlite3+, duckdb 

    pip install data-transport[cloud,nosql,other,all]git+https://github.com/lnyemba/data-transport.git

Options to install components in square brackets, these components are 

**warehouse** - Apache Iceberg, Apache Drill

**cloud**  - to support nextcloud, s3

**nosql** - support for mongodb, couchdb

**other**  - support for files, rabbitmq, http

    pip install data-transport[nosql,cloud,warehouse,all]@git+https://github.com/lnyemba/data-transport.git

## Additional features

    - In addition to read/write, there is support for functions for pre/post processing
    - CLI interface to add to registry, run ETL
    - scales and integrates into shared environments like apache zeppelin; jupyterhub; SageMaker; ...

## Learn More

We have available notebooks with sample code to read/write against mongodb, couchdb, Netezza, PostgreSQL, Google Bigquery, Databricks, Microsoft SQL Server, MySQL ... Visit [data-transport homepage](https://healthcareio.the-phi.com/data-transport)