|
|
|
|
@ -13,20 +13,28 @@ Data transport is a simple framework that:
|
|
|
|
|
|
|
|
|
|
## Installation
|
|
|
|
|
|
|
|
|
|
Within the virtual environment perform the following :
|
|
|
|
|
Within the virtual environment perform the following (the following will install everything):
|
|
|
|
|
|
|
|
|
|
pip install git+https://github.com/lnyemba/data-transport.git
|
|
|
|
|
pip install data-transport[all]@git+https://github.com/lnyemba/data-transport.git
|
|
|
|
|
|
|
|
|
|
Options to install components in square brackets
|
|
|
|
|
Options to install components in square brackets are **nosql**; **cloud**; **other** and **warehouse**
|
|
|
|
|
|
|
|
|
|
pip install data-transport[nosql,cloud,warehouse,all]@git+https://github.com/lnyemba/data-transport.git
|
|
|
|
|
pip install data-transport[nosql,cloud,other, warehouse,all]@git+https://github.com/lnyemba/data-transport.git
|
|
|
|
|
|
|
|
|
|
The components available:
|
|
|
|
|
|
|
|
|
|
0. sql by default netezza; mysql; postgresql; duckdb; sqlite3; sqlserver
|
|
|
|
|
1. nosql mongodb/ferretdb; couchdb
|
|
|
|
|
2. cloud s3; bigquery; databricks
|
|
|
|
|
3. other files; http; rabbitmq
|
|
|
|
|
4. warehouse apache drill; apache iceberg
|
|
|
|
|
|
|
|
|
|
## Additional features
|
|
|
|
|
|
|
|
|
|
- In addition to read/write, there is support for functions for pre/post processing
|
|
|
|
|
- Reads are separated from writes to avoid accidental writes.
|
|
|
|
|
- Streaming (for large volumes of data) by specifying chunksize
|
|
|
|
|
- CLI interface to add to registry, run ETL
|
|
|
|
|
- scales and integrates into shared environments like apache zeppelin; jupyterhub; SageMaker; ...
|
|
|
|
|
- Implements best-pracices for collaborative environments like apache zeppelin; jupyterhub; SageMaker; ...
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Learn More
|
|
|
|
|
|