data-transport
/
data-transport
1
1
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity
Labels Milestones
New Issue

Enable Streaming #26

Closed
opened 2 years ago by steve · 1 comments
steve commented 2 years ago
Owner

It's important to enable streaming (for large volumes of data) for reads/writes

  • SQLAlchemy
  • pyspark
It's important to enable streaming (for large volumes of data) for reads/writes - [ ] SQLAlchemy - [ ] pyspark
steve added this to the version 2.4 milestone 2 years ago
steve commented 2 years ago
Poster
Owner

with sqlalchemy we can just specify chucksize parameter to generate streams. In the context of ETL this should be accounted for i.e writing down each chunk

with sqlalchemy we can just specify chucksize parameter to generate streams. In the context of ETL this should be accounted for i.e writing down each chunk
steve closed this issue 2 years ago
steve referenced this issue from a commit 6 months ago
Merge pull request #26 from lnyemba/v2.2.0
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
v2.2.0
v2.4
v2.0.4
v2.0
dev
nextcloud
s3
Labels
Apply labels
Clear labels
No items
No Label
Milestone
Set milestone
Clear milestone
No items
No Milestone
version 2.4
Projects
Clear projects
No project
Assignees
Assign users
Clear assignees
No Assignees
1 Participants
Notifications
Due Date

No due date set.

Dependencies

No dependencies set.

Reference: data-transport/data-transport#26
Write Preview
Loading…
Cancel
Save
There is no content yet.
Delete Branch '%!s(<nil>)'

Deleting a branch is permanent. It CANNOT be undone. Continue?

No
Yes