You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
data-transport/transport/warehouse/iceberg.py

152 lines
5.7 KiB
Python

"""
dependency:
- spark and SPARK_HOME environment variable must be set
self._session = SparkSession.builder.appName("data-transport").getOrCreate()
self._mode = 'append' if 'mode' not in _args else _args['mode']
rdd.coalesce(10).write.format('iceberg').mode('append').save(_table)