data-transport

4.3 KiB

Raw Blame History

None <html lang="en"> <head> </head>

Writing to Google Bigquery¶

Insure you have a Google Bigquery service account key on disk
The service key location is set as an environment variable BQ_KEY
The dataset will be automatically created within the project associated with the service key

The cell below creates a dataframe that will be stored within Google Bigquery

In [3]:

#
# Writing to Google Bigquery database
#
import transport
from transport import providers
import pandas as pd
import os

PRIVATE_KEY = os.environ['BQ_KEY'] #-- location of the service key
DATASET = 'demo'
_data = pd.DataFrame({"name":['James Bond','Steve Rogers','Steve Nyemba'],'age':[55,150,44]})
bqw = transport.get.writer(provider=providers.BIGQUERY,dataset=DATASET,table='friends',private_key=PRIVATE_KEY)
bqw.write(_data,if_exists='replace') #-- default is append
print (['data transport version ', transport.__version__])

100%|██████████| 1/1 [00:00<00:00, 10106.76it/s]

['data transport version ', '2.0.4']

Reading from Google Bigquery¶

The cell below reads the data that has been written by the cell above and computes the average age within a Google Bigquery (simple query).

Basic read of the designated table (friends) created above
Execute an aggregate SQL against the table

NOTE

By design read object are separated from write objects in order to avoid accidental writes to the database. Read objects are created with transport.get.reader whereas write objects are created with transport.get.writer

In [2]:

import transport
from transport import providers
import os
PRIVATE_KEY=os.environ['BQ_KEY']
pgr = transport.get.reader(provider=providers.BIGQUERY,dataset='demo',table='friends',private_key=PRIVATE_KEY)
_df = pgr.read()
_query = 'SELECT COUNT(*) _counts, AVG(age) from demo.friends'
_sdf = pgr.read(sql=_query)
print (_df)
print ('--------- STATISTICS ------------')
print (_sdf)

Downloading: 100%|██████████|
Downloading: 100%|██████████|
           name  age
0    James Bond   55
1  Steve Rogers  150
2  Steve Nyemba   44
--------- STATISTICS ------------
   _counts   f0_
0        3  83.0

An auth-file is a file that contains database parameters used to access the database. For code in shared environments, we recommend

Having the auth-file stored on disk
and the location of the file is set to an environment variable.

To generate a template of the auth-file open the file generator wizard found at visit https://healthcareio.the-phi.com/data-transport

In [ ]:

</html>

4.3 KiB Raw Blame History

Writing to Google Bigquery¶

Reading from Google Bigquery¶

4.3 KiB

Raw Blame History