You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
166 lines
8.4 KiB
HTML
166 lines
8.4 KiB
HTML
<script>
|
|
_layout = {on:{load:{'plugin-intro':['www/html/_notes/plugins-intro.html']}}}
|
|
var _plugins = {}
|
|
_plugins.open = function (_id){
|
|
$('.tab-content').slideUp(function (){
|
|
})
|
|
_id = '.'+_id
|
|
$(_id).slideDown()
|
|
|
|
}
|
|
_plugins.copy = function(_node) {
|
|
var _code = $(_node.parentNode).text().trim().replace(/ {8}/g,'').replace(/ {4}/g,'\t').replace(/\r/g,'\n')
|
|
navigator.clipboard.writeText(_code);
|
|
$(_node).empty()
|
|
$(_node).html('<i class="fa-solid fa-check"></i>')
|
|
setTimeout(function(){
|
|
$(_node).empty()
|
|
$(_node).html('<i class="fa-regular fa-copy"></i>')
|
|
},750)
|
|
}
|
|
$(document).ready(function(){
|
|
bootup.init('{{system.context}}',_layout)
|
|
// _plugins.open('makeplug-src')
|
|
var label = $('.process-steps .tabs label')[0]
|
|
label.click()
|
|
})
|
|
|
|
</script>
|
|
<style>
|
|
/* .source-code {background-color:#f3f3f3; color:black;} */
|
|
.source-code b {color: yellow;}
|
|
.split-1 {display:grid; grid-template-columns: 45% auto; gap:4px}
|
|
.split-2 {display:grid; grid-template-columns: 55% auto; gap:4px}
|
|
|
|
.process-steps .list { font-size:16px; padding:4px; margin-left:8px; background-color: #f3f3f3;}
|
|
.process-steps .list div {margin:8px;}
|
|
/* .fa-copy {float:right; cursor:pointer; padding:4px; margin:4px; background-color: transparent;} */
|
|
/* .fa-copy:hover {color:#4682b4;} */
|
|
</style>
|
|
<div style="min-height:650px;">
|
|
<div class="split-2">
|
|
<div>
|
|
<h3>Plugins: Usage & Development</h3>
|
|
<p>
|
|
<div id="plugin-intro" >
|
|
</div>
|
|
</p>
|
|
<p>
|
|
<h3>Plugins: Registry</h3>
|
|
|
|
The plugins registry is a registry of plugins intended to be used in pre/post processing. This feature comes in handy :
|
|
<ul>
|
|
<div><i class="fa-solid fa-minus"></i> During ETL: Cleanup data, adding columns enforcing data-typing, removing/encrypting PHI ... </div>
|
|
<div><i class="fa-solid fa-minus"></i> In a collaborative environment (Jupyter-x; Zeppelin; AWS Service Workbench)</div>
|
|
</ul>
|
|
|
|
|
|
</p>
|
|
<p>
|
|
<h3>Plugins: Architecture & Design</h3>
|
|
Plugins are designed around <b class="active" onclick="window.open('https://en.wikipedia.org/wiki/Plug-in_(computing)','_doc')">plugin architecture</b> using <b class="active" onclick="window.open('https://en.wikipedia.org/wiki/Iterator_pattern','doc')">Iterator design-pattern</b>. In that respect and function as a <b>pipeline</b> i.e executed sequentially in the order in which they are expressed in the parameter. Effectively the output of one function will be the input to the next.
|
|
<p>
|
|
<div class="border figure" align="center" >
|
|
<img src="www/html/_assets/images/plugins-components.png">
|
|
<div class="small border-top" align="center" style="margin-top:4px; padding-top:4px;">Data Transport UML Plugin Component View</div>
|
|
</div>
|
|
|
|
</p>
|
|
</p>
|
|
</div>
|
|
<div class="process-steps" >
|
|
<h3>Quick Start</h3>
|
|
<div class="tabs">
|
|
<input type="radio" name="step" id="makeplug">
|
|
<label for="makeplug" onclick="_plugins.open('makeplug-src')">1. Make Plugin</label>
|
|
|
|
<input type="radio" name="step" id="regplug">
|
|
<label for="regplug" onclick="_plugins.open('regplug-src')">2. Register Plugin</label>
|
|
|
|
<input type="radio" name="step" id="useplug">
|
|
<label for="useplug" onclick="_plugins.open('useplug-src')">3. Use The Plugin</label>
|
|
|
|
</div>
|
|
|
|
<div class="tab-content makeplug-src">
|
|
<p>
|
|
|
|
<div class="split-x">
|
|
|
|
|
|
<div>
|
|
<ul class="list">
|
|
<div><i class="fa-solid fa-minus"></i> The code here shows a function that will be registered as <b><i>"autoincrement"</i></b>.</div>
|
|
<div><i class="fa-solid fa-minus"></i> The data, will always be a <b>pandas.DataFrame</b></b></div>
|
|
<div><i class="fa-solid fa-minus"></i> For the sake of this example the file will be <b>my-plugin.py</b></div>
|
|
</ul>
|
|
|
|
</div>
|
|
<div class="source-code"><i class="fa-regular fa-copy" onclick="_plugins.copy(this)"></i>
|
|
<b>import</b> transport
|
|
<div><b>import</b> numpy <b>as</b> np</div>
|
|
<br>
|
|
<br>_index = 0
|
|
<br><div>@transport.Plugin(name='autoincrement')</div>
|
|
<div><b>def</b> _incr (_data):</div>
|
|
<ul>
|
|
<div><b>global</b> _index</div>
|
|
<div>_data['_id'] = _index + np.arange(_data.shape[0])</div>
|
|
<div>_index = _data.shape[0]</div>
|
|
<div><b>return</b> _data</div>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
</p>
|
|
</div>
|
|
<!-- register plugin-->
|
|
<div class="tab-content regplug-src">
|
|
<p>
|
|
<div class="">
|
|
|
|
<div>
|
|
<b>data-transport</b> comes with a built-in command line interface (CLI). It allows plugins to be registered and reused.
|
|
<ul class="list">
|
|
<div><i class="fa-solid fa-minus"></i> Registered functions are stored in <b><i>$HOME/.data-transport/plugins/code</i></b></div>
|
|
<div><i class="fa-solid fa-minus"></i> Any updates to <b>my-plugin.py</b> will require re-registering the file</div>
|
|
<div><i class="fa-solid fa-minus"></i> Additional plugin registry functions (list, test) are available </div>
|
|
|
|
</ul>
|
|
</div>
|
|
<div>
|
|
<div class="source-code"><i class="fa-regular fa-copy" onclick="_plugins.copy(this)"></i>
|
|
<b>$</b> transport plugin-add demo ./my-plugin.py
|
|
</div>
|
|
<p>
|
|
The following command allows <b>data-transport</b> to determine what is knows about the function i.e <b>real name</b> and <b>name</b> to be used in code.
|
|
<div class="source-code"><i class="fa-regular fa-copy" onclick="_plugins.copy(this)"></i>
|
|
<b>$</b> transport plugin-test demo.autoincrement
|
|
</div>
|
|
</p>
|
|
</div>
|
|
</div>
|
|
</p>
|
|
</div>
|
|
<!-- useplug-src -->
|
|
<div class="tab-content useplug-src">
|
|
<p>
|
|
Once registered, the plugins are ready for use within code or configuration file (auth-file).
|
|
<div class="source-code"><i class="fa-regular fa-copy" onclick="_plugins.copy(this)"></i><b>import</b> transport
|
|
<div><b>from</b> transport <b>import</b> providers</div>
|
|
|
|
_args = {
|
|
<div>"provider":providers.HTTP,</div>
|
|
<div>"url":"https://raw.githubusercontent.com/codeforamerica/ohana-api/master/data/sample-csv/addresses.csv",</div>
|
|
<div>"plugins":["demo@autoincrement"]</div>
|
|
}
|
|
|
|
<div>reader = transport.get.reader(**_args)</div>
|
|
<div>_data = reader.read()</div>
|
|
<div>print (_data.head())</div>
|
|
|
|
</div>
|
|
</p>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div> |