# Connectome Data Platform

Connectome Data Platform


# Quick Start

Getting started with Connectome is really simple and fun and it involves three main steps. 1) Import data; 2) Manage data; and 3) Make use of data.

#### 1. Create a workspace

A workspace is like a regular office in your company, which could consist of a sales, a marketing, a finance, and an operation department. You will get prompted for creating a new workspace when you first sign up, or you can create a new one by clicking on your existing workspace name on the top-left side of your screen, which shows a dropdown and then you can click Create New. One account can have as many workspaces as you need. Maybe your enterprise has multiple business units, for instance, an enterprise may have a F\&B business, a logistic business and a digital business.

<figure><img src="/files/hh2Dw6s5eYFsQGK6eahQ" alt=""><figcaption></figcaption></figure>

#### 2. Connect a data source

Once you set up your workspace, you can connect data sources to your workspace. Data source is where our data platform can draw data from. It can be a database (e.g. Microsoft SQL Server, Google Sheet), an existing IT system in your business (e.g. Oracle's netsuite, POS system), or simply upload an existing data file you have (e.g. Excel spreadsheet, CSV). Once we connect to your data source, the data will be imported to the data platform. The imported data will form a Collection, which could be understood as a "dataset".

<figure><img src="https://assets.website-files.com/6304ed6e20eae75b2bbb2afa/63fdc0b7e20c7538ef74989d_Screenshot%202023-02-28%20at%204.51.59%20PM.png" alt=""><figcaption></figcaption></figure>

#### 3. View your data

Once you connect with different data sources, you can find your collection in our data catalog, which is a centralized access point to all your data. By centralizing the access, we simplify the way you manage your data, improved overall security and simplify the use of data within your organization.

<figure><img src="https://assets.website-files.com/6304ed6e20eae75b2bbb2afa/63fdc14df94aa0f83249835f_Screenshot%202023-02-28%20at%204.53.49%20PM.png" alt=""><figcaption></figcaption></figure>

By clicking one of the collections, you can get access to the collection page. We have build tools that help you get ready with your data, for example, we help you better format your date column so you can filter by date, or we help you merge two datasets to create new dataset, for example if you have a collection storing member data and another collection storing purchase data, we can merge two dataset to create a purchase list detailing the member detail for each transaction.

<figure><img src="https://assets.website-files.com/6304ed6e20eae75b2bbb2afa/63fdc2e7e9908468522c959f_Screenshot%202023-02-28%20at%205.01.13%20PM.png" alt=""><figcaption><p>You can create new columns by clicking on the plus button on the right of the table at one of your collection page.</p></figcaption></figure>

#### 4. Make use of your data now

There are so many ways you can make use of your data, create visualization to discover insight, create automation to streamline operation, develop new apps to create new products/services. You can achieve any of these by using one of many integrations we have at the data platform.

<figure><img src="https://assets.website-files.com/6304ed6e20eae75b2bbb2afa/63fdc48c89f4f040ee80bb8d_Screenshot%202023-02-28%20at%205.08.19%20PM.png" alt=""><figcaption><p>You can access the integration panel by clicking the Integration button right above the data table in your collection page.</p></figcaption></figure>


# Workspace Invite

Creating Invites

You can add new users by adding their registration email. You can add the users before they signup as well, as long as they are using the same email when they sign up.

<figure><img src="/files/r8Z0CKaT43nnmv79pZgK" alt=""><figcaption></figcaption></figure>

Click **Add New Users** at the upper right to open the pop up window.

<figure><img src="/files/I2oajgPlcAP87X4doc6a" alt=""><figcaption></figcaption></figure>

## Accepting Invites

1. You will head to See All Workspace located at the dropdown menu for your workspaces.
2. At the top of the window, you will see the invites you have.&#x20;
3. Click accept and you will be redirected to the workspace.

<figure><img src="/files/Ebf6hfzIM3Q9usEE4PQR" alt=""><figcaption></figcaption></figure>


# Connect to Google Sheet

Google sheet is one of the most used spreadsheet program in the world, but it has its short comings like the limited of rows, not able to create dashboard out of it and data governance.

1. Navigate to Add Collection Page by clicking Add New Collection

<figure><img src="/files/jKeJckJumQu7NRQWzTuy" alt=""><figcaption><p>You can find the button at the side menu</p></figcaption></figure>

2. Find the Google Sheet Integration Item

<figure><img src="/files/dyiy98aUzK89nJZUHcFX" alt=""><figcaption><p>It's under 3rd Party Integration</p></figcaption></figure>

3. Find the instruction and enter the detail for your spreadsheet

<figure><img src="/files/Akhi3I4qVR51QFIsa51a" alt=""><figcaption></figcaption></figure>

4. Copy the service account email and add it as your editor in your spreadsheet

To put it simply, a [service account](https://cloud.google.com/iam/docs/service-account-overview) is an account used by computer to access information in Google. By adding our service account (which is managed by Google Cloud) to your spreadsheet, you are connecting your spreadsheet with our platform which is secured by Google Cloud.&#x20;

<figure><img src="/files/9ZgM67Cv5kNXjiyeq9nP" alt=""><figcaption><p>You just have to paste the email onto the input field</p></figcaption></figure>

5. Copy the spreadsheet URL, sheet name, data range back to Connectome

<figure><img src="/files/gG3kkMtVM6SiRP6XdCrC" alt=""><figcaption><p>Copy your spreadsheet url to Connectome</p></figcaption></figure>

**Sheet Name** can be found at the bottom of your screen

<figure><img src="/files/r4dlhKZJjroRUz1npiEk" alt=""><figcaption></figcaption></figure>

**Data Range** indicates which range of data will you like us to access in Connectome. An example will be A:B if you want us to access all cells in column A and column B.

6. Press Next and you will be brought to the collection page linked to your spreadsheet!


# Upload Custom Data

You may already have existing data that you want to upload to our platform. Connectome accepts custom file upload for the common file formats so you can get started quickly.

&#x20;Currently, we support CSV, Excel, and PDF (beta) formats, but we are planning to incorporate more data format in the future!

<figure><img src="/files/eXQ2BqEn0YDvycMdVkJw" alt=""><figcaption></figcaption></figure>


# API

Got your own APIs? Connect your platform to ours with the easy webhook service

<figure><img src="/files/r75nYxifF9ycpsXmfdzI" alt=""><figcaption></figcaption></figure>

We will make a GET request to your data platform and do it periodically so your data will be synced with us.&#x20;

Sometimes you would like us to access a subfield of the returned data only, you can specify the fields at step 5.&#x20;


# Webhook

Instead of us making frequent request to your service, you can send us data whenever you want via webhook.

Sometimes it may be inefficient when we are making periodic requests to your service. Maybe your data renews in a very irregular pattern or your data changes frequently, you may want to push your data to Connectome by yourself. You can make use of our webhook to set up a connection instead.

Connectome's webhook makes use of our DATA API. Further syntax can be found [here](/advance/data-api).


# Collection

Collection can be understood as a data table. It can be a sales, membership or product catalog dataset.

<figure><img src="/files/emBe5vg58lKJEVzfZFzy" alt=""><figcaption></figcaption></figure>

Similar to an Excel spreadsheet, collection stores your data. Unlike your Excel spreadsheet, collection can store and access million and million rows of data with ease without breaking your computer.&#x20;

## Querying your data

For the initiated, we use [MongoDB](https://www.mongodb.com/) as our primary data storage platform, meaning you can make use of their query functionality to access your data.&#x20;

But our main mission is to enable non-technical users to also make use of their own data. We use OpenAI's [GPT4](https://openai.com/gpt-4), one of the world most powerful natural language AI. Users can use natural language to find the data they need, instead of learning how to write query, SQL or any programming language.

<figure><img src="/files/k8DIG96QKo9AgT18R8B1" alt=""><figcaption><p>Our AI will convert your prompt into a query which you can then use to find the data you need.</p></figcaption></figure>

##


# Export to Excel

Coming soon


# Google Cloud


# Overview

Our platform is built on top of [Google Cloud](https://cloud.google.com/), one of the top three cloud service providers in the world, providing you with the best-performing and most secured cloud services. We make use different of their service to enrich your data experience meanwhile providing an easy-to-use UI so you do not have to be a cloud engineers or data scientists before you can drive value with your data.

<figure><img src="/files/OPHxXisnSnxgtASsWOTu" alt=""><figcaption></figcaption></figure>


# Google Cloud Storage

We can periodically (as frequently as every 15 minutes) upload your data as tabular data (CSV format) to Google Cloud Storage. You can then use the uploaded data for downstream usecase, such as, integrating with Google Cloud's looker studio.

<figure><img src="/files/meC0iL3GAuAuRymcufHU" alt=""><figcaption></figcaption></figure>

You can create multiple file to Cloud storage that is different from each other.&#x20;


# Looker Studio

Setting up a looker studio with a Connectome data source involves the following steps:

1. Setting up Google Cloud Storage&#x20;
2. Setting up Looker Studio data source with Google Cloud Storage connection string
3. Done! :relaxed:

## Getting Connection String

In the [previous session](/google-cloud/google-cloud-storage), we covered how to set up Google Cloud Storage. You can then click **View Connection String** to view your connection string.

<figure><img src="/files/vVioS1hYyRQds3OW5Xii" alt=""><figcaption></figcaption></figure>

## Setting up Looker Studio Data Source

You will then copy the connection string and paste it into the field in Looker Studio.

<figure><img src="/files/JV7vIiCeD4SSFJ487ERM" alt=""><figcaption></figcaption></figure>


# DATA API

Connect your data to your app instantly and securely

Our mission is to empower organizations by **democratizing access to data**. We believe that seamless and secure data access is the key to unlocking its full potential. Our Data API offers a user-friendly and secure gateway to your organization's data assets, ensuring that you can access and manage your data anytime, anywhere, without compromising on security.

## What you can do with Data API

* Read (with query or not)
* Write (with batch or not)
* Update
* Delete

## How to access Data API

1. Go to your collection&#x20;
2. Click Integration and select API
3. The description shows how you can access the API using different programming language.

<figure><img src="/files/IBdL4HC0v1D76joP1rKh" alt=""><figcaption></figcaption></figure>

## How to use API call

### Read Data with query

By default, the read API return all the data in the collection which can be inefficient and so we recommend you use **read with query**. Essentially, it allows you to **1)** filter, **2)** project (select the fields you want), **3)** sort, and **4)** limit (limit the amount of data returned). For example, imagine your collection looks like this in JSON format.

```json
[
    {
        "amount": 10,
        "title": "Movie ABC"
    }, 
    {
        "amount": 100,
        "title": "Movie BDD"
    }, 
    {
        "amount": 39,
        "title": "Movie XYZ"
    }
]
```

You can access your data using the `requests` library with Python. The syntax for filter is based on [MongoDB query language](https://www.mongodb.com/basics/examples).&#x20;

```python
# Import Library
import requests

# Request data using POST
data = requests.post(
    url='https://app.connectomesystem.com/collection/readwithquery/<read credential>/<collection id>',
    json={
        "query": {"amount":{"$gte":30}},
        "project": {"amount": 1, "title": 1},
        "sort": {"amount": 1},
        "limit": 10
        }
    )
    
```

### Update data with upsert

Allows to update the data in the specific field(s) and insert new data if the data in search do not exist.

#### Usage:

```python
# Import library
import requests

# Only the POST method allows in the update method.
resp = requests.post(
    url='https://app.connectomesystem.com/collection/updatewithupsert/<write credential/<collection id>',
    json={
        "search query": {
            "<field name 1>": "foo",
            "<field name 2>": "bar"
        },
        "data": {
            "<field name 3>": "baz"
        },
        "upsert": True
    }
)
```

<table><thead><tr><th width="187">Key</th><th>Description</th></tr></thead><tbody><tr><td>search query</td><td>Used to identify which data need to be updated. Field names are the fields that are defined in your collection.</td></tr><tr><td>data</td><td>The modification to apply</td></tr><tr><td>upsert</td><td>Boolean. Creates a new document if no documents match the <code>filter</code>. For more details see <a href="https://www.mongodb.com/docs/manual/reference/method/db.collection.update/#std-label-upsert-behavior">upsert behavior.</a> Default to <code>False</code>.</td></tr></tbody></table>

Noted that this function can return an error code due to no result found by the search query.

### Upload with batch data

Inserts multiple documents into a collection

#### Usage:

```python
# Import library
import requests

# Only the POST method allows in the upload method.
resp = requests.post(
    url='https://app.connectomesystem.com/collection/batchwrite/<write credential>/<collection id>',
    json={
        [
            {
                "field 1": "foo"
            },
            {
                "field 1": "bar"
            }
        ]
    }
)
```

This API parameter needs an array of data. The batch data will be inserted into the collection all at one time.

### Delete batch data with a query

Remove all the data that match the condition defined in the query.

#### Usage:

```python
# Import library
import requests

# Only the DELETE method allows in the remove method.
resp = requests.delete(
    url='https://app.connectomesystem.com/collection/batchdelete/<write credential>/<collection id>',
    json={
        'field 1': 'foo',
        'field 2': 'bar'
    }
)
```

The API parameter takes a query in JSON format. Any data will be removed if match the query.


# Dashboard

Instead of looking at thousands of row of data in your Excel sheet, you can create amazing visualization with our tools so you can dive deeper in your data and discover insights.

There are two main ways you can visualize your data in our platform.&#x20;

## Connectome Visualization

<figure><img src="/files/YA79ZRqsIQwasfIB9LNL" alt=""><figcaption></figcaption></figure>

Our built-in visualization tool makes use of [Perspective](https://perspective.finos.org/), an open-source data visualization tool. This tool gives you the most secure way to create quick visualization for your data. You can:

* Filtering&#x20;
* Create multiple charts on one page
* Save charts
* Export charts as PNG, Excel

## Looker Studio

[Looker Studio](https://developers.google.com/looker-studio/visualization) is a free visualization tool offered by Google Cloud. You can integrate your data to Looker via Google Cloud Storage.&#x20;

<figure><img src="/files/VPBGgwwt7vx2CTOTUzSQ" alt=""><figcaption></figcaption></figure>

Users will be able to open the dashboard on their PC, tablet and mobile phone.&#x20;


# Note

Writing notes allows you stay organized with your projects.

Documentation is crucial for several reasons:

1. **Reproducibility**: It serves as a guide to reproduce analysis, ensuring the results obtained are not a one-time event.
2. **Collaboration**: It facilitates the sharing of knowledge and methods across teams, enhancing collaborative efforts.
3. **Clarity**: It provides clear understanding about the purpose and workings of the code, making it easier to understand and modify.
4. **Onboarding**: It helps new team members understand the existing projects quickly.
5. **Maintenance**: It simplifies the process of updating or modifying the code in the future.
6. **Professionalism**: It exhibits a professional approach to work, reflecting the best practices in data science.

<figure><img src="/files/kKz3dQMhmx7BfV7osiLN" alt=""><figcaption><p>You can access note by clicking the note tab in the side menu.</p></figcaption></figure>

We allow [Markdown](https://www.markdownguide.org/getting-started/) in our note so you can create all sorts of note.

* Text decoration

<figure><img src="/files/4UT958sAKS72sgqmMM3k" alt=""><figcaption></figcaption></figure>

* Tables

<figure><img src="/files/4XxOotggRbRWTfa6TLJp" alt=""><figcaption></figcaption></figure>

* Hyperlink


# Report

Query data from your collection and download the data in excel format

Coming soon