Skip to main content

Ingest from Bucket Workflow

This workflow allows you to ingest data from an AWS or GCP bucket for which you have credentials.

For more detailed information about what this workflow is doing, see the ingest-from-bucket documentation in GitHub.

Creating the workflow​

For general information on creating workflows in ApertureDB Cloud see Creating and Deleting Workflows.

Configure your workflow by selecting:

If using S3 it will look like this:

S3 Options setup dialog

If using GS, it will look like this:

GS Options setup dialog

If you are unsure how to get the credentials for your bucket, see the Getting Credentials section.

Once you have filled the credentials in, if you scroll down, you will see 3 toggles, for what types of items you want to add. They are all selected by default.

Common Options setup dialog

Once you have filled in the fields, click "Submit". Your workflow will be created and will start running.

See the results​

Results will start being available in your database as soon as your bucket status is 'Started'.

To view data you have ingested, go to the Web UI for your instance.

Getting Credentials​

If you need help getting the proper credentials for your bucket, the following are some hints which can help for users with standard configurations

Amazon Web Services​

From the console, first type in 'IAM' to the search box in the top:

Locate IAM

Select 'IAM', and find 'Users' in the menu on the left.

Locate Users in IAM

Once you select 'Users', find the user that you will use to access the data. Use search if you have many users. Click on the link in the 'User name' column.

Select User

Once you are in the page for the user, click on the tab on the right content side that says 'Security Credentials'

Find Security Credentials

Now scroll down in the right content side until you see a section labled 'Access keys'. Choose 'Create Access Key'.

Select Create Access Key

Now, choose 'Application Running Outside AWS' and click on 'next'

Select Create Access Key

Choose a name that will mean something to you, and click 'Create access key'

Name Access Key

Now retrieve the information to your access key, either by selecting the copy for the access key and secret key, or by downloading a csv.

Name Access Key

Once you no longer need your key, delete it.

Delete Access Key

Google Cloud​

First, log into your google cloud console.

Select "IAM" from the hamburger menu.

Select IAM

Next, select "Service accounts" from the menu on the left.

Select Service Accounts

Once at the service accounts, if you need to create an account, choose 'Create Service Account' and choose a name.

From this page you select your account by clicking on the name.

Select Account

Select the 'keys' tab.

Select Keys Tab

From the keys tab, select 'Add Key'.

Add Key

This will launch a popup. The default type should be 'json', which is what we require here. Click on Create.

Choose Key Type

It will display a file download dialog. Save the file, then open in any file editor, and that is the key to copy into the configuration. When finished with the key, you should delete it by pressing the trashcan in the keys menu for the service account.

Delete Keys