Ingest from Bucket Workflow
This workflow allows you to ingest data from an AWS or GCP bucket for which you have credentials.
For more detailed information about what this workflow is doing, see the ingest-from-bucket
documentation in GitHub.
Creating the workflow
For general information on creating workflows in ApertureDB Cloud see Creating and Deleting Workflows.
Configure your workflow by selecting:
If using S3 it will look like this:
If using GS, it will look like this:If you are unsure how to get the credentials for your bucket, see the Getting Credentials section.
Once you have filled the credentials in, if you scroll down, you will see 3 toggles, for what types of items you want to add. They are all selected by default.
Once you have filled in the fields, click "Submit". Your workflow will be created and will start running.
See the results
Results will start being available in your database as soon as your bucket status is 'Started'.
To view data you have ingested, go to the Web UI for your instance.
Getting Credentials
If you need help getting the proper credentials for your bucket, the following are some hints which can help for users with standard configurations
Amazon Web Services
From the console, first type in 'IAM' to the search box in the top:
Select 'IAM', and find 'Users' in the menu on the left.
Once you select 'Users', find the user that you will use to access the data. Use search if you have many users. Click on the link in the 'User name' column.
Once you are in the page for the user, click on the tab on the right content side that says 'Security Credentials'
Now scroll down in the right content side until you see a section labled 'Access keys'. Choose 'Create Access Key'.Now, choose 'Application Running Outside AWS' and click on 'next'Choose a name that will mean something to you, and click 'Create access key'Now retrieve the information to your access key, either by selecting the copy for the access key and secret key, or by downloading a csv.
Once you no longer need your key, delete it.
Google Cloud
First, log into your google cloud console.
Select "IAM" from the hamburger menu.
Next, select "Service accounts" from the menu on the left.
Once at the service accounts, if you need to create an account, choose 'Create Service Account' and choose a name.
From this page you select your account by clicking on the name.
Select the 'keys' tab.
From the keys tab, select 'Add Key'.
This will launch a popup. The default type should be 'json', which is what we require here. Click on Create.
It will display a file download dialog. Save the file, then open in any file editor, and that is the key to copy into the configuration. When finished with the key, you should delete it by pressing the trashcan in the keys menu for the service account.