Adding data from AWS S3

See how you can connect your own AWS S3 buckets to tasks on Mindkosh.

To connect a new task with AWS S3, follow these steps.

Create an IAM user on AWS

Create an IAM user on AWS, and attach a policy with read access rights over the bucket and location you intend to use. When asked to choose the AWS credential type. select Access key - Programmatic access. We do not require AWS console access.

An example policy you can attach to the new user is given below. Be sure to replace example-bucket with your own bucket name.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::<example-bucket>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<example-bucket>/*"
            ]
        }
    ]
}

Remember to save the API Access keys

Once you have created a new user, you will be prompted to save the Access keys you just created. Make sure to click on the "Download .csv" button. If you close the tab, and reopen it, you will not be able to view the Secret access key required to access data from your bucket.

Add access keys

Once you have the access keys for the new user, add them to your organization. Go to the Organization page and click on the Manage AWS access keys button. Next, enter the Access keys in the dialog box that opens.

You only need to do this once. Once the keys are added, they will be used to access data for any task created with S3 by any member of your organization. Remember, if you do not want to give a user the ability to create tasks, move him to the Annotator user group from the Organization page.

Create a task with S3

Enter the location of the images when creating the task. We will scan the location for all image files, and create a task from them. To make sure you enter the right location, click on the Copy S3 URI button in the top right corner of location's page on AWS S3 console.

Note that we only save the location of the images and not the images themselves. We then stream the images directly from the cloud using short-lived pre-signed URLs to access the images from the browser.