Self-hosted S3 Compatible Gateway

For a complete list of the supported architectures and API calls for the S3 Gateway, see S3 Compatibility.

Minimum Requirements

✅ 1 CPU

✅ 2GB of RAM

Depending on the load and throughput, more resources may be required.

Dependencies

✅ Uplink CLI Tutorial

✅ Create Access Grant in CLI or Create an Access Grant

Steps:

1. Get and install Gateway ST

2. Configure Gateway ST

3. Run Gateway ST

4. Configure AWS CLI to use Gateway ST

5. Try it out!

Get and install Gateway ST

Download, unzip, and install the binary for your OS:

curl https://github.com/storj/gateway-st/releases/latest/download/gateway_windows_amd64.exe.zip -o gateway_windows_amd64.exe.zip; Expand-Archive gateway_windows_amd64.exe.zip -Destination . -Force

curl https://github.com/storj/gateway-st/releases/latest/download/gateway_windows_amd64.exe.zip -o gateway_windows_amd64.exe.zip; Expand-Archive gateway_windows_amd64.exe.zip -Destination . -Force

Configure Gateway ST

You have two ways to configure your Gateway ST:

1. Interactive Setup (only if it is your first setup)

2. Using an Access Grant

Interactive Setup

1. Setup your S3 gateway by running the following command and following the instructions provided by the wizard:

./gateway.exe setup

./gateway.exe setup

2. Enter the numeric choice or satellite address corresponding to the satellite you've created your account on.

The satellite address should be entered as <nodeid>@<address>:<port> for example: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777, or just use the number from the list:

3. Choose an access name (this step may not yet be implemented in the version of S3 Gateway you are using - if you don't see this prompt, skip to step 5 below).

4. Enter the Create Access Grant in CLI you generated:

5. Create and confirm an encryption passphrase, which is used to encrypt your files before they are uploaded:

6. Your S3 gateway is configured and ready to use!

Using an Access Grant

You can use two methods to obtain an Access Grant:

1. Create an Access to an Object

2. Create an Access Grant

Now we got our access grant and can configure the gateway as follows:

./gateway setup --access 14aV.... --non-interactive

./gateway setup --access 14aV.... --non-interactive

This command will register the provided access as the default access in the gateway config file.

Run Gateway ST

The gateway functions as a daemon. Start it and leave it running.

./gateway.exe run

./gateway.exe run

The gateway should output your S3-compatible endpoint, access key, and secret key.

Configure AWS CLI to use Gateway ST

Once you do, in a new terminal session, configure it with your Gateway's credentials:

$ aws configure
---
AWS Access Key ID: [Enter your Gateway's Access Key]
AWS Secret Access Key: [Enter your Gateway's Secret Key]
Default region name: [null]
Default output format: [null]

$ aws configure set default.s3.multipart_threshold 64MB

$ aws configure
---
AWS Access Key ID: [Enter your Gateway's Access Key]
AWS Secret Access Key: [Enter your Gateway's Secret Key]
Default region name: [null]
Default output format: [null]

$ aws configure set default.s3.multipart_threshold 64MB

Then, test out some AWS S3 CLI commands!

Try it out!

Create a bucket

aws s3 --endpoint=http://localhost:7777 mb s3://bucket-name

aws s3 --endpoint=http://localhost:7777 mb s3://bucket-name

Upload an object

aws s3 --endpoint=http://localhost:7777 cp your-large-file.mp4 s3://bucket-name/your-large-file.mp4

aws s3 --endpoint=http://localhost:7777 cp your-large-file.mp4 s3://bucket-name/your-large-file.mp4

List objects in a bucket

aws s3 --endpoint=http://localhost:7777 ls s3://bucket-name/

aws s3 --endpoint=http://localhost:7777 ls s3://bucket-name/

Download an object

aws s3 --endpoint=http://localhost:7777 cp s3://bucket-name/your-large-file.mp4 ~/Downloads/your-large-file.mp4

aws s3 --endpoint=http://localhost:7777 cp s3://bucket-name/your-large-file.mp4 ~/Downloads/your-large-file.mp4

Generate a URL for an object

aws s3 --endpoint=http://localhost:7777 presign s3://bucket-name/your-large-file.mp4

aws s3 --endpoint=http://localhost:7777 presign s3://bucket-name/your-large-file.mp4

(This URL will allow live video streaming through your browser or VLC)

Delete an object

aws s3 --endpoint=http://localhost:7777 rm s3://bucket-name/your-large-file.mp4

aws s3 --endpoint=http://localhost:7777 rm s3://bucket-name/your-large-file.mp4

All Commands

cp - Copies a local file or S3 object to another location locally or in S3

ls - List S3 objects and common prefixes under a prefix or all S3 buckets

mb - Creates an S3 bucket

mv - Moves a local file or S3 object to another location locally or in S3.

presign - Generate a pre-signed URL for an S3 object. This allows anyone who receives the pre-signed URL to retrieve the S3 object with an HTTP GET request.

rb - Deletes an empty S3 bucket

rm - Deletes an S3 object

sync - Syncs directories and S3 prefixes. Recursively copies new and updated files from the source directory to the destination. Only creates folders in the destination if they contain one or more files

And that's it! You've learned how to use our S3-compatible Gateway. Ideally, you'll see how easy it is to swap out AWS for the Uplink, going forward.

Advanced usage

Advanced usage for the single-tenant Gateway

Adding Access Grants

You can add several access grants to the config.yaml, using this format:

access: 14aV.... # default Access
accesses.site: 26NBm..... # the Access with name "site"

access: 14aV.... # default Access
accesses.site: 26NBm..... # the Access with name "site"

You can see the path to the default config file config.yaml with this command:

./gateway help

./gateway help

Running options

You can run a gateway with specifying the access grant (or its name for example --access site) with the option --access, for example:

./gateway run --access 14aV....

./gateway run --access 14aV....

Running Gateway ST to host a static website

You can also run a gateway to handle a bucket as a static website.

./gateway run --access 14aV.... --website

./gateway run --access 14aV.... --website

Now you can navigate to http://localhost:7777/site/ to see the bucket site as XML or to http://localhost:7777/site/index.html to see a static page, uploaded to the bucket site.

Running Gateway ST to host a static website with cache

You can use the Minio caching technology in conjunction with the hosting of a static website.

The following example uses /mnt/drive1, /mnt/drive2 ,/mnt/cache1 ... /mnt/cache3 for caching, while excluding all objects under bucket mybucket and all objects with '.pdf' extensions on a S3 Gateway setup. Objects are cached if they have been accessed three times or more. Cache max usage is restricted to 80% of disk capacity in this example. Garbage collection is triggered when the high watermark is reached (i.e. at 72% of cache disk usage) and will clear the least recently accessed entries until the disk usage drops to the low watermark - i.e. cache disk usage drops to 56% (70% of 80% quota).

Export the environment variables before running the Gateway:

$env:MINIO_CACHE="on"
$env:MINIO_CACHE_EXCLUDE="*.pdf,mybucket/*"
$env:MINIO_CACHE_QUOTA=80
$env:MINIO_CACHE_AFTER=3
$env:MINIO_CACHE_WATERMARK_LOW=70
$env:MINIO_CACHE_WATERMARK_HIGH=90

$env:MINIO_CACHE="on"
$env:MINIO_CACHE_EXCLUDE="*.pdf,mybucket/*"
$env:MINIO_CACHE_QUOTA=80
$env:MINIO_CACHE_AFTER=3
$env:MINIO_CACHE_WATERMARK_LOW=70
$env:MINIO_CACHE_WATERMARK_HIGH=90