Using Storj with Aspera

Integration

To integrate Storj with Aspera, you will need to create S3 credentials in Storj and add them within Aspera.

This document is written for IBM Aspera HSTS version 4.4.4 on Redhat Linux.

Refer to Aspera's Setup and Configuration in Amazon S3 documentation for more detailed configuration.

Requirements

An active Storj account
A bucket for use with IBM Aspera in your Storj instance
An valid install of IBM Aspera High Speed Transfer Server

Create an Account

To begin, you will need to create a Storj account.

Navigate to https://www.storj.io/signup?partner=Aspera to sign up, or log in https://storj.io/login if you already have an account.

Create a Bucket

Once you have your Storj account you can create a bucket for your data to be stored in.

Navigate to Browse on the left side menu.
Click on the New Bucket button.
Assign the bucket an easily identifiable name, such as "my-bucket".
Optional: Enable Object Lock (required for immutability in many applications).
- If you enable Object Lock, you can also set a default retention period using either Governance or Compliance Mode
Optional: Enable Object Versioning (note that this will be enabled by default if Object Lock is enabled)
Click Create bucket

Generate S3 credentials

Storj has an Amazon S3 compatible API and you'll need generate S3 credentials to use it. S3 credentials consist of an access key, secret key, and endpoint.

Create S3 credentials in the Storj console:

Navigate to Access Keys on the left side menu.
Click the New Access Key button.
When the New Access dialog comes up, set specifications according to the following guidelines:
- Name: The name of the credentials (e.g. my-access)
- Type: S3 Credentials
Choose Full Access or Advanced
- In most cases, you DO NOT want to choose full access.
Provide Access encryption Information
In order to see the data uploaded to your bucket in the Storj console, you must unlock the bucket with the same encryption passphrase as the credentials.
- Use the current passphrase: this is default option
- Advanced: you may provide a different encryption phrase either your own or generate a new one.
  - Enter a new passphrase: use this option, if you would like to provide your own new encryption phrase
  - Generate 12-word passphrase: use this option, if you would like to generate a new encryption phrase
Select the permissions you want to allow:
- Read
- Write
- List
- Delete
Select the object lock permissions you want to allow
- PutObjectRetention
- GetObjectRetention
- BypassGovernanceRetention
- PutObjectLegalHold
- GetObjectLegalHold
- PutObjectLockConfiguration
- GetObjectLockConfiguration
Choose the buckets you want the access to include:
- All Buckets
- Select Buckets
Set an expiration
Click Create Access to finish creation of your S3 credentials
Your S3 credentials are created. Write them down and store them, or click the Download all button. You will need these credentials for the following steps.

Object Lock Permission Details

Permission Name	Description
PutObjectRetention	Allows you to set retention policies, protecting objects from deletion or modification until the retention period expires.
GetObjectRetention	Allows you to view the retention settings of objects, helping ensure compliance with retention policies.
BypassGovernanceRetention	Allows you to bypass governance-mode retention, enabling deletion of objects before the retention period ends.
PutObjectLegalHold	Allows you to place a legal hold on objects, preventing deletion or modification regardless of retention policies.
GetObjectLegalHold	Allows you to view the legal hold status of objects, which is useful for auditing and compliance purposes.
PutObjectLockConfiguration	Allows you to set retention policies on the specified bucket, automatically applying them to every new object added to that bucket.
GetObjectLockConfiguration	Allows you to view the default retention policies configured for the specified bucket.

Connecting Aspera to Storj

These instructions are designed for IBM Aspera HSTS on Redhat.

Creating a Document Root for Storj in Aspera

Aspera user interfaces allow dynamic connections to Amazon S3, but not to S3 compatible endpoints such as Storj. Because of this limitation, it it necessary to configure an Aspera HSTS with a document root to connect to Storj. This document root will allow Aspera users to connect to Storj transparently without complex client configuration.

# vi /opt/aspera/etc/aspera.conf
# asconfigurator -x "set_user_data;user_name,[account];absolute,s3://[s3_account_id]:[s3_secret_key]]@gateway.storjshare.io/[bucket]"

# vi /opt/aspera/etc/aspera.conf
# asconfigurator -x "set_user_data;user_name,[account];absolute,s3://[s3_account_id]:[s3_secret_key]]@gateway.storjshare.io/[bucket]"

Install the Aspera Trapd Service

The Aspera Trapd service enables Aspera to write to object storage.
If you have not previously enabled it, run the following command.

dnf install initscripts chkconfig
/opt/aspera/bin/astrap-config.sh enable

dnf install initscripts chkconfig
/opt/aspera/bin/astrap-config.sh enable

You may alternatively create a similar file_system node beneath the default tag to enable Storj for all accounts.

Edit Aspera S3 Properties to Require HTTPS

For security reasons, Storj requires all S3 compatible traffic to use HTTPS rather than HTTP. Aspera's default configuration attempts to connect to S3 service via HTTP, which is incompatible with Storj. Update Aspera's s3.properties file to require HTTPS.

# vi /opt/aspera/etc/trapd/s3.properties

# vi /opt/aspera/etc/trapd/s3.properties

# Set to true to use HTTPS for s3 storage
# Default is false
s3service.https-only=true

# Set to true to use HTTPS for s3 storage
# Default is false
s3service.https-only=true

Testing Storj + Aspera Integration Locally

After restarting the Aspera service, you should be able to test local transfers to Storj using the following command:

# systemctl restart asperanoded
# ascp -P 33001 -v [testfile] [account]@localhost:/

# systemctl restart asperanoded
# ascp -P 33001 -v [testfile] [account]@localhost:/