Migrate from Wasabi to Storj

This article will discuss the migration process from Wasabi to Storj using the rclone tool, a command-line program to manage files on cloud storage.

For a cost comparison between Wasabi and Storj, see https://www.storj.io/landing/wasabi-comparison.

Prerequisites

Before starting the migration process, you'll need to have the following:

  1. Access to both your Wasabi and Storj accounts.

  2. Installed and configured rclone on your machine.

Why Rclone?

Rclone is a command-line program written in Go language, which is designed to sync files and directories from different cloud storage providers. It allows for easy migration, syncs directories and files, checks file hashes, and even modifies drives. It works with a wide range of cloud storage providers, including Wasabi and Storj, which makes it an excellent tool for our use-case.

Install rclone

Visit https://rclone.org/install/ for instructions on how to install rclone.

Create Access Credentials

Before using rclone, we'll need to create access and a secret key for Wasabi and an access grant for Storj.

Wasabi Access and Secret Key

Use your Wasabi account credentials to log in to the Wasabi Management Console.

  1. Click Access Keys from the left side menu

  2. Click CREATE ACCESS KEY button under the Access Keys heading

create a wasabi access key

Once the key is created, you can view and copy the Access Key and Secret Key. Make sure to store these securely as the Secret Key cannot be retrieved again and will be needed later on.

Create Storj Access Grant

A Storj access grant is a serialized, self-contained credential that allows users to access a specific bucket, or object, within a Storj project. It encapsulates everything needed for authentication and authorization on the Storj network.

Create Access Grant in the Storj Console:

  1. Navigate to Access Keys on the left side menu.

  2. Click the New Access Key button.

  3. When the New Access dialog comes up, set specifications according to the following guidelines:

    • Name: The name of the credentials (e.g. my-access-grant)
    • Type: Access Grant

  4. Click Next to provide permissions, either Full Access or Advanced:

    • Permissions: All

    • Buckets: Feel free to specify the bucket (e.g. my-bucket), or leave as “All”

    • End date: provide an expiration date for these credentials (optional)

  5. Click Next to provide Access encryption Information

    In order to see the data uploaded to your bucket in the web console, you must unlock the bucket with the same encryption passphrase as the credentials.

    • Use the current passphrase: this is default option

    • Advanced: you may provide a different encryption phrase either your own or generate a new one.

      • Enter a new passphrase: use this option, if you would like to provide your own new encryption phrase

      • Generate 12-word passphrase: use this option, if you would like to generate a new encryption phrase

This passphrase is important! Encryption keys derived from it are used to encrypt your data at rest, and your data will have to be re-uploaded if you want it to change!

Importantly, if you want two access grants to have access to the same data, they must use the same passphrase. You won't be able to access your data if the passphrase in your access grant is different than the passphrase you uploaded the data with.

Please note that Storj does not know or store your encryption passphrase, so if you lose it, you will not be able to recover your files.

  1. Click Create Access to finish creation of your Access key.

  2. Click Confirm the Confirm details pop-up message

  3. Your Access Grant is created. Write it down and store it, or click the Download button. You will need the Access Grant for the following steps.

Update Rclone config

After getting your access keys for Storj and Wasabi, you need to configure rclone.

We'll edit the rclone config directory directly, but you can also run rclone config for a more interactive experience.

We'll edit the rclone config file directly, you can find where it is stored by running the following:

rclone config file
rclone config file

Command output will look like this

Configuration file is stored at: /Users/dan/.config/rclone/rclone.conf

Edit ~/.config/rclone/rclone.conf with the access keys created above and the values below (see sample file).

Wasabi:

  • [wasabi]: This is the section name for the configuration.

  • type = s3: This refers to the type of storage you are interacting with, in this case 's3' for Simple Storage Service, a type of cloud storage.

  • provider = Wasabi: This indicates that the provider of the storage service is Wasabi.

  • access_key_id: Replace with your actual Wasabi access key created previously.

  • secret_access_key: Replace with your actual Wasabi secret key created previously.

  • endpoint: This points to the Wasabi service endpoint. The value depends on the region of your Wasabi storage. Replace the placeholder with the endpoint that corresponds to your Wasabi service region.

  • acl = private: This represents the Access Control List (ACL) policy for your objects. 'private' means that the objects are only accessible to the owner of the bucket.

Storj:

  • [storj]: This is the section name for the configuration.

  • type = storj: This indicates that the type of storage is Storj.

  • access_grant: The access_grant is a serialized access grant string which encapsulates all necessary information to list or download objects. Replace with the actual access grant created previously.

~/.config/rclone/rclone.conf

[wasabi]
type = s3
provider = Wasabi
access_key_id =  access_key # REPLACE ME
secret_access_key = secret_key  # REPLACE ME
endpoint = s3.us-central-1.wasabisys.com # REPLACE ME
acl = private

[storj]
type = storj
access_grant = access_grant # REPLACE ME
[wasabi]
type = s3
provider = Wasabi
access_key_id =  access_key # REPLACE ME
secret_access_key = secret_key  # REPLACE ME
endpoint = s3.us-central-1.wasabisys.com # REPLACE ME
acl = private

[storj]
type = storj
access_grant = access_grant # REPLACE ME

Migrate data

Once the configuration is done, you can use the rclone sync command to migrate data from Wasabi to Storj. The syntax is as follows:

Be mindful of potential network costs due to high egress traffic when running rclone for large data migrations on your machine.

Replace my-wasabi-bucket with the name of your specific bucket. This command will sync the contents from your Wasabi bucket to your Storj bucket, effectively migrating the data.

rclone sync --progress wasabi:my-wasabi-bucket storj:my-wasabi-bucket
rclone sync --progress wasabi:my-wasabi-bucket storj:my-wasabi-bucket

Post-Migration Steps

After migration, validate the data integrity in your Storj bucket by running rclone check

rclone check wasabi:my-wasabi-bucket storj:my-wasabi-bucket
rclone check wasabi:my-wasabi-bucket storj:my-wasabi-bucket

This command will compare the source (Wasabi) and destination (Storj) and report any discrepancies.

You can also see the contents of your Wasabi bucket in the Storj Web Console.

Previous
Migrate from Backblaze to Storj
Next
Create buckets