Rucio
Overview
Rucio is an open-source software framework that provides scientific collaborations with the functionality to organize, manage, and access their data at scale. This tutorial walks through the steps needed to configure a Rucio Storage Element (RSE) with a demo of Rucio.
These configuration instructions were adapted from the draft instructions being developed in the Rucio GitHub project: Create documentation page for S3-type storage
Setup a demo environment
Prerequisites:
- Docker is installed and running: https://docs.docker.com/get-docker/
Already have a Rucio environment?
Skip ahead to Rucio
Clone Rucio and jump into the docker container
Note that if you get an error like No such container: dev_rucio_1
run docker ps
to identify the name of your rucio container (it may be something like "dev-rucio-1)
The rest of the commands are to be executed within the Docker container
Initialize the database
Update the ca-certificates package
The image has python3 linked in /usr/bin/python
but yum requires python2
One workaround is to change the first line of /usr/bin/yum
and /usr/libexec/urlgrabber-ext-down
to use /usr/bin/python2
.
vi /usr/bin/yum
< now change "python" to "python2" in the first line
sdf
vi /usr/libexec/urlgrabber-ext-down
< now change "python" to "python2" in the first line
Configure the Storj RSE
The gfal protocol was chosen because s3boto was recently removed and the draft rucio doc (linked here) suggested using it to support s3. gfal is used to send requests like get, put, delete, rename, and copy to the RSE.
Create rse-accounts.cfg
Upload and download a file
What Next?
Explore the Rucio Documentation
https://rucio.cern.ch/documentation/
Set up a Rucio Client
Rucio CLI
Rucio WebUI
Getting additional Rucio Support
If you are looking for additional support, Rucio has a mailing list rucio-users@googlegroups.com as well as a slack support channel.