Two mul or not two mul: how I found a 20% improvement in ed25519 in golang
Low level optimizations are tricky and sometimes unintuitive. We'll take a look at a story of optimizing ed25519 signing and verification.
Egon Elbre
Storj Engineering Blog
Learn about the latest developments in the Storj network and the technology that powers it.
Go Integration Tests with Postgres
When writing server side projects in Go, at some point you will also need to test against a database. Let's take a look at different ways of using Postgres with different performance characteristics. The final approach shows how you can set up a clean database in 20ms (there are a few caveats).
Egon Elbre
Finding and Tracking Resource Leaks in Go
Forgetting to close a file, a connection, or some other resource is a rather common issue in Go. Usually you can spot them with good code review practices, but what if you wanted to automate it and you don't have a suitable linter at hand?
Egon Elbre
Production Ready Go Concurrency
Concurrency is one of those things that's easy to get wrong, even with Go concurrency features. Let's review things you should consider while writing a concurrency production code.
Egon Elbre
Lensm, A Tool for Viewing Disassembly
I couldn’t find a great tool for viewing disassembly, so I wrote it myself over the weekend.
Egon Elbre
Storj Open Development -Part 2
In October 2021, Storj announced we were going to adopt an open development strategy for the storage node development efforts. The goal was to enable our community—and the wider open source community—to contribute to the development of the network. We started this effort by moving all node issues...
Brandon Iglesias
Finding Goroutine Leaks in Tests
A leaked goroutine at the end of a test can indicate several problems. Let's first, take a look at the most common ones before tackling an approach to finding them.Problem: DeadlockFirst, we can have a goroutine that is blocked. As an example:func LeakySumSquares(c...
Egon Elbre
Use Storj DCS from Cloud-native Environments Using the Sidecar Pattern
Data stored in Storj Decentralized Cloud Storage can be accessed in multiple ways:With native “uplink” protocol, which connects directly to the nodes where the data is storedWith using S3 compatible REST API, using an S3 gateway:Either the hosted S3 gateway, operated by Storj LabsOr with running ...
Marton Elek
Demystifying Technical Debt
Technical debt has been bothering me for a while. It looks like a scary monster in the closet. It seems somehow a catchall for different design mistakes, code worsening over time, legacy codebases, and intentional design mistakes due to time constraints.
Egon Elbre
Storj Open Development Announcement
Storj Open Development Announcement
Clement Sam
Using Storj DCS with GitHub Actions
GitHub Actions is their system to automate, customize, and execute software development workflows in the GitHub repository. This article will inform you how to upload files to a Storj DCS bucket from a GitHub Actions workflow.The Storj DCS Public Network Stats is one of the projects at Storj wher...
Kaloyan Raev
Open Source and Open Data: Storj DCS Network Statistics
We recently began publicly exposing more data about the network in a way that could be used on-demand and programmatically. If you missed it, we have started publishing what we think is the most important network statistics on our new Storj DCS Public Network Statistics page. Now, if you’re a non-technical person, this may not be what you expected. Here’s an explanation of why we took this approach.
Brandon Iglesias
A Tale of Two Copies
It was the best of times, it was the worst of times. That's when I hit a performance mystery that sent me down a multi-day rabbit hole of adventure. I was writing some code to take some entries, append them into a fixed size in-memory buffer, and then flush that buffer to disk when it was full.
Jeff Wendling
Automatically Store Your Tesla Sentry Mode and Dashcam Videos on the Decentralized Cloud
You can automatically transfer Sentry Mode and Dashcam video clips over WiFi to cloud storage and make room for more videos the next day. We used a Raspberry Pi (a small, low cost, low power computer about the size of an Altoids tin) plugged into the USB port in the dashboard to store the video files. When the Tesla pulls into the garage at night, the Raspberry Pi connects via WiFi and uploads all the videos to Storj DCS cloud storage, then clears off the drive for use the next day. This will also work for videos recorded in Track Mode if you have one of the performance models, making it easy to share any of the videos with your friends.
Krista Spriggs
Introducing DRPC: Our Replacement for gRPC
In 2016, Google launched gRPC, which has overall taken the systems programming community by storm. gRPC stands for something with a G, Remote Procedure Call; it's a mechanism for easily defining interfaces between two different remote services. Building a new decentralized storage platform from the ground up in Go, obviously, we considered using gRPC to simplify our development process in peer-to-peer remote procedure calling. In fact, I'm not even sure we really considered anything else. Fast forward to the latter half of 2019, and we had 170k lines of Go, a beta network of over 4 PB, real live active users, and it turns out the gRPC bed we made for ourselves was not all roses. So we rewrote gRPC and migrated our live network. DRPC is an open-source, drop-in replacement that handles everything we needed from gRPC (and most likely, everything you need) in under 3000 lines of Go. It now powers our full network of tens of thousands of servers and countless clients.
JT Olio and Jeff Wending
Visualizing Decentralized Data Distribution with the Linksharing Object Map
At Storj Labs we're distributed system junkies. We enjoy building highly distributed, ridiculously resilient software. The Storj Network is currently spread across over 10,000 uncorrelated endpoints, and that number is growing fast.The global substrate of diverse, uncorrelated endpoints across wh...
Brandon Iglesias
Choosing Cockroach DB for Horizontal Scalability
Here at Storj Labs we just migrated our production databases from PostgreSQL to CockroachDB. We want to share why we did this and what our experience was. TL;DR Our experience has convinced us that CockroachDB is the best horizontally scalable database choice in 2020.
Krista Spriggs and Jessica Grebenschikov
Flexible File Sharing With Macaroons
Sharing is a vital piece of any online storage system. Or, to be more precise, access control is a vital piece of such systems. When you store a file, you need to be able to designate whether other people or automated agents are allowed to retrieve the data, delete it, or put something else in it...
Paul Cannon
Our 3-Step Interview Process for Engineering Candidates
In case you hadn't heard, Storj Labs is building a decentralized cloud object storage service. Why would we do such a challenging thing? At a basic level, it's because we believe the internet can be better than it currently is and we see how to improve it. We believe your data is worse off being ...
JT Olio