Add S3 storage to your VPS or Dedicated server

Help us grow :) Please share our articles for reach.

With more and more companies and even private individuals getting their own dedicated servers these days, it is a good idea to look at storage, and how to expand it on the cheap. Dedicated servers come with expandable storage options in most cases, but extra disk space can get pricey quite quickly. VPS servers come with even less storage options.

There are other options however – S3 / cloud storage. With ‘S3’ we refer to Amazon’s Simple Storage Solution or S3. This is object-level storage, which is differs from they way (for example) a hard-drive stores data. So although this is an option, keep in mind that that you will be using a 3rd party for storage and not a drive(s) attached to your server.

Now that we have that explanation out of the way, let’s look at how object storage works. The common name for S3 storage is a bucket. All your files will go in the bucket, and the bucket will be connected to your server as a mount. The best use for this would be to move backups to the cloud, but it can also be used for data storage.

Using S3 as a Filesystem

The way to use S3 as storage on your dedicated server is with the help of a utility called s3fs-fuse. FUSE stands for Filesystem in Userspace, and it creates a mounted virtual filesystem. The s3fs utility interfaces with the S3 protocol. It supports a large subset of POSIX, including reading, writing, creating directories, and setting file metadata.

One of the nice benefits of using S3 over traditional storage is that there’s virtually no limit on storage size, or what you can store. Where it can get pricey though, is with egress / api calls. But that is a problem we will completely circumvent by using the right vendor for our storage needs.

If you need access to large (and cheap) storage for your applications, then S3 storage makes sense and it is a nice bridge between the two storage systems. However, note that while it works very well, it is not going to be on the same performance level as storing and retrieving files stored on the server itself.

S3: who to use

There’s no doubt that the mighty Amazon.com is the industry leader when it comes to S3 storage, since they literally wrote the book on the topic (and the protocol, for that matter). However, there are other players in the market as well. We prefer Wasabi.com due to their simple and transparent billing method. With Wasabi, you pay only for your storage, and not to access the data (called egress & api fees).

Regardless of which provider you decide to go with, you need to create a bucket for your storage, as well as a IAM user, and give it permission to access the bucket you wish to mount. At the end, you’ll get a secret access key, which you will need. You will also get your network endpoint, which is the URL for your bucket. This is important.

For the sake of this article, let’s say our bucket is called sleuth-srv1s3 and the endpoint is s3.us-east-2.wasabisys.com

Installing Fuse

CentOS users:

$ yum install install epel-release
$ yum install s3fs-fuse

Ubuntu Users:

$ sudo apt install s3fs

Now that you have s3fs-fuse installed, we can connect our new bucket to our server. s3fs-fuse allows the use of a custom password file, so we can go ahead and create one now:

$ touch /etc/passwd-s3f

$ chmod 0600 /etc/passwd-s3f

We CHMOD the file permisson to 0600 to prevent getting any connection / authentication errors due to the password file having wrong permissions. Now is where we will use our IAM credentials for our bucket. When creating the bucket and IAM, you will have gotten a access key, and secret key. You can add that to your password file as follows:

$ echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > /etc/passwd-s3f

You should be able to mount the bucket now. As this is linux, the best place to mount anything is in the /mnt folder. First – create the folder under /mnt where you will mount the bucket to. Then we will mount our S3 bucket:

$ mkdir /mnt/sleuth-srv1s3

$ s3fs sleuth-srv1s3 /mnt/sleuth-srv1s3 -o passwd_file=/etc/passwd-s3fs -o allow_other -o url=https://s3.us-east-2.wasabisys.com

Of course, you can name your /mnt mount whatever is easiest for you, but the bucket name needs to be exact as per what you created with your storage provider, else it will not work. 

You should now be able to copy files to the /mnt/sleuth-srv1se folder (or your equivalent of it).

Should you require this mount to be available on boot, you will need to add it to your /etc/fstab file. There’s more than one way to do this, but we suggest opening the file in vi and adding this line at the end:

$ s3fs#bucket-name /mnt/bucket-name fuse _netdev,allow_other,passwd_file=/etc/passwd-s3fs,url=https://s3.us-east-2.wasabisys.com/ 0 0

This is all you need to add bulk S3 storage to your VPS Server or Dedicated Server.