Add MEGA S4 (and other few questions, please)

[davidsmith91]

Add Mega S4 please
Mega doesn't support metadata headers like for example the one used to act as mtime.
So I think it's a perfect use case for JuiceFS: it's cheap (the cheapest), it has good redundancy etc.
But I've read that it doesn't support certain metadata and/or s3 headers stuff, so I think using something like JuiceFS to separate metadata from the data would be useful

Does JuiceFS requires any headers or metadata functionality to work with S3 object storage? Because mega is s3 compatible but has some missing things for example the mtime doesn't get saved (X-Amz-Meta-Mtime this one)

And I think maybe it has also other headers that are not supported (sorry I'm not much knowledgeable about s3 storage)
But the data can be seen/used using something like rclone configured as a s3 compatible remote.. but as I said it doesn't support X-Amz-meta-Mtime so I cannot use rclone (I need to keep the mtime).

Also, I was searching for something that could handle file locks (avoid race conditions), and in general save all the metadata, be faster (rclone metadata cache is not good because it isn't persistent but stays only in RAM, and if I scroll Finder gets stuck because it starts loading all the files etc, also in general I need something fast because I basically want to have Mega S4 acting as a "storage expansion" to my laptops, server and Android smartphones. But so that it can be usable, like without losing edits if two people are editing the same file or things like that.

I just want to be able to trust the storage to act as a real filesystem which I'm used to, without "not knowing" that I will lose this or that property, and without being painfully slow or having race conditions and conflicts etc

And I also need client-side encryption so that I know I will be the only one to see the data

I want it to act as a filesystem , and from what I understood, JuiceFS serves this purpose, right?
Can JuiceFS be a solution for me?

Is Mega S4 already usable?
Also, is there anyway to use JuiceFS on android?
Thanks

[davidsmith91]

Here is the specs of Mega S4.. as you can see it doesn't support a LOT of s3 headers and metadata: https://github.com/meganz/s4-specs

Would it work with juicefs anyway?

[solracsf]

Are you able to test a juicefs objbench command on it?

[zhijian-pro]

Juicefs mount does not depend on s3 header and metadata.
If the objbench test passes, then it can be used.

[davidsmith91]

[davidsmith91]

@zhijian-pro @solracsf It's really really slow. I'm using Mega S4 european server, but I'm in South East Asia.

I'll try Backblaze B2.

Do you think that the data server must be close? I was looking into JuiceFS for this reason. Because I have the data server in the west, so I experienced too much slowness with rclone etc, and after searching for a solution, I learned the concept of separating metadata from the data (also because Mega S4 didn't support metadata properly as a backend), and then I ended up trying JuiceFS.

I tried use redis server locally in docker on my laptop, and I get 15 files for a total of 130mb uploaded in 1m40s, that's too long compared to the upload speed I have (around 80Megabit/s). I also tried to use redis server on an european VPS, but it was unusable, Finder (MacOS File Explorer) would get stuck etc and it was in general really slow.

Do you think it's the distance from Europe, or just Mega being slow?

I tried the command below with few variations and I also tried using --writeback etc but I felt it was too much of "an illusion" lol, I want to know when files are really finished copying etc.

juicefs mount
--storage s3
--cache-size 1024
--buffer-size 1024
--max-uploads 30
--prefetch 2
"$REDIS_URL" "$MOUNTDIR"

I have around 4 terabytes of data that I need to "backup" in the cloud and then I will use certain part of those data (for example some project dir) to work, or maybe a folder where I save videos or photos from the smartphone (using the S3 Gateway runned on my laptop in local wifi, for example).

I don't have a big monthly budget so I was looking into cheap S3 providers. Mega S4 is the cheapest. Then I think Backblaze B2 is the cheapest after Mega. Things like AWS, Google, Azure, are too expensive, I think (I was just scared by looking at them)

[solracsf]

Using --writeback, the files are first stored locally and then uploaded in the background. This should be used in your case IMO.

Second, links between Asia and Europe will always be "slow", some slower than others, it will depend of the routes between providers.

You should seek for a local provider if possible. Try iDrive-e2 in SGP: https://blog.idrive.com/2023/05/11/idrive-e2-offering-hot-s3-compatible-object-storage-with-new-storage-region-in-singapore/

[davidsmith91]

Using --writeback, the files are first stored locally and then uploaded in the background. This should be used in your case IMO.

Second, links between Asia and Europe will always be "slow", some slower than others, it will depend of the routes between providers.

You should seek for a local provider if possible. Try iDrive-e2 in SGP: https://blog.idrive.com/2023/05/11/idrive-e2-offering-hot-s3-compatible-object-storage-with-new-storage-region-in-singapore/

But --writeback makes it seem that the data are instantly copied on destination, at least on MacOS using Finder. So that can make me think a file is there when it's still not uploaded, and if I lose the internet connection and I deleted the local copy of the file thinking it was uploaded, it will be lost, right? I can pay attention to that, but I can see my girlfriend losing files with a setup like that?
Also, I like the idea that the files or "pieces" are uploaded together so there aren't too many connections to europe (because the distance brings latency). So like instead of doing 15 uploads of 10 mb, is there a way to make it do only one upload of 150mb? So that after the initial connection, the upload itself would just be limited by the bandwidth, right?

I will look into iDrive in Singapore, but what if I then go to south america? Lol. Or back to europe? Etc.
I think maybe I have to differentiate between "old data" or "backup data" that I don't search/use often, and data that I use daily (so that I read or update daily). Maybe?

And maybe have, Idk, the old data on Mega in Europe or whatever, and maybe the newer data on a cloud server closer to me.
I change country every 1-3 months. I could be now, then I could go to other places, with thousands of kilometers of distance anyway.

Maybe the solution is just to have an hard disk with me and copy that every X time on the cloud (as a backup)?
I would have liked to just have an "infinite seamless cloud storage" that integrates with my devices seamlessly and that I use daily as if it was a local filesystem, so I know that even if I lose my devices or if I need my gf to see the same files, I just have a centralised point of storage with no race conditions (because JuiceFS implements locking of the files etc).

Am I just dreaming about something that is not possible?

Basically the question is: how would you use ONLY cloud storage as your storage, as a digital nomad, changing country every 1-3 months? No hard disks, no anything. Just every data as soon as possible saved to the cloud + a cache, or just directly using the cloud mounted and almost never using local space.

I want to remove the manual or automatic "backup" thing from the equation, and just have my data always on the cloud, and then when for example I have to use "folder X", it will be cached on my pc so that it is not slow.

I thought JuiceFS could do this? It seems to be able to do that?

I thought: if I separe metadata from data, I shouldn't have the issue of searching/looking for files anymore. This is the main thing. I don't need to download lots of data every hour/every day. I just need to look for specific files and data between the 4 Terabytes of data that I have. And having the metadata locally, for example, or on a VPS near me, enables me to do that.
Then, the only other thing I should have, in theory I think, is a cache mechanism.

@solracsf @zhijian-pro Why is the latency of the data server an issue? Shouldn't the latency just be felt when establishing the connection, but then the bandwidth should be the bottleneck? Instead here, it looks like Mega S4 is being slower than my bandwidth. I think maybe because every chunk is uploaded? Or there are more operations than just "uploading"? So the latency is felt more. Is there anyway to prevent this?

Shouldn't the data server be able to be used without caring about the latency because the metadata can be stored separately in a place with low latency?

[solracsf]

Giving your setup, just use a commercial "Drive", like the one Mega offers.

[davidsmith91]

Giving your setup, just use a commercial "Drive", like the one Mega offers.

I already tried.
The one with reasonable price like Mega Drive, have issues.
For example Mega Drive is painfully slow because it has its own encryption. I think it's only usable when using their own apps.
They use a client-side encryption that was proven faulty. I cannot use a different encryption.

When having lots of files of few terabytes like I have, it's painfully slow.
It cannot be used well with rclone because they don't have standard ways to be accessed because of their client-side not standard encryption.

I already tried mounting through megacmd , and also rclone (that uses not official libraries that causes other issues).

Also, in both cases, they don't have things like file locking etc.

Mega is the cheapest cloud both for Mega Drive and Mega S4.
I also think that the Drives are slower than S3 storages because the latter is usually more professionally used? For example, Mega S4 seems to be quicker than Mega Drive. But maybe it's just my impression.

The issue with Mega Drive was it being not compatible with many things, and the fact that the metadata are on the same place of the data.

Metadata separation should be the solution to latency, if I'm not wrong. Should be the solution to searching, indexing, browsing the files without having to wait loading etc. That's the main issue, so that's why I switched to juicefs

I tried also pCloud, Filen, they all seem to have similar issue. They start to be too slow when having multiple files, or they use their own encryption that is either not open source or not good or uncompatible, the apps maybe are not open source so you have to trust their word, or they don't work good with rclone etc, or they don't have file locking etc.

And JuiceFS cannot be used without S3. JuiceFS would enable me to use cloud as a filesystem, giving me the possibility to expect a normal filesystem behavior, not the weird behaviour of things like rclone mount etc. Rclone founder says himself: Rclone is not a sync client.

JuiceFS instead, it seems that it solves the issue of cloud not having the same behavior that one would expect from a local storage with just having it's local filesystem.

Basically, I would like to see cloud storage as using a slow hard disk: just slower in trasferring data, but with the same functioning and behavior of a local harddisk. Having metadata separated from the data, I thought I could expect that.

Like: given that I'm ok on having a bandwidth that transfer 10 Megabytes (not Megabits) per second, can I have a cloud storage that can be used on my pc as if it was an hard disk connected to my pc + that can be accessed also by other PCs or devices (as if having an hard drive in your home network).

I only have one geographic area from which they will be used, and it's always where I am, so I can even just keep the Redis database local with absolute zero latency. But I need a client that won't need low latency for the data. Just for the metadata. And that so, it will only have 200-300ms latency when STARTING to upload or download something, but then will use maximum bandwidth available.

So for example, if I have a 100 Megabit connection, it's 12.5 Megabyte/second. So I would expect a 1250 Megabytes file to be uploaded in 100 seconds + 200-300 ms latency to establish connection + 0ms latency for the metadata that I have on localhost Redis database. And if I have 10 files of 125 megabytes, I would expect 100 seconds + 200-300ms latency multiplied x 10 (each file), + 0ms latency for local metadata.

Is this possible with JuiceFS, or is the latency acting in other ways that I don't know?

@zhijian-pro

[zhijian-pro]

In summary, metadata and object storage are the two main factors that slow down the speed. It is expected that these components be deployed in the nearest area to you to achieve a better experience. If the metadata latency is very high, the experience will be very poor, with almost all latency increasing. If the object storage latency is high, then reading and writing will be slower.

If you resolve the metadata latency issue, then the bandwidth of the object storage is the main performance bottleneck. You can make a plan as you wish, and use juicefs stats to analyze where the bottleneck is after mounting.

Additionally, you might try our cloud service products, which may solve your problem. Cloud services have the features of data cross-region replication and mirrored file system.

[davidsmith91]

Ok i will look into your cloud services. And what is the best way to use juicefs on android?