Save compressing?

Hirmuolio · 2018-03-27T21:48:42.463Z

The world folder gets pretty fast after traveling for a while.

But the save files also compress extremely well. My 150 MB save folder compresses into 9 MB archive (7z fastest).

So compressing the save folder/files would drastically reduce the size of the save.

Just throwing this idea here to be told how hard it would be to do.

darktoes · 2018-03-27T21:54:30.885Z

That would probably cause issues with saves you’re trying to play, but I’m definitely gonna go compress my old saves. Thanks for the info.

kevin.granade · 2018-03-27T23:59:55.253Z

It’s pretty trivial to do, though it limits compression ratio if each file is compressed individually.
The kind of annoying thing is it requires taking a dependency on a compression library.

darktoes · 2018-03-28T00:03:25.682Z

Wouldn’t it mess with load/calculation times if it has everything it’s trying to read compressed?

Hirmuolio · 2018-03-28T11:31:55.080Z

Even if the files are compressed individually the save game would become much smaller. The majority of the save size comes from the tens of thousands of small ~7000 B map files. These compress into ~600 B each.

Compressing/decompressing shouldn’t have very big impact on load times. I think the game only loads the parts that are near you so only some of the files would need to be opened and saved at once. Compressing one map block folder is so fast that you don’t even see the 7z to open before it already closes. It is even possible that on HDD the loading and saving could be faster as there is lesss to write and save to disk.

Coolthulhu · 2018-03-29T01:41:05.062Z

though it limits compression ratio if each file is compressed individually

zstd https://github.com/facebook/zstd can use a common pre-learned dictionary for many files, which would handle our (very similar) jsons well.
Though I’m not sure about how legal would it be, since it the implementation is dual licensed under GPL2 and BSD.

kevin.granade · 2018-03-29T04:50:41.774Z

This header-only library would make adding gzip compression to dda pretty trivial: https://github.com/mateidavid/zstr
Just adjust functions that open the save files with the wrappers from this library and you’re done.

I’m not a fan of adding a dependency on something relatively obscure like zstd, but if it delivers on that high compression ratio with tiny files it’ll be worth it.