Saves. Concept of new savefiles system

Game Talk The Drawing Board
1

Just one look in save files was enough to find a lot of disadvantages in current saving system.

Main problem: savefiles’ size and speed of that system.

Even my current world is 14.5 MB, while it could be MAXIMUM ~40-60 KB

0. Just a question. Why the hell you write “16”, but not ► (0x10)? There’re upto 256 types of tiles, so, you can use one byte instead UPTO THREE BYTES.

[spoiler=1: Weak speed up and huge compression by similarity of tiles]1. You allow to appear same numbers a lot of times. You can write “16 16 16 16 16 16 16 16” as ◘► and use 2 bytes instead 23. One field of 12x12:

0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
may take from (12+10)*12 = 244 bytes to

255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
(36+10)*12=552 bytes.

Let’s see how many bytes will it take if we shorten this shit from: ♀♀(0x00) to ♀♀(0xFF) - from 3 GODDAMN BYTES TO 3 GODDAMN BYTES. Or even lesser - since shortening may be upto 12, their maximum value is 12 12, and it’s 12^2 = 144 different combinations, so, we can even shorten those ♀ IN ONE BYTE: Р(0x90).
That’s: 220 / 3 to 552 / 3 = 73 to 184 less times!
In contrary, that system will require to build special “matrix of map”, where you should show which bytes means shortening, which one - true data. Or, just to reserve 1 ID to show next value is shortening.
So… if we take our field of “16” (►) and shorten it by applying everything written above, we get: “(0xFF)Р►” Any explanation needed? (0xFF) means next symbol is shortening. Р (0x90) is ♀♀ (0xB), so, it means that symbol ► (0x10) is repeated 12 times and first ♀ means that line ♀► is repeated 12 times, but by Y-axis.
[/spoiler]

I don’t think we need to split every region to submaps, and every submap to 4 map-tiles. Just no comments, i described that already. Check #5 comment.

Todo.

2

If you’re concerned about disk space usage, I could resurrect one of the branches that had compression enabled, they were abandoned because they didn’t make save/load any faster, which was the primary motivation for them. If save/load speed is your concern, then I can focus on finishing the streaming save/load support, so that you never have to read/write more than a few hundred KB of map data at a time.
Carefully encoding map data as the smallest integer value possible is a dead end, compression is much better at realizing space savings, and it doesn’t force the map data to be fragile with respect to future changes like integer encoding does.

3

I smell floppy drives. But yeah, it wasn’t any faster earlier.

4

Also, even if it is a very good point, and even if the people who made it otherwise were blubbering idiots (they are not), please use a respectful tone. Your arguments will be more effective and better received without an aggressive tone and swears.

5

Course doing something like that would also remove any sort of map editing through save files as well.

6
Course doing something like that would also remove any sort of map editing through save files as well.
I don't that someone really edits maps. Even if someone need to, he can use internal editor or any HEX-editor.

Damn. It’s so long to write, i am too lazy to describe how to change datafields…

If you're concerned about disk space usage, I could resurrect one of the branches that had compression enabled, they were abandoned because they didn't make save/load any faster, which was the primary motivation for them. If save/load speed is your concern, then I can focus on finishing the streaming save/load support, so that you never have to read/write more than a few hundred KB of map data at a time.
It takes so long because it opens or creates savefiles in huge amounts. Opening ~3000 files is pretty slow operation. I just made one little program to see how much time it takes to open and write a field of 12x12 of "16" 4 times, delimited with "----". 2 seconds (1.993, to be accurate). 3025 files took 2 seconds to be written and now they all are 5.16 mb. Of course i didn't do anything about datafields. Those datafields are one thing that very slow and takes a lot of memory, time and nerves. [img]http://clip2net.com/clip/m192794/1396488287-clip-30kb.jpg[/img]
Also, even if it is a very good point, and even if the people who made it otherwise were blubbering idiots (they are not), please use a respectful tone. Your arguments will be more effective and better received without an aggressive tone and swears.
Nah. Sorry. I've got used to use this tone. I can easily say why it was implemented so ineffectivily - because it's easier to write.
7

Get used to people thinking you’re obnoxious, then.

8

I already have, lol.

9

After 3-years in game, my save folder is 30,889 files across 173 folders, accounting for 194 MB.

Seems like maybe it could be steamlined but I dunno if that should be a top priority.

10

[quote=“EditorRUS, post:6, topic:5694”]

Course doing something like that would also remove any sort of map editing through save files as well.

I don’t that someone really edits maps. Even if someone need to, he can use internal editor or any HEX-editor.[/quote]
Yea, it comes up from time to time, and it’s really helpful when you need it.

The solution as I said is never to read/write that many files at once, even right now if you periodically save/exit/load you’ll never have to read or write many save files, and if I finish out the streaming writeout feature, and jsonize map saves, you’ll never have to read or write more than about 100 at a time. The only time you have to read/write every map file at once right now is if the values used for the map features is changed, which requires rewriting all of them, which is precisely a misfeature of encoding your save as integers.

[quote=“juliawang87, post:9, topic:5694”]After 3-years in game, my save folder is 30,889 files across 173 folders, accounting for 194 MB.

Seems like maybe it could be steamlined but I dunno if that should be a top priority.[/quote]
And as I also said, if this is a real problem we can compress those save files down to like 1/20th of their size or so. It hasn’t been a priority since I’ve only heard about 5 people complain about it once I chunked them up last release. (before that the way the game handled files it kept everything in memory, which meant that you eventually crashed the game from exploring too far on low memory systems.)

11

Just to see any difference between writing 30 KB and 14.5 MB:

It’s not big, but have value.

12

I’d say most along the lines of “get used to being ignored, then, even when you might have a point”. Of course, in this case it doesn’t seem to matter, since he’s also ignoring responses (except where they fit his biases, of course). Oh, well.

For reference, I just compressed my save directory from 41 MB to about 2.6 using nothing but gzip, which strongly suggests that just a little (fast!) compression can go a long way. But, honestly, as kevingranade has said, it’s literally not worthwhile to switch from a mostly-functional straight-forward format to a binary mess. Especially not one created by some ad-hoc run-length encoding scheme that would be poorly documented and poorly understood. There is much more value in clarity than there is in saving a few bytes. See also premature optimization.

13

How fast is gzip? Would compressing slow the read/write significantly?

14

Shouldn’t slow it perceptibly; gzip is stupidly fast, by compression algorithm standards. That said, given the data involved, even a relatively slow algorithm would still be unnoticeable, at a guess.

15

I tested compressing the saves about 8 months ago, and there was no perceptibe speed difference either positive or negative.