Why exacly JSON?

[Jahanni]

Hey,

why exactly JSON?

I’m wondering why you chose the JSON over XML. You are already making the first workarounds and wrappers for the picojson code because you are dissatisfied by the interface and need to check the validity of the data every time by hand.

Well, these are exactly the strengths of XML. Despite of that, several awesome extensions could make the dev’s life much easier, XPath being one of them, basically reducing tree traversal algorithms to very few lines. XSLT is another one, a nice language to transform the data files directly to documentation.

I believe that, in general, JSON should be used where low overhead (i.e. network communication), JavaScript or Python is required, XML for everything else. I’ve been reading the code for a few weeks now and there might be large potentials for code size reduction, esp. in error handling for not well-formed JSON documents and missing field descriptors. JSON, being quite low-fat, is nice but it lacks the extensibility that could make game updates much easier. As the map and savegame formats are currently a bunch of numbers, I might create a branch, trying out the XML idea and see for myself why you didn’t want XML.

Jahanni

Oh, btw, no offence intended. I’m honestly interested in pro-JSON arguments.

EDIT: …and you can provide default values for all elements that are automatically inserted during the parsing. I just saw that someone pointed it out on another thread.

[Soron]

Although I don’t think anyone is entirely happy with JSON (it’s not easy enough for modders to write, and isn’t necessarily as robust as other options), my gut feeling is that XML isn’t the right fit for data files with the current infrastructure that we have.

My hesitation stems mainly from the fact that XML, while potentially more robust, is also more intimidating to modders with a low amount of technical expertise. I’d prefer to reduce barriers to entry, and make it easier for people to mod the game with just a text editor.

As for why JSON in particular, I’ve heard from Kevin that the reason is because using picojson meant we could avoid adding extra libraries. Specifically, it’s still possible to run the game with just ncurses. I’ve heard people mention they’d prefer something like YAML, but I’ve also heard Kevin say that he couldn’t find any low-overhead YAML readers for us to use.

Now, save files are potentially different than data files. In particular, we don’t need semi-technical users to edit them. I’d certainly like to have more robust save files (rather than the current ones, which are little more than a raw data dump). If you can get something that’s a significant step towards forward-compatible saves (i.e., something where adding a new data member doesn’t lead to segfaults at game load), and something that we can use, then I’ll personally work to get that merge-worthy and incorporated.

[GlyphGryph]

XML is just the most terrible, and I’m incredibly glad we didn’t use it. It’s just tedious. I wasn’t happy with the choice of JSON (though I agreed with it, for the same reasons it was originally proposed) but XML definitely would have been worse. YAML would have actually been appropriate, and would have worked quite well, and is actually still something we might switch to. (YAML can parse JSON, so it would be pretty trivial)

But yeah, the problem is that all the libraries for handling this stuff that we looked at are terrible. Like, really huge and bloated. It’s incredibly sad.

[Jahanni]

My hesitation stems mainly from the fact that XML, while potentially more robust, is also more intimidating to modders with a low amount of technical expertise. I’d prefer to reduce barriers to entry, and make it easier for people to mod the game with just a text editor.

Agreed. I never thought that people would edit the JSON files directly. The textual format is easy to understand; as easy as to edit it with JSON editors as well as regular text editors. Apart from that, there are quite a lot of nice Eclipse extensions (for Code::Blocks probably as well) and standalone applications that check the XML file against the attached schema while typing. Altough I feel that the syntax is much more intimidating than the JSON one, the schema verification might make up for this.

Now, save files are potentially different than data files. In particular, we don’t need semi-technical users to edit them. I’d certainly like to have more robust save files (rather than the current ones, which are little more than a raw data dump). If you can get something that’s a significant step towards forward-compatible saves (i.e., something where adding a new data member doesn’t lead to segfaults at game load), and something that we can use, then I’ll personally work to get that merge-worthy and incorporated.

XML is just the most terrible, and I’m incredibly glad we didn’t use it. It’s just tedious.

Interesting. It definetly is for hand-editing and introduces more running time overhead during load/store but I find actually using it from C or C++ couldn’t be much easier. Well, that’s just like… my opinion

I haven’t yet taken an in-depth look into the savegame code but I’ll try to write something up soon. I’ll let you know then.

[kevin.granade]

The primary goal of the files in data/raw are to expose those to modders for easy use.
Modders in general aren’t going to have eclipse, code::blocks, or any other specialized tool, they’re going to by typing things up in notepad, nano, geany, or god forbid* in forum posts.
XML is absolutely terrible for hand-written files.
JSON isn’t awesome either, but it’s manageable, and it wins based on the dependency issue (it’s embedded in our source, so 0 external dependency)
It’s REALLY hard to quantify, but I’ve heard from several people that if Cataclysm didn’t have such light dependencies, they would be working on something else, I’m not totally sure, but I might even fall into this category.
As GlyphGryph says, we’d really like to use YAML instead, but all the C++ YAML libries I’ve found have been similar in size to cataclysm itself, with far more intricate build systems. I haven’t looked at any XML libraries, but I suspect they’re similarly large, especially anything with XPath or similar support.
There might be some kind of simple config file format that would have a simple parser we could inline, but I think our needs for structured data are too great for that kind of thing.
A dishonorable mention is the MISTAKE lots of projects make, aka writing your own parser/format, this is frankly a terrible idea, and I’m greatly heartened by the fact that no one has suggested it.

For the save files, the most sensible thing to migrate to IMO is actually either JSON or ObjectBlobs (http://www.celsiusgs.com/blog/?p=477), since it’s emitted/consumed by the same program, schema stuff doesn’t seem to make that much sense to me, so simple serialization may be better. (again with the dependency aversion)

*I’m not criticising you if you’re doing it this way, I just find it an incredibly frustrating way to do things, and if you persevere through it and write up some content anyway, you have my thanks.

[Soron]

Agreed. I never thought that people would edit the JSON files directly. The textual format is easy to understand; as easy as to edit it with JSON editors as well as regular text editors.

Want me to point to any of several threads where nascent modders are clearly editing it in a text editor, due to stuff like missing/extra commas ;)?

Apart from that, there are quite a lot of nice Eclipse extensions (for Code::Blocks probably as well)

Hold it right there - we have enough trouble getting new coders to use Code::Blocks well, and prior to 0.5 we had a couple people who wanted to try editing the data files, but didn’t know how to compile the game. Our goal with the data files is to open up some forms of modding to non-coders; anything where you say “it’s not hard, if you install an IDE plugin” runs counter to that goal.

No offence intended, btw; I just wanna make it clear what our expected use-case is.

It’s REALLY hard to quantify, but I’ve heard from several people that if Cataclysm didn’t have such light dependencies, they would be working on something else, I’m not totally sure, but I might even fall into this category.

For me, it’s the build procedure, rather than the dependencies (which are indirectly a part of building, true). Currently, our Ubuntu/Debian build complexity is roughly: download source, type copy-pasteable apt-get line, type “make”, run game. Having something that simple is really really nice.

[KA101]

I’m just gonna pipe up and say that my mod work for Liberal Crime Squad started in XML, and the chief problems (found/fixed…so far) were due to missing a tag and subsequently not catching a default in the C++.

Hadn’t heard of json before you folks picked it.

Frankly my biggest barrier to mod-entry is compilers. I couldn’t get Pascal!Gearhead2 to compile with any modded code whatsoever and am not sure what sort of hardware code:blocks needs. Will my several-years-old Dell vostro 1400 handle it?

-KA101, who plays open-source because it’s cheaper, more secure, and generally less hardware-demanding than commercial games

[Jahanni]

I’m just feeling lucky that you aren’t that kind of devs who call out a flame war when their “religion” is criticized

Well okay, I got your points. I obviously totally missed the intended non-nerd-moddable audience. Altough I think that XML might still be a good solution for data files that are not to be tinkered with directly, mixing up several mark up languages is probably a bad idea. Since yesterday, I’ve been writing up an XML savegame thing and the code looks nice so far. However, as I’m writing in C++ (not C) and I’m used to memory managers (as in libboost), I usually don’t care about malloc/new’ing and free/delete’ing objects. The snippets will probably eat your RAM before you can say “Zed”.

So I’ll just forget the XML stuff and help you finding a good YAML parser

[Soron]

Have you SEEN how often we complain about the existing code :P? I mean, the game itself is great, but the implementation could use some work. I’ve seen isolated cases of defensiveness when decisions are questioned, but for the most part I think the sentiment is “this sucks, and that’s awesome. Can we turn the suck into awesome?”.

As for managing dynamic memory: yeah, if you DO use dynamic memory, it’s something that you need to be good about cleaning up when done. If you just need dynamic sizing of arrays, using STL components would probably suffice (and is kosher); but if you’ve got anything fancier planned, yeah, needs to be cleaned up properly.