(The use case mentioned on your last link there was for a specific circumstance for which Lua was absolute overkill, not necessarily that Lua itself was impractical for the game. More accurately, they literally use Lua produced by a code generation tool to call every single function in turn, rather than using Lua to write scripts or making use of Lua’s own ability to work with data. Using Lua as a batch engine is definitely the wrong way to use it.)
I missed your replies on the old thread on the SMF board, so apologies for that. Anyhoo, long story short, when I decided to come back to Cataclysm’s new builds last week, and then noticed a couple days ago that Lua support was still in, I was a little surprised and decided to check and see if Zhilkin had managed to convince you to keep it.
Guess not. =)
To answer your “essential” question in a perhaps glib way, Lua (or scripting in general) is essential for one thing and one thing only: modification of game logic as part of data-driven design (not to be confused with data-oriented design, which is basically C code working with master object tables). Here’s a Stack Exchange thread which has a bunch of great statements and linked articles on why data-driven design is valuable, including from one person who wrote an article literally titled “The Case Against Data-Driven Design” without irony, and yet who still wholeheartedly supports data-driven design. =)
As I mentioned last time in way too many words, Lua’s “fundamental problem” is not the ratio, but the chicken-and-egg problem; Lua support is rudimentary if existent at all, which prevents mods from using it, which prevents features from being requested or added, which makes Lua support rudimentary if existent at all. In other words, I do think that piling on more mods literally is the fix to the problem. Zhilkin’s example mods are sometimes a bit blithe or silly, but having tried to work with the scripting as it exists (just to allay any illusions that were conveyed from the last time, I actually don’t know Lua myself!), the existing Lua support is only useful for the most blithe and silly tasks, and yet attempts to add more generally useful functionality (more events, more map interactions, more exported data structures, etc.) are being ignored. That line count of procedurally-generated code is somewhat disingenuous: those lines are all dead code until/unless anyone uses them, which means no performance impact on anything other than the compilation process – which the data-driven approach attempts to sidestep in the first place. Then they’re no longer dead code; they’re live code, being called by actual rapidly-prototypable and rapidly-mutable Lua functionality, which people can externalise gradually and leave the game engine to focus more robustly on the data-handling structures. This cost/benefit ratio approaches zero; it’s not the externalisation in itself that’s the end goal, but the ability to modify the externalisations.
Lua does not increase productivity, especially not of core engine programmers. It increases mutability. If anything there will always be a negative impact on productivity for the game engine and perhaps on the people who are accustomed to doing gameplay programming directly in the engine. Mutability is the factor we’re trying to isolate: the ability to change the game quickly and rapidly to suit a wide variety of perceived goals and interests rather than being forced to reach consensus.
In other words, the idea behind scripting isn’t solely to externalise engine support in the root build, if the programmers are the ones doing the design – you already have direct access to everything you could possibly want. The externalisation of engine features is designed to give designers and especially end users the agency to create their own changes: plotlines, stories, balance, esoteric features, characters, cheats, etc. It’s tacitly useful to externalise the content in the root build so that it can be changed quickly by end users whenever they want. Having to recompile the entire project to change a single undesired feature is both difficult and unsustainable, given the whole iteration-versus-recursion thing I mentioned last time, but freedom is a big part of it.
In my case, I want to add realism content, story related content, and gameplay related content that is subjective enough that people wouldn’t want it in the whole; e.g., intense vehicle realism and physics; rebalanced map generation; Sims-like NPC personality variables, interactions, conversations, relationships, romances, reactions to your behaviour, etc.; negative morale accumulations when near brutality, like pulping zombies, standing in the midst of a room covered in blood and gore, suddenly spotting a bunch of human bodies, etc., inversely proportional to discipline and insanity; manufacturing equipment to mass produce parts like steel plates and re-establish industry (giving a function to every Macguffin machine that appears in factories); and so on.
These can be introduced into the core game, sure, but when subjective content is automatically distributed to everyone, negative reactions are common even if they have the option of turning it off (heck, I’m sure most people are already working up a list on the things I mentioned there). On the other hand, people tend not to be bothered by subjective content that they can locate and opt-in for instead. The maintainability is also a thing. Anything that has to be mainlined does have to be maintained, but outside the mainline only the original author is responsible for it and if it becomes obsolescent, then it’s the author’s responsibility – and, dependent on the permissions given anyone else can step in to maintain it if they want. Opening up external mods and content with scripted game logic allows good content to gravitate and bad content to deprecate, without any particular burden on the main build other than the (admittedly substantial) effort of keeping the engine portable enough to allow external scripts in the first place.
Because we don’t have functional game logic in Lua, I can’t provide an example, but playing a moddable major studio game should provide all of the necessary examples of what’s needed in a scripting language. Even then, people will be hurting for more scripting language functionality; the most useful scripting language is the most feature complete, not the one that’s pared down to the arbitrarily-defined features that fulfill the desired use cases. The purpose of scripting is to fulfill all of the use cases that the game doesn’t fulfill on its own, and therefore there will never, ever, ever be a version of Lua with minimal bindings that will be more useful than core engine code.
So, presuming it’s axed, you have only one of two choices.
The first choice is to require all scripted mods to merge their very specific subjective code into the cohesive whole, which acts as dead code unless a mod uses it. That conveys all the attendant penalties, such as veto, debate, bloat, and, as the article you linked mentions, “debt” from maintenance.
Working on the master is also inherently, although subtly, a form of subordination: instead of being permitted to write your own content that suits your own purposes, you must write content to suit someone else’s purposes – not just yours-as-Kevin’s specifically, but the community-at-large’s as well, which can introduce the “tyranny of the majority” concept we see in criminological contexts (the notion that sometimes it is most just to ignore everyone and do what’s morally right rather than succumbing to the mob). Some people like working with a team, still others like the game enough that they are willing to roll with it, some people reject it and would prefer to work alone (me, for example), and the occasional Others create the drama that this community sees from time to time (also me, I guess ;-)). (Another counterpoint is, do you feel like the Lua support is a mob that you are subordinate to, I guess.)
In addition, C++ is not simply something that can be set up easily. To come perilously close to ad hominem, but with a good intention, is your dislike of Lua and desire to remove it based on not wanting to learn it? I don’t compellingly want to learn it, either, but my operating system and expertise is far more oriented toward design and not oriented toward programming, and I imagine a fair few other people are of the same category. Working directly with C++ has a much higher bar to entry than writing game logic in Lua does. People on *NIX systems have the advantage that because the OS is designed by programmers for programmers, build environments are pre-installed and practically part of the kernel itself, and it’s as simple as a few apt-gets and lines in the console to do rapid prototyping on their own. On Windows, which is actually a substantial majority of any user base (also a chicken-and-egg problem, but that’s another rant), people who have limited programming experience would simply prefer to learn something “simpler” and work within an engine than to go through all the necessary steps of managed code to integrate into a single project and the whole. (A favourite statement of mine is, if I have to set up a development environment to work on someone else’s game, why wouldn’t I just work on my own game instead?)
The other option, which is really a non-option, is to require independent forks of the source code that will never be reintroduced into the whole. That still has all of the penalties for end-users of working with code.
The reason why Bethesda’s games are so popular and why most of the work over the last few years that should have been spent on my own game has instead been on modding theirs (in spite of their absolute horror show of a game engine, voice actors ranging from decent to atrocious, and abortions of plot) is because of the longevity provided by their modding community as well as the easy approachability of the modding process. (Valve/Bethesda like to make silly claims like “only 3% of users use the Workshop” – discounting the substantial number of people who use mods via other external communities, or few who write their own.)
Cataclysm’s engine is at least halfway decent (especially for a roguelike), but has a lot of subjective content in it that players are essentially forced to accept. The game changes substantially with experimental builds, usually for the better… but, for example, the case of the backlash against the corpse nymphs is another. Some can be modded out, but certain gameplay behaviours are literally stuck unless you have the programming talent and attitude necessary to jump into the main build, describe the importance of your feature, and then (if you’re one of the dozen or so active people who also directly contributes) either a) support both your alternative and the original with the source changes you make or b) convince the majority that they must adopt the changes if making it an option is either unfeasible or undesirable.
With externalised game logic, this occurs emergently rather than deliberately. This is all regardless of the language. If a single core engine is maintained with external scripting available, in any language – Python, Java, hell, even JavaScript as AngelScript – then designers are given the ability to write program code without, you know, writing program code. This attracts independent narratives and expansion over geologic time, rather than brightly-burning-bulbs that furiously add amazing features to a single room and then burn out.
In other words, Lua itself? Couldn’t care less about it. Scripting? Absolutely imperative, and the vast majority of scripted content that’s in the game does need to be exported from low-level code to high-level code so that people have the option of changing it and balancing it, rather than having to do everything by committee. That’s fundamental to modifiability and allowing interested people the ability to create and tweak to get the game experience they want. Cataclysm is excellent without it, but could be transcendent with it.
Since Lua’s the scripting language we’ve got, I’m defending it vigorously unless some other alternative is proposed – if we were going for anything, I’d actually prefer Python (which I also don’t know, at least not very well =)) since that also gives us the ability to run external OS-dependent DLLs/libs/shared-objects/what-have-yous, but Python is allegedly one of the worst things to link to anything ever and although its bytecode interpreter is still rather fast, its compilation and JIT is much slower than Lua’s.
You do bring up an interesting point: there is literally no advantage to using Lua over judicious use of engine-supported JSON, provided that the JSON enables all of the matching functionality. But since you’d need the same full writeup of intended bindings for either engine-supported JSON or just external Lua, I think Lua’s the way to go there.
The more popular the project gets and the more people contribute their own modifications to the world at large without having to fork the project or submit it for internal review, the higher the benefit that is realised for the same linear cost, as compared to hard code which is lower fixed cost, but also fixed benefit.
