Update 39: January 2024
Hi all! We hope the new year is treating you right.
As mentioned in last month’s update, we’ve got some data and technical learnings from the November Stress Tests & December Playtest to pass along this month.
We spent the month of January on a fun environment art focused mini-milestone we named our “Biome Jam,” while also continuing to grind away at the usual work that you’ve come to expect.
The goal of the Biome Jam was to see how quickly we could create new environmental themes. We weren’t looking to make complete zones (and mostly reused the same terrain), but rather convey the theme through minimal use of new textures, props, lighting, and fog.
This was meant to give us some insight into future work, while also just being fun and giving you a taste of things to come (both in the nearer term, as well as long-term).
Keep in mind that images you’ll see below are hitting the 60-80% quality bar for future work - but we think the convey things pretty well already.
So, let’s get started:
Concept Art
NPC Concepts, including:
Harpies
Faeries
Bears
Wolves
Fluffy Spiders
Centaurs
Various Birdmen (and women… and undead)
Random other creatures
Biome Jam Swamp Props
Spider Dungeon
High Elf City
Character Art & Animations
Female Gnomes are now playable
Iteration on Female Gnomes
Female Wood Elves created
Male Wood Elf started
Spider created and added to the game
Orcs created
Female Ogre Animations continued
Female Gnome Animations started
Spider Animated
Environment Art
Biome Jam!
Biomes include:
Forest City (some other images shown in last update as well)
Snowy
Swamp
Temperate Coast & Hills
Volcanic
Did a quick experiment using texture and fog variations to see how much it changed the vibes in one zone
Continued to test out our new terrain pipeline
Concepted and grey boxed a location only known as “Party City”
Experimented with an environmental lightning effect
Audio
New dungeon ambience sound design in Wyrmsbane
Replaced a few UI sounds
New Skeleton Idle sounds
New Elemental Idle sounds
Fixed a few spell sound bugs
Design
Content
Added some new mob spawners to Sungreet Strand
Deep Dunes minor population adjustments
Bone Construct in Tomb of the Last Wyrmsbane has been reworked
Spiders can now be found in the world
Faction
Adjusted starting player factions for various classes and races
Adjusted Primary Faction of several NPCs
Added Spider factions
Traits
Added Trait: Sneak
General Abilities
Added Ability: Reclaim Essence
Added Ability: Cure Blindness
Added Ability: Sneak
The Placate and Lull line of Abilities now has a slight chance to break early
The Placate and Lull line of Abilities now has a slight chance to critical fail, causing primary aggro and social aggro
The chance for most Invisibility spells to break early has been slightly reduced
Dispel Magic can now be casted on non-group members
Fixed Purge ability to properly be a spell and have the normal attributes of a spell
Fireburst now does a flat 15 damage
Minor Regeneration health regeneration reduced from 3 health per tick to 2 health per tick
Several Auras' ability gem are now properly colored purple as they are group buffs
Fixed various ability descriptions
Heal now level 8 down from 16
Divine Protection & Arcane Protection now properly apply Soul Exhaustion upon fading.
Bard
Adjusted the acquisition level of various songs
Added Aria movement speed/secondary melee slow line of songs
Elementalist
The Fire Elemental Pet's Backlash of Fire aura has had its Damage Shield damage reduced from 1 (L1) to 30 (L60) to 1 (L1) to 15 (L60).
Enchanter
Added Melee Haste Ability Line from Shaman starting at Level 4
Melee Slow Ability Line now starts at Level 4
Inquisitor
Added Spell Slow Line with Abilities: Mind Exhaustion, Mind Enervation, Mind Fatigue, Mind Lethargy, Mind Debilitation
Necromancer
Added Ability: Sacrifice
Paladin
Re-added Ability: Mark of the Crusader
Mark of the Crusader now only procs off of Primary hand and Ranged attacks
Mark of the Crusader now also removes enemy Damage Shields
Restorative Smite now only heals yourself, not your entire group
Now has access to Divine Aegis ability
Rogue
Vial of Smoke ability now requires Vial of Smoke reagent
Added Ability: Sap
Stealth now has a 10 second cooldown
Shadow Knight
Aura of Vampirism now also procs off Ranged attacks
Added Ability: Sacrifice Minion
Shaman
Added Ability Line: Susceptibility (Susceptibility, Susceptibility II, Susceptibility III, Susceptibility IV, Susceptibility V)
Added Ability Line: Wolf Spirit Pet (Tiers 1-8)
Added Ability: Blind
Removed Ability: Swiftness (This ability is now only available to Enchanter)
Melee Haste line now begins at Level 8
Restorative Spores is now properly usable at Level 28
Spellblade
Added Ability: Singe
Spellblades now start with Singe
Added Ability: Unstable Barrier
Added Ability: Aura of the Arcane
Fixed a bug where later tiers of the Chilling Embrace line would still proc the same lower level Chilling Embrace debuff
Wizard
Added Spell Haste Ability line from Enchanter
Tech
Fixes
Fixed a couple spots on one of the crest rivers where the water spline width wasn't large enough
No longer automatically target self when casting a beneficial spell with no target; instead, it self casts the spell with no target
Camera movement amount is no longer frame rate dependent
Fixed an issue where some settings were not actually being changed when you changed them in the settings window
Fixed target cast bar appearing empty briefly after the cast completes
Lots of updates and fixes to the character controller:
Fixed character getting stuck when walking under objects
Smoothed out jump speed bonus
Polish on newly added crouch
Ladders now work and show a climbing animation
Added run/walk toggle
Several fixes for scaled up entities (animation, physics, hitbox)
Fixed the Disengage (ARC) ability
Wired up animation for turning in place
Fixed camera jitter at high FPS
Improved animation and perception of other players movement
Additions
Increased the number of abilities that the pet window supports from 4 to 8
Added confirmation when trying to sell an equipped item
Added graphics settings section to the settings UI
Added reset to defaults button in the user settings interface
Added hotkey to take a screenshot during play (F12 by default)
Added vsync setting
Guild invite now uses Confirmation UI for visual consistency with other confirmations, and added auto-decline timer functionality to the confirmation ui
Allow /assist [name] command to find npcs by name
You can now replace a hot-button with another that is on your cursor
Nametags are now clickable to target an entity
Added a new /devnotes tool that lets us place notes in our database, that we can see and interact with in the game
This will allow us to more easily share location-based feedback, bugs, etc and speeds up development
December Playtest & Tech Summary
I. Public Test Data
The data above proved to be interesting for us. We looked at both the Public Playtest in December, as well as the previous Stress Test. Since we carried over player data from the Stress Test, we could look at who participated in both tests or only one. This led us to look at some questions around retention that we’ll be digging into more as we go. Ali touches on this below.
A fair number of Stress Test players did not come back for the Public Test. We’ll dig into whether or not this was due to the short nature of tests, them being unavailable during those times, finding the game is just not for them (this is to be expected for many), or if it was just due to the technical difficulties faced during the Stress Test as mentioned below: ie. “The game was laggy during the Stress Test, I’ll come back later in development.”
If any of these apply to you, hit us up in Discord or on one of the livestreams and let us know.
Overall, having over 2000 people participate that weekend was cool to see. And we really appreciate folks taking the time to check out the game, even in its early form.
<—The rest of this was written by Ali and covers more of the technical details —>
II. Stress Test: Profiling and optimization
The team held two stress tests in November that were meant to test the limits of the servers in preparation for the December open test. Here were some findings that led to optimizations and fixes crucial for the December test.
Thank you to all who participated!
A. The Patching Process
Getting the game out to testers on the first stress test was quite stressful (hah!). We were using FTP servers to push the patch files to testers using our launcher, but that quickly flooded our bandwidth capacity. Furthermore, changing strategies on the spot was not possible because the endpoint was hardcoded on the launcher. This meant that our only recourse was to push a fully patched executable to clients.
In the second stress test, we changed strategies to using CDN distribution and HTTP downloads. This made patching much better for 99% of testers as once CDNs cache the data, it was very fast for testers to download.
It just so happened that one of the CDN nodes was having difficulties that day, however, and we found users out of the UK struggling to patch. The patch was failing after 2-3%. The worst part was that for those testers, we could not just give them a fully patched client like we were able to during the first stress test. The launcher depended on receiving a manifest from the server to validate files. Sorry, chaps.
In preparation for the December open test, we fixed a number of bugs on the launcher that were uncovered during the second stress test, and we also created a procedure where we could instruct users to completely bypass patching and launch the game directly.
Luckily, we did not need to use this in December as all the CDN nodes were working well for us!
B. NPC movement and AI limits per frame
NPC Pathing, NPC Awareness and Aggro were all taking an unreasonable amount of time per frame in the stress test. This was the reason the game was borderline unplayable during the start of the first stress test.
We were able to gather good logs and profiles that directed us to those bottlenecks that only show up at high user counts.
C. Lighting everything in real time
Due to MnM’s dynamic time of day system, and the sheer size and amount of lights in some of the zones (looking at you Night Harbor), baked lighting is not a possibility for us. Therefore, the team decided early on to focus on getting real-time lighting working well and optimizing for that.
Leading up to the test, though, Night Harbor was getting a lot of work put into it, and the zone was developing at a very rapid pace. The team was not able to optimize lighting before the first November stress test, and the full ramifications of this were not known until the test was well under way. With the added load of 500+ players and thousands of corpses littered all over all receiving lighting and casting shadows, frame rate really suffered for testers.
Some optimizations were necessary, so:
Light culling was implemented in Night Harbor
Corpses no longer cast shadows (since they’re on the floor anyways)
Light cascades were adjusted so that the shadow map was not taking up gigabytes of video memory
Some improvements were noticeable in the second test, and many more came before the December test.
D. Cloth physics culling
This was another thing that was only caught at very high user counts. Only certain humanoids have cloth physics in Night Harbor, and there are not too many of them until you add hundreds of players (and their corpses), and then the frame rate suffers for it.
Two immediate optimizations were obvious:
Dead things don’t need cloth physics
Things that are behind you or are occluded by other geometry can skip their physics calculations
A third optimization was not so obvious, but was critical: Every humanoid, whether or not they had a visible cape and robe, still had running cloth physics on those layers. Once this was discovered, it was easy to check if those layers are in use, and disable cloth physics on those layers if not.
We were also able to optimize the calculations so that we did not affect the fidelity of the cloth physics, but were using far fewer CPU cycles.
E. Player Retention
The poor frame rate in the first test, coupled with the poorly performing server were good things to find during the stress test. However, we believe this contributed to the above mentioned drop off in players that played during the stress tests and did not play again during the public test.
We also noted a much shorter session time in the stress test. Granted, the servers were up for a much shorter period, but also if a player logs in and finds their play experience to be not ideal, we can see why they would log off and not come back!
Hopefully now that we have optimized the server and client we can keep performance running well for the remainder of the Proof of Concept period.
F. Bots, bots, bots
The stress tests really highlighted how necessary it is for us to be able to test things regularly at high user counts. Therefore, the team prioritized writing simple bots for the game that can do basic actions like move around, do simple combat, loot items, move items around their inventory, talk (in gibberish), and consider things. They also log out and log in often and can combine all those actions in configurable frequencies.
This was key to testing the above optimizations without requiring additional stress tests that could have gotten too repetitive. However, nothing replaces real human testers!
III. Public Test Findings
The December public test greatly benefited from the November stress tests breaking so badly. The team felt much better prepared, and therefore the sessions were much less hectic behind the scenes.
We also took a strategy of not really doing anything that impacts play experience (such as cramming as many players on one server as possible) and ended up putting up 5 servers:
2 in the US West
2 in the US East
1 in Europe
The US East 1 and US West 1 servers still ended up getting a decent amount of load (and therefore yielded good load test data).
Our key technical findings:
A. Public Test Server Framerate Findings
We still noticed significant slowdown on the server, which manifested as slow responses to actions (casting, moving items around, quest hand-ins, etc), as well as jerky NPC movements.
During the test, we lacked the tooling to really get good profiling data past “this is not ideal,” and could not get good data on what the server frame rate was actually running at. This prompted a quick rethink on our profiling tool stack.
We have greatly increased the amount of data we have access to in a live production environment. We also got great load test data that we could use to replicate the environment using our army of bots.
This has led to many optimizations. We found multiple new bottlenecks. The worst one was when a character was loading in, the process of checking if they had any saved pets was creating an almost second long pause of the entire server... for everyone!
Talk about worst case scenarios!
We also found additional bottlenecks in character saving (this has now been moved to its own thread), aggro, client awareness, and client-server sync netcode at high player counts.
While we did not have accurate figures of US East 1’s (the most populated server) frame rate during peak hours, we estimate the effects were only felt when frame rate dipped to below 1 frame per second. At its worst, we estimate it was at 0.1 frames per second (yikes).
As it stands today after optimization, simulated loads show frame rates of approx 30 fps, which we have never achieved before.
We look forward to more stress testing sessions to validate these results!
B. Crafting Stations
The only other technical issue we faced during the public test were to do with a strange state that you could end up in if you accept a trade with a crafting station open.
The design of crafting stations as of the test made it only a matter of time until this happens: The anvil, for example, could only be used by one player at a time. The entire game only has 3 anvils. Therefore, queues formed at the anvils. Players did not want to close the anvil screen as they may lose their position at the top of the queue, and therefore would be trading resources with other players with the anvil window open.
This had a random small chance of yeeting the character into oblivion (lol). The anvil would break and the character would be kicked off the server and would be stuck in a void unable to log back in. This also was spamming errors like mad on the server causing a lot of lag.
The only fix was to restart the zone. We restarted Night Harbor once on US East 1 when this was first encountered, then when the cause was known, we disabled crafting stations on all servers and restarted Night Harbor on all servers (including US East 1 again).
For the remainder of the test, the team acted as crafting stations, trading copper ore for copper gear!
IV. Player placed object example: Campfire
A lot of work has gone into player placeable objects, or as we call them internally Widgets.
A. Placement limitations
Widgets now can have a minimum distance from other widgets defined. This is to avoid a mass spam of widgets in one place creating lag or otherwise being a nuisance.
B. Synced states (dimmed)
Widgets may now also have states and sync states across the server-client netcode. Certain actions or timers may move a widget from one state to another, such as a campfire nearing expiry or having additional wood refill the campfire.
C. Persistence
Widgets now save and can persist through restarts, or can persist indefinitely if required. Widgets can also have their own inventories and placed items persist.
V. Resting and State Detection
The team has been trialing a new RPG system for being rested or tired. It also ties into character states. For example, we can now tell if it's raining and we are not under shelter. Eventually, this will lead our character to be “wet” and for that discomfort to have effects that must be managed. Not being under shelter while it rains also prevents you from resting.
You rest when you are near a campfire or within an inn or tavern. Being rested will provide various effects, but currently gives you an experience bonus until you get tired.
All these interactions are currently in proof of concept phase and are only represented via status text prompts. The plan is for it to eventually be visible on a visual UI element.
VI. No Rent Items
This was a big system that came in with a small change, due to an accumulation of functionality that could be utilized. The challenge with No Rent items is that they expire when you are offline, and therefore the server has no access to your data.
However, due to other systems that needed functionality like this, we are now able to check at the pre-login phase if that item should actually be expired and deleted.
At least one developer was really happy about this and immediately implemented some pretty powerful items that are No Rent.
VII. Guilds
We are all very excited to have guilds be a thing that is in the game. We have been wanting to create a guild ourselves for many years now, and we were so glad to be able to make one finally.
Guilds are created at a guild creation NPC, cost a little bit of silver, and come with their own guild management UI.
VIII. Housing
In an end of year curveball, houses have been implemented in a very basic state.
You can own a house (but not yet buy a house), and you can have co-owners, guests, and permissions defined.
Doors may be locked or unlocked, and the whole house may be marked private and will kick Gnomish trespassers (who can as of now fit through many windows) to just outside the main door.
In Closing
As you can see, it was another busy month. Though, a couple of the tech items listed in Ali’s summary were being wrapped up in December.
We’re currently looking ahead at how best to tackle 2024. Our planning will get us closer to being able to define a landing point for the Proof of Concept phase we’ve been working in the last couple of years.
We look forward to discussing that with you more in the coming months.
Folks continue to ask how to support us and the answer to that is the same: Please spread the word!
Thanks for reading this far!
We love you guys!
See you in the community.