Blizzard Developer on Sharding, Layers, and The Blood Moon Event - Season of Discovery

Classic Posted 2024/02/18 at 5:13 PM by Rokman

interesting insight on Twitter

Tom Ellis

Caveat, there's no announcement of any changes in this, this is all just fascinating info if you're into that sort of thing.

Layering, Sharding, let's dive in.

So, in what we call Mainline (modern game, latest expansion) we use tech called Sharding ever since Warlord of Draenor. Once upon a time, a realm ran one copy of each zone locally on it's own hardware, if you ran after someone and they crossed into another zone, you followed them, it was all running in the same place, you could kite mobs across zones, escort quests had NPCs pathing across zone boundaries.

Problem, as hardware improved and performance improvements were made, expansion after expansion we raised the cap on the number of concurrent players we allowed logged in. If everyone wants to play on Tich, and we have the technical capacity to lower queue times, why wouldn't you? Ah, but the realm still only ran one copy of the world, eventually the realm has so many people on it that if even a small percentage of them congregate in the same place its going to be way more than the CPU core that's running that zone can handle, in comes the lag, you can multithread to your hearts content, eventually you're going to have a single thread bottleneck that will limit what you can do.

This all came to a head in Warlords of Draenor when in the face of a very successful launch the starting zones in WoD were completely unable to handle the load they faced. The sadly departed @kurtismcc (not dead, just left Blizzard...) worked for three days and nights (we don't condone that sort of behavior anymore!) and created the first version of sharding, this new tech allowed us to run multiple copies of the world and split people between them. Finally we could run multiple copies of Draenor and handle the load between them.

With Legion this tech was made into a fully fleshed out feature and each individual zone could now run limitless copies, once a threshold of players in the zone was reach, we spun up another copy, repeat as necessary. It made the game feel less cohesive, but it meant the realm cap and zone capacity were now disconnected entirely, it also gave us the ability to have specific values for specific zones (not all zones are equal in terms of CPU cost).

Good times. Now, along comes Classic and we have problems again.

Classic launch was heavily targeting #NoChanges, they wanted the game to feel as identical to original launch as possible. Sharding was clearly out, but there was also the strong belief that launch was going to be big (but no where near as big as it turned out ;)) and interest would drop off fairly quickly as people got their nostalgia hit and moved on. In that world we didn't want to have to launch hundreds of realms at the only tiny realm caps we used to use and then have lots of dead realms a month later. So we had to find a way to use modern sized realm caps but not crush the zones with players. There were also two other notable tech problems with Sharding that needed addressing to support Classic behavior, mobs can't path across zone boundaries in a Sharded world, each shard (copy of a zone) can be running on completely different datacenters, so while technically solvable its very expensive and complicated to try and make this work and lastly in PvP situations it was greatly desired that crossing a zone boundary shouldn't be an escape, both people should keep chasing each other into the new zone and in a Sharded world, they could end up on different destination shards.

In steps the legendary Omar, he took sharding and started stitching the shards together into something that became known as Layers. Yes, a Layer is actually just sharding with a lot of sticky tape. By gluing these shards together it became possible to fly messages around and handle mobs moving between zones, players were assigned a specific Layer so if two people were in Westfall and ran into Duskwood, they would remain on the same layer and so end up in the same copy of Duskwood as each other. Hurrah, great success! This tech was required to have any hope in hell managing the AQ gate opening, eugh, shudder. That's a whole other story.

Through Classics original launch we slowly kept lowing the max number of layers as originally we had promised Layers were temporary, but (un)fortunately you didn't stop playing Classic, it wasn't the flash in the pan some had predicted. You really, really liked Classic. The servers still had tons of players on, so fulfilling this promise was tough and it was only towards the very end of Classic we were able to run single-layer, and even then, that was a very spicy Layer.

From TBC onwards we had to embraced layering as an evergreen thing, the realms were just too big and demand was too high to think about single-layering anymore.

Now remember what Layering was designed for, trying to make a semi-permanent copy of the world that you would live on for the duration of that week? Well people didn't play the game that way. The Asmonlayer was born, meta-gaming meant that if the Alliance were blocking Blackrock on Layer 1, well then the Horde will all try and get to Layer 2 instead. The game evolved around Layers existence and while Layers were once imagined to be largely invisible to regular day-to-day play them became very much part of gamesmanship of Classic.

Ok, history over, what's going on now and why is Layering suddenly a hot topic in SoD?

Layering is more rigid than Sharding. We dynamically spin up layers based on load, and historically we didn't collapse layers when load fell, it was felt important once upon a time that the world feel full at peak and empty off peak, just like it was back in the day. But, like a lot of our early assumptions, that didn't match what people really wanted. They wanted to see other people (shocking in an MMO I know) they wanted to be able to form random groups with strangers, they wanted enough people for the world to feel populated regardless of the time. With SoD we for the first time enabled the tech that collapses layers as load falls so it tries to keep every layer significantly populated, that change has been *mostly* successful, it does have some behavior we'd like to change (moving you multiple times, potentially splitting up groups etc). But as a result you're now seeing yourselves "layering" more often, normally this was an opt-in thing, if you join a party with someone on another layer you expect to move, now for the first time in Classic you could be moved randomly without warning.

The biggest problem we are slamming into at the moment is some early code assumptions for layering are now very problematic, all zones in a layer have the same number of players that can be in the zone. We can't for example say, 500 people in Ashenvale, 1000 people in Orgrimmar, they all use the same value. This is a nightmare when zones perform very differently based on what players are doing in that zone (idling is less CPU than a giant PvP slugfest). Layering is also very expensive from a hardware perspective, remember if the entire server goes to STV during Bloodmoon, we spin up entirely layers to handle the extra STV, so we may be running 16 empty copies of every other zone just because we need 16 STVs, gross. There's also a host of features that zones can't have in a Layered world that Mainline Sharding can do, mainline can do Warmode zones that balance faction 50/50 for example, and we have have different targets for population based on PvE vs PvP.

So right now, world PvP and layering is proving problematic and hard to balance. As always, we're working on it and good things will come. Come Monday we're going to diving into reports of people getting split up when they were already on a Layer together (that's not expected, layering should occur on party changes only) and Classic team already mentioned they're working on a change to keep parties together in the event of a layer change due to excessive load.

The baby just woke up so I'm going to stop jabbering on here and get back to #DadThings, happy Sunday!

Tom Ellis

Layering Part 2: Design & Engineering Clash!

Dramatic title, not actually any conflict here. So what's the deal with the increase in grumbling around layering in Season of Discovery. Because we've not actually changed anything, Classic, TBC and Wrath all ran the same tech.

The Classic team is trying to add content to Classic that "feels" like it belongs in Classic. One of those things is World PvP. Tarran Mill vs Southshore for example is iconic, it spent much of Vanilla being effectively a permanent outdoor PvP battlefield. So they created the Ashenvale event, and now the STV Bloodmoon event to try and add some extra spice and carrot to this behaviour, overall when it works folks enjoy it, good stuff. But why has it been so problematic from a layering standpoint?

Three issues player have faced -
1. Lag.
2. Inability to group up on the same layer.
3. Getting removed from a layer you were on mid-fight.

Let's hit each of these, lets start with lag. Lag that we're talking about here is lag in the simulation itself, i.e. you're hitting buttons but the game is taking a long time to acknowledge those pushes. This lag is caused primarily by a lack of available CPU, there are some single threads that can only run on a single core and if that core is at 100% it can't go any faster, if there's more work than it can handle in one server tick, that normally 50ms tick is going to take 500ms and you're going to start noticing delays on things.

All sorts of things contribute to this lag, players moving around, players casting abilities, mobs spawning etc, these all take CPU time to compute, the main thing we can do to control lag is usually balance the number of players in a zone because players are by far the most expensive thing for the server to handle. The worst thing for the server is lots of players on top of each other fighting, because then every Arcane Explosion has to get calculated against hundreds of players, are they in line of sight, do any of them have resists, shields, how much damage do they take, are they in range of the AoE... everything the server has to figure out.

So, PvE fighting in a clump is expensive, PvP is really expensive. But we just made content that does exactly the thing that the server is worst at! Argh! This isn't a failure of design, Blizzard isn't driven by engineering when it comes to gameplay, generally speaking its engineering's job to allow Design to do whatever makes the game the most fun (there are limits ofc, resources aren't infinite). So, when Ashenvale first started running into lag, the first port of call was to reduce the player targets for the zones (remember in Layering, zones all share the same cap so this meant we were having less players in *all* zones). Less players, less work for the server, less lag. Throughout phase 1 we were able to dial in these numbers to find one that got as many players as we could into the zone without it falling over (some lag in mass-PvP is honestly expected as players want to see the sea of enemies).

2. Alright, folks can't manage to group up together and while forming a raid people are getting split up like crazy into different layers, why? The layer system has a few values, one of those is the "Target" number of people we want in a given zone, another is the "Max" number we want in a zone. Remember when you log in for the first time in a week, you're assigned to a specific Layer, that's your layer for the week. When you group up you can move to another layer but outside of that group you should return to your originally assigned layer. Once a zone reaches its "Target" value, we start ignoring that layer reservation, basically if you were assigned Layer 1 for the week and you travel to STV and the game finds Layer 1 is over its Target value, we will instead put you on a less populated STV (and layer, since zones=layers in Classic). This can be the first way that you struggle to group up with friends.

But, you're smart, so you form a party or a raid with the person you're trying to play with. At that point we will ignore the Target value for the zone because you really want to play together. You both end up on the same Layer, hurray!

Everyone does the same thing. Uh-uh, now we have a *lot* of people in the same STV, we're going to start lagging if we can't keep the population under control, but everyone keeps pulling in more people via groups. A new value takes effect here, "Max". When we exceed the Max value for a zone, we stop caring if you have a layer reservation, and we stop caring if you're in a party with someone. The copy of this zone is full, we can't keep sending people there or its going to fall over.

This is one of the big issues you're facing at the moment in STV. Since everyone is forming 5 man groups and usually one person arrives ahead of the others, its running into the Max value for the zone and not allowing the rest of your group to come into the same copy of the zone. Think about it, if the zone can handle 500 people, and allows them all in then they each invite 4 more people, ah crap, we're going to crash. So we have to have a hard limit somewhere.

3. Getting layered unexpectedly after you've managed to all get together. This is the first one that I would consider a bug and not just a bad user experience (some bad UX when the alternative is the server dying is OK). Based on what we currently know, if 5 people are all together in the zone in a group, they shouldn't get split apart after that point, but folks are reporting just that, so that's one we need to dive into and figure out what's happening. Now, the good news is Classic is already working on a change that will keep parties together (so even if you move layers, you *all* move layers together) which will mitigate this issue but there's still a behaviour here that isn't expected that we need to get smarter on.

So there you go, that's enough for a Sunday deep dive. The TL;DR? Groups are hard for the layering system to handle, and once a zone is full the UX at that point is gross. As the Classic team already mentioned theirs changes coming to help improve that, and my end I've been noodling the Target & Max values to try and find the sweet spot between Lag and Groups.

As always, if you find this sort of thing fascinating, consider a career in Design or Engineering! Game's are fun and hillariously complex beasts to work on but very rewarding.

12 3

Comment by 770204Actual communication of current issues, it's amazing. I gotta say, it actually gives me faith that things are getting worked on and improved upon instead of radio silence. Thanks for the infoComment by Basadomaybe one day blizzard will invest in servers that are significantly better and we don't need this layering/shard crapComment by TheRedBaron20Haven't really experience much in the way of lag myself, though the lack of mining nodes in the open world is horrendous. Increasing the spawning of mobs and nodes seems like a good compromise.

Of course we shouldn't be getting layered away from our groups, that's definitely not a good idea and is likely not intended, right?Comment by Seniaamaybe one day blizzard will invest in servers that are significantly better and we don't need this layering/shard crapComment by Basadomaybe one day blizzard will invest in servers that are significantly better and we don't need this layering/shard crapComment by BlightyIt's almost as if incentivising the entire server to converge on the same zone at the same time is a bad idea...

There are 3 other zones which share the same level bracket as STV; Dustwallow marsh, Badlands, and Swamp of Sorrows. Surely they could incorporate those zones with their own thematic "blood moon" event and split the population a bit. Could have different rewards unique to each zone to incentivise people to switch around.

2024 and these poor developers are having to put in a shift to compensate for trash tier servers. $74Bn company everyone...Comment by johntravolta342maybe one day blizzard will invest in servers that are significantly better and we don't need this layering/shard crapComment by iamcevrus2I love that they're still like "we didn't know Classic was going to be so popular at launch". All the data the players saw prior to launch indicated it was gonna be popular as ever, but I guess when you hamstring yourself by having useless load testing AND ignoring amount of name saves, not that shocking they were ill-prepared. Also nice they're still glossing over the fact they had new realms and WAITED and WAITED SOME MORE to launch them because they didn't want to have to merge them later when numbers declined. Nothing like treating your customers like garbage because an action in the future might make you look kinda bad (hint: guaranteed big bad is worse than maybe middle bad). They made playing with friends on the realm you wanted (and sometimes at all) a nightmare that first week and it still overperformed.

As for layering and sharding the following tidbit is especially egregious "some early code assumptions for layering are now very problematic". So they learned nothing from the whole "not being able to do anything with the base bag" for the longest time. I love that they're shocked when they need/have to do major changes to stuff in-game. Who could've guessed that an ever-changing live game might need to be able to evolve?Comment by BenefitzThat sure is a lot of words to say "we can't fix the lag during the Blood Moon event, but here's how layering works so you know why we can't fix it".Comment by etnies445maybe one day blizzard will invest in servers that are significantly better and we don't need this layering/shard crapComment by etnies445That sure is a lot of words to say "we can't fix the lag during the Blood Moon event, but here's how layering works so you know why we can't fix it".Comment by Basadomaybe one day blizzard will invest in servers that are significantly better and we don't need this layering/shard crapComment by LolhaiCommunication is nice, and watching Blizzard (Classic team or otherwise) interface with the community throughout SoD for the first time in over a decade has made me far more trusting that these issues genuinely are being looked into.

In my opinion, it's just a terrible shame that we were ever subjected to the sharding/layering technology to begin with. It utterly killed the social aspect of WoW overnight when it was first implemented back in WoD, and to this day continues to run antithetical to the spirit of MMOs.

Long gone are the golden days when I could visit Goldshire, or outside SW/OG and see the same familiar faces chatting and dueling one another. I suppose it was more profitable to attempt to shoehorn as many players as possible into the same server, even if sharding (and layering) means that there's a very real possibility that two people could spend years on the same server, yet never even know of one another's existence.Comment by ablockThat sure is a lot of words to say "we can't fix the lag during the Blood Moon event, but here's how layering works so you know why we can't fix it".Comment by ElizarTVClassic WoW/ SOD/ Classic + is the future of World of Warcraft. Give up on the "Modern/retail" version and put every single ounce of effort & money into making this version of the game the best version that has ever existed. Wipe the slate and start over Blizzard, now is your chance.Comment by Himeamaybe one day blizzard will invest in servers that are significantly better and we don't need this layering/shard crapComment by MollertenThat sure is a lot of words to say "we can't fix the lag during the Blood Moon event, but here's how layering works so you know why we can't fix it".Comment by DarkAvatarIf you don't want/can't expend more money for servers, just go back to sharding system.
The benefits are way more relevant than losing a chase between zones, or bring Teremus to Stormwind.Comment by MyFingerIsOnCDThis kind of discussion would make a great Blizzard podcast. Far more interesting than Ion and Morgan espousing their gaming "philosophy." Informative write up from Blizz here.Comment by DrakknfyreMaybe instead of explaining the feature he should instead take that time to help FIX sharding in retail. Currently it's broken AF and seemingly happens at random regardless of the zone occupancy and it's absolutely screwing content up. With so much group and time-centered content, getting ripped from one shard and thrown into another one where the event is either not going, has gone differently, or everything has reset is not fun. It's not fun the first time it happens. It's not fun the 5000th time it happens. And it's especially not fun when you're actively in combat, especially against, say, A WORLD BOSS, and poof, you're now standing in front of the boss alone, and they're at full health.

12 3