[h1]Hello Riftbreakers! [/h1] It is time for another co-op mode development progress update for The Riftbreaker. A lot of good things happened since the last time we shared the news with you, including some major developments and improvements. A lot of those upgrades happened thanks to the data you provided us with during the closed beta. We appreciate your help so far! .We even managed to complete a 3-person playthrough live on stream several times. Here’s a new VOD from one of those streams: [previewyoutube=pWSf7mgK6ao;full][/previewyoutube] As you can see, the game runs quite well, and even though the performance still leaves a lot to be desired, especially in the later stages of the game, we feel we are on the right track. Plus, we have a new, fancy 3-face cam setup and managed to convince some EXOR members to break their ‘stage’ fright! But you’re here because you want to learn more about our development process, not just to watch a video. Let’s take a look at what our brave programmers have been doing for these past couple of weeks. [h2]A PROMISING START[/h2] At the very beginning of the year, on January 2nd, one of our programmers, Starbugs, sent us a screenshot from his Survival run. It was a single mech, a Headquarters, and a ton of defensive towers - obviously running an infinite resources cheat. The screenshot was taken at the end of the run, and Startbugs told us that he managed to almost complete a run. It was an improvement since the game used to crash within 15 minutes in similar conditions. It was a positive message to start the year, but we were careful to share this enthusiasm. [img]https://clan.cloudflare.steamstatic.com/images//34659267/cc3704240f76c3c4fe8727a23a7ff31c7bf65876.png[/img] [b][i]Yes, we use Skype. Yes, we know about other software. No, we won't switch. Yes, we are stubborn.[/i][/b] This is a screencap from our group chat on the 2nd of January 2024. It goes like this: [b]Starbugs:[/b] Some good news for the start of the year. 5 more minutes till I finish a survival run in multiplayer! [b]Angin3:[/b] But you’re cheating! Doesn’t count! [b]Starbugs:[/b] shut up [b]Prosatanos:[/b] That’s not a real base, still cool tho [b]Starbugs:[/b] Smartasses. I was trying to be positive here. It used to crash after 15 minutes. You ruined everything! I won’t tell you anything anymore! Most of us were skeptical at this point, as the “base” Starbugs presented wasn’t representative of what you usually see in the game. There were no factories, no power generation setup, or basically anything else. That’s important because every structure you build increases the computational cost for the game. A setup consisting of an HQ and a couple dozen towers using an “unlimited money” cheat is far less resource-intensive. Our snarky comments about Starbugs’ (cheater) tactics did not manage to extinguish his inner flame. Both he and lukaasm braved on, playing through multiplayer run after multiplayer run, fixing numerous bugs on their way. [img]https://clan.cloudflare.steamstatic.com/images//34659267/f87dcdc275ed68910af6e400d6616315de2ae639.png[/img] [i][b]The benchmark results point out a continuous improvement in data transfer optimization. The reduction of bytes transferred is one of the most important things we can do to increase performance and ensure a comfortable play experience. At present, the transfer rate is acceptable for a fast internet connection. However, we still have much to improve.[/b][/i] Thanks to your active participation in the closed beta of the multiplayer mode and the data you provided, our team could deduce which components they should pay attention to. Some bugs the programmers encountered during gameplay were a direct result of some game systems being unoptimized for online play. Reworking these allowed our team to improve the game's performance significantly and eliminate many bugs in the process. As you can see in the screenshot above, the results are clearly reflected in our benchmark scores - the lower these lines go, the better the performance. [h2]THREE-PLAYER SURVIVAL MODE RUN THROUGH THE STEAM NETWORK[/h2] [previewyoutube=vxkzEbHqPns;full][/previewyoutube] [b][i]When you play with others, you might actually have the time to build some more walls! Who would have thought?[/i][/b] After a couple of weeks of intense development and rapid improvements, The Run™ happened. We managed to complete a Survival Mode run, playing as a three-person team. We have done this before, but it was the first time the game felt truly stable, and the performance was acceptable (for this stage of development). We were ecstatic. Then, we repeated our feat a couple more times during our development streams at www.twitch.tv/exorstudios, gradually improving our streaming setup. At one point, one of us even connected from outside the office - and the experience was okay as well. Sure, the distance between the office and Paweł, who worked from home at that time, was more or less ten kilometers. Still - it was an outside network, the data was transferred through Steam’s datagram relay network, and we completed a full playthrough of the Survival mode. It was a success - no doubt. [h2]CO-OP CAMPAIGN PROGRESS[/h2] [previewyoutube=xFK_p354XoQ;full][/previewyoutube] [b][i]We all have to sacrifice something for the greater good in co-op. Even your own mech upgrades (I choose to completely ignore the fact that maintenance tools would have made buildings cheaper, which would benefit everyone).[/i][/b] The programmers were quite sure that they could pull off even more. They decided to try running an entire campaign in co-op mode. Given that no one has tried that before, it was a huge undertaking. They started by making sure that the save system was functional. It turned out that it needed a little bit of encouragement, but after some bugfixing, we are now able to save the state of the game on the server. It is currently stored on the server itself and can be loaded from the menu as usual. Once the game is running, the clients can join in, and if they participated in the game, they also receive all their equipment back. This is a very basic implementation - we are still unsure what it will look like in the final game. We have a couple of options to consider - keeping saves on the server, giving each player a copy of the save file, etc. However, we will make our final decisions closer to release - now, we are focused on getting the game running in the first place. [previewyoutube=YEiWif3RAxs;full][/previewyoutube] [b][i]If it's ugly, but works, is it really that ugly?[/i][/b] With a functional save system in place, it was time to start the playthrough. Since the guys playing were programmers, they were able to debug the game on the go and fix all the game-blocking issues they encountered. Naturally, there were a lot of crash bugs, things not working as expected, and other minor inconveniences. However, the bugs that are not obvious at first glance are always the hardest to spot and fix - and there were quite a lot of them. [h2]ERADICATING BUGS[/h2] [previewyoutube=91EitLF2Ma4;full][/previewyoutube] [b][i]You can take a guess whether the power plant survived or not.[/i][/b] One of the first such bugs was connected to the radar. As you know, the mech in The Riftbreaker has a short-range radar connected to it, revealing points of interest within a certain radius around the mech. That radar also reveals the fog of war, marking the map cells as ‘discovered.’ Those cells are quite small in our game, and a lot of them can fit into the 15-meter radius of the short-range radar. It turns out that the clients synchronized ALL those revealed cells with the server, each and every frame the game was running. The more players there were, the worse it got. Hundreds of kilobytes of precious data, up to 30 times a second. This bug has been fixed, and clients only send new information to the server. [previewyoutube=IInORNOJY2A;full][/previewyoutube] [b][i]Remeber to always communicate clearly.[/i][/b] Another bug our team discovered was connected to the objective system and the way we handle its updates. Whenever a player was given a new objective, finished their current one, or when an objective failed (it can happen!), we synchronized the state of the objective system between the client and the server. Sounds logical, right? We thought so, too, until we realized that the mission timer ticking down was an objective update, too. That meant the need to synchronize the system every second. Still doesn’t sound too bad. What made matters truly awful was the fact that we forgot about the mission log - the menu page where you can see the entire history of your campaign. All changes in your mission goals are stored there, including timers. Suddenly, we were facing an update with a massive amount of data every second. We made the decision to exclude timers from the synchronization process, which fixed that bug. [previewyoutube=RA_gXo6E5SQ;full][/previewyoutube] [b][i]Our tests are usually carefully planned and conducted according to our internal rulesets. Usually. Sometimes we just jump in the deep end and see what happens.[/i][/b] In a similar fashion, the research also caused problems with an unnecessarily high number of updates, but for a different reason. When you add any technology item to the research queue, a timer is displayed, showing you how much longer it takes until it’s done. In reality, however, research is not time-based at all. Every tech item in The Riftbreaker costs a certain amount of download points. Communication Hubs produce those download points at the rate of 1 point per Hub per level. The amount of that resource is updated every frame, meaning it has to be synchronized every frame. Considering that download progress couldn’t have been updated without sending over information about the entire research tree, you get way too much data way too often. Decoupling the research progress and download values from the rest of the research system solved the issue. [previewyoutube=235ZruwrsVg;full][/previewyoutube] [b][i]Show me what you got.[/i][/b] By this point, you should see a pattern emerging - our systems often send more information than necessary. It was no different in the case of the energy graph system. Each building, power plant, and energy connector is a part of the energy grid in the game. They can get truly massive, as there is no limit to how many structures you can place. What if we told you that the first versions of the co-op build wanted a full synchronization every single frame? Sub-optimal is the mildest way to put it. Luckily, compartmentalization, packing, and reduction of the update frequency also fixed those issues. [previewyoutube=xOkFHrU6P40;full][/previewyoutube] [b][i]Words to live by.[/i][/b] In multiplayer mode, The Riftbreaker relies upon constant communication between the server and its clients to keep the game world state in check. Problems occur when the syncing happens too often or too rarely, as you will learn from our next example. If one player started an HQ upgrade process while the other player was somewhere else on the map, it was possible to get an infinite loop. When the player who was away came back to the base during the construction time, their game would notice that fact… and start another HQ upgrade process underneath. Rinse and repeat, infinite level 2 upgrades, infinite level 2 waves. We decided that some events need to be broadcast globally and reach all players, regardless of their position on the map. That fixed the issue. [previewyoutube=5nKMY6qyR6k;full][/previewyoutube] [b][i]You can set up some pretty powerful tactics while working with others.[/i][/b] Apart from fixing bugs, Starbugs and lukaasm ran a round of optimizations on every game component that looked suspiciously big or caused spikes in transfer. If you watched any of our co-op attempt streams, you might have noticed that whenever a player wanted to join the game, there was a huge stutter as they connected to the server. Turns out that the culprit was the MechComponent - a set of properties that every mech in our game has, giving the players the ability to move around and take game actions. The component was absolutely huge for multiplayer standards, generating up to 500 kilobytes of traffic per second per mech. After optimizing that data structure, the boys managed to reduce that rate to about 30 kilobytes a second. It's still too much, but it's a lot closer to the target. [h2]OUR GOAL AT THE MOMENT[/h2] [previewyoutube=ZKf7DD82EMw;full][/previewyoutube] [b][i]This is one of the symptoms of replication bugs. Thay make the game really hard to play sometimes and might have game-breaking consequences.[/i][/b] The biggest problem we are facing right now is replication. Replication is the process of recreating gameplay state changes from client to server and vice versa. In order to save performance, we divided some processes into those that happen on the client side and those that happen on the server side. When combined, we should have the full picture of what events are happening in the game world - ‘should’ being the keyword here. At the moment, it is possible for the server to ‘think’ that it sent all the necessary data to the clients, but some of that information never reaches their PCs. As a result, we see some strange bugs. For example, if you build walls, some of them are not visible to other players. The buildings player A builds can’t be sold or upgraded by player B. Creatures that died during combat on one player’s screen show up as standing corpses for the other. [previewyoutube=BfBKK2j35Nc;full][/previewyoutube] [b][i]After a while you get used to it, but you shouldn't have to get used to something like that. It's our focus at the moment.[/i][/b] There are many possible reasons for this behavior, and we are currently working through them to prevent this phenomenon from being a problem. The server is convinced that the data packets have been uploaded and sent to the client. However, they either never reach their destination or never get sent at all. It is also possible that the packets arrive out of order, and the data in them can no longer be applied to the game state. Either way - we have easy ways to reproduce these issues since they happen all the time. Data packet loss is something that can’t be prevented entirely. All that is left is to experiment, find all the problems that it’s causing, and work around them. It is definitely a complicated task, but we’re sure that we will have it fixed. As soon as we are able to finish a full campaign playthrough, we will begin the next round of the iteration process. [h2]WHAT IT ALL MEANS FOR THE BETA[/h2] [previewyoutube=vILYClnXE4Y;full][/previewyoutube] [b][i]We hope to let you gang up like this with your friends as soon as possible.[/i][/b] For the time being, unfortunately, not as much as you would expect. Despite the huge progress we’ve made, we’re still not ready to release the co-op mode to the beta testers. We have our hands full trying to fix the issues we have discovered ourselves. The list of known problems would probably grow by an order of magnitude once other people joined in. Their feedback would be lost or become irrelevant over time. Once again, we need to ask you for your patience. We will update the beta once we have a stable build and decide that we are ready for public scrutiny. [previewyoutube=LP-gXu_985g;full][/previewyoutube] [b][i]I mean, how long can you play with yourself before you get bored?[/i][/b] Still, there is good news. The previous round of beta testing, which was focused solely on PVP and networking aspects of the game, was a major success for us. The data you gave us has led to major breakthroughs, and we are sure that it is going to repeat once we add co-op into the mix. No effort from a team of 15 people (not all of whom work on the multiplayer mode) can beat the collective power of the internet. For this reason, you can be sure that we will give you the chance to try out the co-op in closed beta conditions as soon as we are physically (and mentally) ready to take on the challenge. [h2]CONCLUSION[/h2] [previewyoutube=BJZid8mWKSg;full][/previewyoutube] [b][i]A perfect summary of our progress. We are going in the right direction, but sometimes things go sideways![/i][/b] The past few weeks of work have been very fruitful, leading the co-op mode for The Riftbreaker to reach the most promising state yet. It finally feels like a real game and a fun one to boot. While we’re not ready to release a coop beta build yet, we encourage you to sign up for the waiting list, as public testing is very important to us. We plan to start it as soon as we have all the elements in the right places. For the time being, join our Discord at www.discord.gg/exorstudios, where we publish our daily changelogs and share even more information about what’s going on in the studios. Join our streams on Tuesdays and Thursdays at https://www.twitch.tv/exorstudios to have a chance to see our live playthroughs and chat with us. Things are good. Just let us cook a “bit” more. [h2][b]To sign up for The Riftbreaker Multiplayer Beta please fill in the following form:[/b][/h2] We reserve the right to contact only select participants. [h1][url=https://forms.gle/wdFeVGGHE1rwAdvx8]SIGN UP USING THIS GOOGLE FORM[/url][/h1] EXOR Studios