House of Games #64 — Testing, Testing

This week on House of Games we talk about testing, types of tests, and what (not) to do to find bugs, tune in!

Send in a voice message: https://podcasters.spotify.com/pod/show/house-of-games/message

Chapters

00:00 Episode Intro
01:01 How Rune Tests His Games
07:49 Otto’s Experience as a Tester
17:59 Unit Test
26:48 Smoke Test
33:22 Regression Test
37:49 Why Don’t Programmers Find All the Bugs Themselves?
42:59 Stress Test
45:47 Structured Testing vs. Unstructured Testing
52:36 Real-World Applications of Testing Skills
56:34 Reading Comments
58:58 Episode Outro

Your Hosts and Their Games

• https://store.steampowered.com/app/2436180/Sunset_Moon/ — Wishlist Sunset Moon on Steam!
• https://www.play-asia.com/red-colony-trilogy/13/70fmjh — Buy the Red Colony trilogy on Switch!
• https://store.steampowered.com/bundle/24596/Red_Colony_Uncensored_Threesome_Bundle/ — Buy the Red Colony trilogy on Steam!
• https://www.nintendo.com/store/products/knifeboy-rebooted-switch/ — Buy KnifeBoy on Switch!
• https://store.steampowered.com/app/889130/KnifeBoy_Rebooted/ — Buy KnifeBoy on Steam!
• https://ottowretling.medium.com/list/game-development-427e99aefd01 — Read Otto’s blog about game development!
• https://discord.gg/mYM72SQ9CP — The discord server for Game Hub Umeå.

Other Links

• https://doom.com/ — The homepage for the Doom franchise.
• https://rome.ro/ — The homepage for John Romero.
• https://youtu.be/ILkT_HV9DVU?si=G5OGB4MRb1LFwzfM — Lecture on Software Testing by James Bach.
• https://unity.com/ — The homepage for Unity.

Transcript

Episode Intro

[00:00:00] Rune: Halli, Hello everyone and welcome to the buggiest podcast in the world, House of Games, House of Games, House of Games, House of Games. Sorry, just ran into another bug. But, luckily, we are joined by a man who makes a living on testing and fixing your problems. Why doesn’t my wife kiss me anymore? Because you stink is what he told me and lo and behold I brushed my teeth, straightened my back and now I won my wife back. Thank you, Otto! But before we let you sort out our audience’s problems, let us open that door and enter this week’s episode of House of Games!

[00:00:33] Otto: Welcome to the newest episode of House of Games, and it’s been a while since we recorded actually, so I’m really glad to be back and so this episode I thought we were gonna talk about testing.

How Rune Tests His Games

[00:01:01] Otto: So I think, before I go into maybe my testing process and what I think is important when testing, Rune, do you want to go into, how does your process work when creating Sunset Moon?

[00:01:16] Rune: Yeah, I think I’ve touched on this many times, but I mean, the biggest problem for me is to, with the loading and saving functions. And, I learned from my mistakes with, by releasing games on, both Steam and Switch. So, I’m not quite sure where to begin. I mean, now that I’m making Sunset Moon, I feel like it’s, as I’m making the game, I really make sure that everything works before I move on, which is something I didn’t do before. I would just make all these things and then in the end, you want to tie that whole bag together and that takes forever. And then nothing seems to work because all these scripts and functions are not actually talking to each other.

[00:01:56] It’s just, you make them individually, which doesn’t help at all. I mean, that’s like the easiest part is to make a script that works. What’s hard is to make it work with everything else. So by, you know, I’ve made a couple of games at this point. So I tend to make sure everything works before I move on to the next thing. Sometimes it’s just super boring to work on the same problem for over a week, so then maybe you want to move on to something else and make sure you come back before it’s, before you’ve forgotten everything about the thing you made previously but in terms of testing my game, I guess, I feel like the previous games I, I would just sort of make them and then I played them through over and over and over again which is an awful way of testing your games in my opinion because it’s just, I mean you, you, you get so sick and tired of your creation by playing it over and over and over again, but I had no other way to do it because I didn’t know how to do it. But with Sunset Moon one thing I do now is to make a debug menu, which I, which keeps growing while the game is being developed.

[00:03:02] So in Sunset Moon, I have this debug menu now where I have all the items. So as soon as I add an item to the game, I add that to the debug menu. So while I play the game, I can just open the debug menu, pop in there and, and add that item to my inventory and see if, let’s say the function I want to try, let’s say I want to open this door and this door will only open if I enter this box collider and this item is in my inventory, so then it will automatically open.

[00:03:33] So now I can add in, that item to my inventory, I walk up to this box collider and it opens. And, you know, sometimes when you program, it looks great in script and you can even nowadays, you know, ask ChatGPT, does this work? And he will say, yeah, it looks great. And then you actually try the game. It turns out it doesn’t work because, whatever, for some reason, because ChatGPT doesn’t have all the context in this case. So I’m really picky with that nowadays when I test my games.

[00:04:00] I keep on testing and testing and testing and that takes I mean, every time you, you work with scripts and you add stuff and then you have, you hit the play button in Unity, in this case, it’s, it’s sort of, take some time for it to load the whole game. So that’s a, you waste a lot of time doing it that way, but I think in the long term, I, it will work for, in my benefit because it’s, I can see if it actually works. So that’s something I do nowadays. I use, I add a debug menu, so I have more control. What I do once in a while as well, maybe once every two weeks is that I build the whole project and play it in, on my computer. So not via the Unity editor, because that’s another thing I noticed. Sometimes things works in the Unity and it doesn’t work in, when you play the actual game as a game file, if that makes sense, the way you would download it from Steam, let’s say.

[00:04:56] So I do that maybe once every two weeks, play it on the computer to make sure everything works. And I just, because in this case with Sunset Moon, it’s such a big game in terms of all the things that happens in it, so there’s a lot of things to test and that’s quite hard to remember what to test. So here’s another thing I do nowadays, especially when I work with this big game, is that I, I keep a list of, I like to write them by hand, especially because then I can strike them out like the things that works so I write these lists once in a while and then I build a game and I play it on my computer and I sort of, from the start menu in the game, I erase all the save files.

[00:05:35] It’s on a clean slate, so to speak, start the game. I test a couple of things. I save the game, I turn it off, turn it back on to see, are those things still there or are they missing? Like whatever the case may be, sometimes they shouldn’t be there. Sometimes these things should still be there. So I make sure that all those things works and then I, if it works, I tick that thing on my list of things that I’m testing at the moment. I really wonder how many things there are to test in a game, that’s, I don’t know that, but that’s how I sort of do it nowadays.

[00:06:05] I learn from my mistakes by yeah, developing a couple of games, so I feel like I’m, I’m getting pretty good at it, but yeah, testing your game is not fun, but I do recommend doing it while you make the game so you don’t end up, I think KnifeBoy, before I rebooted the game, I developed that game for, it was my first game, so it was like two, three years of developing it.

[00:06:31] But then again, I learned everything from the ground up and watching tutorials and stuff like that. This is before ChatGPT, so I had to learn everything. But then I reckon it took a year just to tie that fucking game together, to make it work, that everything should, you know, communicate and all these things when you turn it off and turn it back on.

[00:06:49] And another thing I noticed, back then was, I knew where to go, so I wouldn’t even bother trying to break my own game. And that’s another thing I learned like you, you want to try to break your game, but yeah I didn’t know that back then. So these are sort of like a, a noob’s way of testing my games, but I feel like I’m getting better at it.

[00:07:13] I’m curious to see how you will rip my game a new asshole once you get your hands on it, but for now, I think that’s a good way, a good tip to people who start developing game. Make sure you make a debug menu and that’s not a fun thing to do and it feels like you’re not getting anywhere when you do it, like the debug menu I have, it took like two three days to make it as fluid and as flexible as it is right now, but it’s totally worth it in the long term because, yeah, it just makes it easier to test the game. So that’s my sort of, take on testing.

Otto’s Experience as a Tester

[00:07:49] Rune: So, Otto, you do this for a living, so how do you do it?

[00:07:53] Otto: Yes. So I think it differs a lot. I mean, just because of the context that when you do it professionally, often you do it in a team. So you don’t usually create the things that you test yourself. You rather get some feature handed to you or a list of features to test. And then you just send the results back to some developer and then somewhere along the line that gets fixed and then you maybe retest it again to make sure it actually got fixed and then it gets implemented. So the loop is basically like this, so imagine that you have some new feature or some new…

[00:08:41] Rune: Are you, are we talking games now, or is it like a software? Can you give a little bit more context on, for your job in particular, and then I think we can sort of transit over to games. But for your current job.

[00:08:53] Otto: Exactly. So for this particular, so I’ve, I, I’m gonna mash together my, both of my experiences, both my roles as testers. So the one was a, I think the easiest way to explain it is like, network management software, and then the other one is, media asset management system.

[00:09:20] So, handling, imagine like a Google Drive except for video files, for making movies, basically.

[00:09:29] Rune: But how does that translate to like, to me, the viewer, or is that, this is just some backend stuff that I will never see?

[00:09:38] Otto: So it doesn’t matter, I think, what the actual software is that you’re testing. The main point is, you have some sort of logic here, and then along the, the path no programmer is perfect, so you will always have some sort of logical inconsistency that will create bugs. That’s, impossible to avoid, I, I would say. So that’s always going to happen in software development. And then. you will have to, the easiest way to find those out is just to test whether the program does what it’s supposed to do or not, and that can be anything from jumping in a game to, you know, doing this whatever backend function in a network software.

[00:10:28] So that really doesn’t matter, I would say in terms of testing. So, so basically what you would do then, so you get some, either you want to adapt the software for a specific customer or you want to add a feature or, it could even be fix a bug. But then you have changed the software in some way and then you want to test that, so, it depends on, I think I’m going to get to a list of tests, the types of tests you can do after this, but, basically in this scenario we’ll just say that it’s a single feature that, you know, we’ve changed the login button, for example, to some other backend stuff and what you want to do as a tester is just to test that it actually logs you in you know, depending on different cases, so if you have this type of account or this type of account and, and so on.

[00:11:27] So just test that and then, you know, what you wanna do in that process, I would say, is to, as you said, try to break it as much as possible, like, you know, instead of, you know, some people try to just do the minimum and then be done with it, but to me, it feels a little bit like being a fireman, and you see smoke from a building and then, you know, you wanna check the, every floor, every room, to make sure there’s no fire, and you wanna, you know, open cupboards, whatever, to make sure there’s nothing that can create a fire, rather than just opening the front door and say, Oh, no fire, and then close the door.

[00:12:13] So what you wanna do is just, not just enter, you know, username and password, but you maybe wanna add some emojis in there, or you wanna, try to change some value to add like a million characters or something and what happens if you do that or, you know, really try to, because in the end you, you do want to check that it works, but you also want to check for things like, that can create unintentional bugs or it could be maybe a security issue for, the program if, you know, you can insert an emoji and then all of a sudden it spits out, I don’t know, code or something to show how it works in the backend. Something like that. So you just wanna try to, to do everything you can to make it misbehave. And if you can’t make it, make it misbehave, then it’s passed, basically. But, during that step, so you have found a bug, perhaps something that doesn’t work. Let’s say that if you have more than 10 characters than maybe the login page just goes to a 404 page or something. We’ll just make this a pretend case. But then, the important thing is just, you found a bug, but that isn’t the end of the work. So, especially when working in a team, you can’t just say that it doesn’t work and then send it back.

[00:13:42] What you have to do, because in the scenario of the programmer who sent you that thing to test. He thinks it works, sends it to you, and then gets back, it doesn’t work. But he doesn’t know what doesn’t work, why doesn’t it work, or what’s the symptoms, and so on. So what you wanna do is, I guess this doesn’t apply when doing it solo, but when working in a team at least.

[00:14:07] So what you wanna do is like Uh, the steps to reproduce the issue, so if he wants to deliberately find that bug as well, how should he do it on his computer? So, is it, you know, you go to the login page, you enter this thing, click on that button, whatever it might be. To do that, that’s super important, because if you don’t have that, then the other person might not find it.

[00:14:35] And also it might be something that’s specific to your machine. Maybe it’s you, you’re using macOS and it’s on, it’s only a bug on macOS, for example. So you have to include a lot of that information as well to make sure that nothing gets missed. And after that, it’s always good to have like screenshots or a video of you actually encountering the bug as well. So, and also, by doing this process, at least when you’re working on a team, then you will also discover yourself how reproducible the bug is. Because sometimes you will encounter a bug that’s, you know, it’s because of, I don’t know, let’s say it’s some weird thing where it’s, when it changes daylight saving or something, when there’s like a brief period of time when the time on the server and the time on your local computer doesn’t match, then there’s a bug or something. And then when you try to reproduce it, you yourself might be unable to do it. That’s happened to me, that you saw that bug, you’re absolutely sure of it, but you can’t make it happen again.

[00:15:47] and then it’s no use reporting it to the developer if you can’t really reproduce it, if it was just a you know, freak occurrence that just happens once in a blue moon.

[00:15:59] Rune: Hmm.

[00:16:01] Otto: And after that it’s, off to the developer and, in the best type of process, because it differs a lot from company to company, I would say how they do testing, but when it’s done really well, you sort of get attached to that bug as tester, so they fix it and then they send back the fix that they did so you can test that as well. So make sure that, did they actually fix the problem or have they introduced some new bug or something? So, so do, you get to be along for that entire journey from finding the bug and all the way until it’s fixed, basically.

[00:16:44] Rune: How long do you work as a bug tester on one project or do you work on various product at the same time, or is it one at a time? And in that case, how many, is it weeks or months or…?

[00:16:56] Otto: I will, I’ll get to that, I think, cause there is, this, different types of tests, and there’s a whole lingo of it that I didn’t know before starting with testing, but, so I will get to that, but, I think, yeah, that’s probably it for the, for the basic loop, you could say for testing so just get some issue test it and collaborate with the developer to, to solve it basically.

[00:17:27] Rune: I do recognize that I got video files from Nintendo and so on when they tested my games and they showed me how they found the bug or whatever.

[00:17:37] Otto: Exactly.

[00:17:39] Rune: I recognize some of it

[00:17:41] Otto: Yeah, I will mention some, some types of tests in this and I guess they’re gonna be not entirely ordered how you would do it necessarily in production, but how I think about it a little bit and they might be out of order.

Unit Test

[00:17:59] Otto: But I’ll start with the first one, which is one of the ones I think is most important, which is called a Unit Test.

[00:18:08] So basically if I, if I got this right, and I am sure that there might be people who can correct me on this, but a Unit Test, as I understand it, is that you test the individual unit of something, like the smallest test you can do. Basically, maybe you change a color in a menu, and then you test that that works, that individual thing, nothing else.

[00:18:34] It’s that single individual unit and usually, all the Unit Tests are done by developers themselves, regardless of the size of company. So, when you test something, then you, or when you develop something, you test it right away to make sure it works. And this is just to make sure that what you deliver to QA is at least somewhat functional, I would say.

[00:19:04] And also, for those who don’t know, QA is Quality Assurance, which is basically testers, if that, if, if you didn’t know that.

[00:19:14] Rune: I didn’t, I heard QA and I know it’s people testing but I had no idea what it actually meant.

[00:19:18] Otto: Yeah, so Quality Assurance, basically. So that, there’s also in the media business, there’s a similar thing called QC, which is Quality Control. But that’s more for testing that a video is produced correctly and things like that, but, things to know at least with testing. So, and I would say, in this, I’ll, I guess I will, talk a little bit about how, how it usually is done and how I would like it to be done. So with the Unit Test, I think usually people try it, so I’ve, I’ve, done a lot of testing where it’s, some sort of web product usually, and, unfortunately many developers try it on like their own machine and not on, hosted on a website or something like that, which means that it works on their computer and then they just send it off.

[00:20:18] And then when I’m supposed to test it, I try it hosted on some web server and then it works completely different because of, it could be everything from security certificates to, they’ve, you know, programmed it in a way that it works locally, but it doesn’t work when you connect to it remotely or something.

[00:20:39] So, sometimes, I guess, in my perfect scenario, the developer would test, like, everything themselves, like, as if there weren’t a QA department, and try to break it themselves, before they deliver it to somebody else. John Romero, who created Doom, he had a really good saying about this, that you are responsible for your own code.

[00:21:10] And he said that, when they created Doom, there was no QA department. You know, you as a developer had to just make sure it worked and he said that you shouldn’t allow anyone else to see your bugs is how you want to do it. So then that you test it rigorously and then when you see, if you do send it off to a QA, a testing department, then, you know, the quality is really good and they, you don’t have to, you know, spend a lot of time sending reports back and forth for something that could be fixed in like five minutes if the developer could have tested it right away.

[00:21:52] He also said something like, what was it? Sorry, I lost my train of thought here. My god. and now, there was another quote also, from a guy named James Bach, which is, he’s also sort of famous for testing, and he said that if you don’t test like properly, as the developer, the, it becomes this sort of weird economy of bugs, in companies, that’s just counterproductive to everyone.

[00:22:35] So, the developer can, you know, create the software fast and then send it off because then they have, they get praised because they create software fast, basically, but then it’s full of bugs because they haven’t taken responsibility for it to, to, to fix it. And then, the, the QA department, the testers, they get a lot of creds because they find a lot of bugs, that they can send back to the developers.

[00:23:02] So they get praised for, you know, finding all of the bugs. And then there’s this back and forth where none of them are doing a good job because, you’re just creating this economy of bugs where it’s, you know, becomes a positive thing instead of a negative, which it should be. So, you know, being responsible for your code, I think, is how I would like it to be, but unfortunately, it’s not always that way. So Unit Testing and testing your own things, very important.

[00:23:38] Rune: Interesting. Yeah, that was, I kept, something else that popped into my mind when you say you’re creating an economy on something bad. I wanted to bring up another example, but I can’t quite remember what I was thinking about. Whatever.

[00:23:57] Interesting.

[00:23:58] Otto: It’s a, it’s a day of confusion in this podcast.

[00:24:01] Rune: yeah. I feel like it was something about, environmental stuff, listen to some of that stuff and, whatever.

[00:24:12] So yeah, now you talked about how you fix the problems, how you wish it worked. Did we touch on the question I interrupted you with halfway through, how long do you…

[00:24:24] Otto: Yeah, so how long it takes. Yeah.

[00:24:27] So on one project, so for, in my experience, so I assume project means product in this case. Or, you know, because the, the things that I worked on has been just continuous project that exists to this day, so it’s just been, you know, now we’re at this version and then we’re you know, fixing the bugs in this version, so it hasn’t been like, you ship this small software and then it’s done and then nothing more. It’s been a continuous thing, I would say.

[00:25:06] Rune: So you’ve been working on that for many, many years.

[00:25:09] Otto: Yeah, you know, in some cases, yeah, it’s been, you know, you start at somewhere in the software and then, at the end of it when you change jobs, then you’ve worked with it maybe for two years and testing things and seeing it grow and, and all of that.

[00:25:26] Rune: Do you feel like you become a part of the development or do you like, do you feel like you’re working on that particular thing? If you worked on Netflix like that, bug testing Netflix, would you say that you worked at Netflix or that you helped developing it? Or do you feel like you’re a third party and just, who happens to know a lot about it?

[00:25:45] Otto: I think, you know, for, I think some people could probably say that, but I think in my case, the teams and organization structure hasn’t been that way, really. It’s been more like, sometimes at least it feels like you have the development, which is the core of, that’s the product basically, and then you have, you basically, you add the testing later on almost, so then it’s, it’s sort of separate, rather than how you would want it to be that testing is a part of doing the whole product, I would say. So, yeah, but that’s Unit Tests that I talked about, so I think I can, go, to the next type because then we’ll get into more, more about how this works and, and why you should test and when you should test and so on.

Smoke Test

[00:26:48] Otto: So, another, interesting test, that I really needed to get translated for me when I heard it the first time is Smoke Test.

[00:26:58] So, if you would guess, do you know what that would mean?

[00:27:02] Rune: Oh, maybe overclocking the machines and see if they start burning?

[00:27:10] Otto: No, actually, it’s not a Stress Test as that’s called, so a Smoke Test is actually, it’s more logical when you hear the explanation…

[00:27:21] Rune: Oh, can I guess one more time?

[00:27:23] Otto: Yeah.

[00:27:23] Rune: That you start using something and then you leave it as if you’re going on a smoke break, and then you come back and see if it’s still running? Is it wrong?

[00:27:32] Otto: Almost, I would say, but I would, maybe that’s also a little bit more Stress Testing, I suppose, but I would, the way I would explain it, it’s like this, imagine the fireman analogy again. You’re testing to see if you see smoke anywhere, and if you see smoke anywhere, there’s probably a fire. So that’s what you do with Smoke Tests, that you do checks of the most vital things to see, does the, the most important things work, and if not, there’s probably some issues that needs to be fixed.

[00:28:12] Oh, here, by the way, I remembered now what John Romero said about creating Doom. So he said this, before doing anything else, before implementing new features, you should always fix bugs first. If you know about the bug fix it before you do anything else because if you don’t then you will build your software on a faulty basis or a faulty foundation.

[00:28:38] So then you create a new feature that relies on the this bug so then when you fix this bug, this new feature gets a bug, because that relied on the system working in a way that enabled this bug. So if you instead fix the bug first, all of the bugs that you know of, and then implement new features, then you won’t have this, again, sort of economy of bugs where you just, you know, it starts breaking down because, this bug fix destroys this actually useful thing.

[00:29:16] Rune: Interesting. Yeah, I actually think I have one of those bugs like, when I started developing my game, I watched, YouTube tutorials on how to make it and stuff like that. So it was sort of like based on keyboard and mouse, but problem with me is that I, first of all, I can’t even play with keyboard and mouse barely.

[00:29:34] And I am not really interested in developing it for PC only. So I always have in mind that it’s going to go to console or at least that people should play with a controller. So for me, there’s a lot of mouse, the mouse is acting weird in the game. So what I’m going to do now is just sort of delete anything that has to do with the mouse and make it work just with the keyboard.

[00:29:58] And, once that’s done, and this is something I’m going to do before I hand over the game to you and eventually the demo, basically I just wanted to be, be, be playable with the controller first and then I think I’m going to reintroduce the mouse because, I feel at this point there’s like a 75% made for controller and then 25% mouse and the mouse part of the game messes everything up. Every time you click somewhere on the screen, it’s just like, I don’t know. It is as if it’s, you can’t interact with anything anymore. Because you clicked with a mouse somewhere, so it’s like thinking you’re using a mouse. But anyway, that was exactly what he mentioned there. I feel like I’m building my game on that bug or something that I don’t even want in a game.

[00:30:45] But obviously at some point you should be able to play the game with a mouse, but whatever.

[00:30:51] Otto: Yeah, interesting. Yeah, exactly. So, that’s also something that I, I think many companies, could learn from, that usually people wanna, you know, produce stuff fast and then you get this where it’s built on a faulty basis and then to fix this bug, there’s three other bugs and so on. So, you know, you could have done it just a little bit slower in the beginning and then, you know, have it, a more stable process.

[00:31:23] But, it’s what it’s, but anyhow, Smoke Test. so then, what you do basically is, when you release a new version, say that, you know, now that we, we have, maybe version 2.0 of this game, for example. So we have all these new quests, and we have implemented this new crafting system, and we have this new, you know, collapsible buildings or something system that’s, everything’s new.

[00:31:55] So when, what you want to do then, for a Smoke Test, is that you want to do the most vital functions first or only, I would say in a Smoke Test to make sure that everything works. So you try, for example, just moving, you try jumping, you try logging in if there’s something like that, you try doing maybe the most vital, you know, if you want to, if you, you have implemented a crafting system, you try just crafting one item to see that it works at all and you leave it at that. And then, you know, surprisingly, some of these things, really, at least, in the products that I worked on didn’t work, when you do Smoke Testing that you expect them to.

[00:32:44] So there is actually, when you introduce this new version with hundreds of changes, then all of a sudden it doesn’t work to change your user profile picture or something and then you just report that. So then that just, it’s a fast way to discover if you know, some, if the new version has a lot of bugs or it’s sort of taking the temperature on the new version, how many bugs or whatever it might be, that it has. Let’s see, what else?

Regression Test

[00:33:22] Otto: Regression Test. Do you know what that is?

[00:33:26] Rune: What did you call it?

[00:33:28] Otto: A Regression Test.

[00:33:30] Rune: Regression. No. No idea. I don’t even know what regression means.

[00:33:36] Otto: so it’s the reverse of progression. So, I, I’m not sure how, so the way that I’ve done it, at least, is that, basically, I don’t know if this is the official version of it or the official way to do it but the way that I’ve done it is that when you release a new version, after doing the Smoke Test and making sure that everything works, then you do basically a full test of all of the features in the entire software.

[00:34:13] And this is, we’ll get into that as well, but all of these things that I’ve mentioned now is part of what’s called, or I think all of them is, at least, called, Structured Testing. We’ll get into that later, but for those who are wondering. Anyhow, so what you do, basically, you’ve released this new version, crafting system, new login page, whatever it is.

[00:34:37] And then you have a list of all of the functions in the game. So, it could be like, you should be able to press spacebar to jump, you should be able to press the WASD keys to move, you should, this menu should do this, whatever it is, you just list all of the, the functions in the game, basically, and this is the list that you always have when you develop new features, you add stuff to this list, so then you go through this list and test everything basically, so, as I said, regression is the opposite of progression. So progress is adding new features and having more functionality and stuff, but regression means that when you add new stuff, then stuff that used to work, stops working. So, it used to be that you could log into this game, but then when you added the crafting system, all of a sudden, you have a regression, you can’t log in anymore, or you can’t move, or you can’t jump, or whatever it might be.

[00:35:45] So, Regression Testing is testing to make sure that after you’ve done an update, you test everything but the update, essentially. So you test everything else to make sure that the thing that you implemented does not fuck with anything else that, that you’ve done previously, that used to work without problem.

[00:36:09] So just go through all of those and just make sure nothing has regressed.

[00:36:17] Rune: But that, it sounds like, it was sort of like what I talked about the way I try my game. I have this list and then I tick them off if they work or not. And I do this, like I said, once every two weeks. That’s cool that that’s something you professionals do too. I thought it was being a genius coming up with that myself, but I was thinking like now that I, now that you said it and it’s like that sort of approves my, approach.

[00:36:43] I just made a note that I’m going to make a list of all the things in my game. Cause what I tend to do is like, I write just a couple of things that I want to go back and check. And now I’m thinking, now that you talked, I was thinking like, well, how many functions do I have in my game? I don’t even know if, if I could, like what is a function? I can’t even think about…

[00:37:04] Otto: Yeah.

[00:37:04] Rune: That, so that must be really hard for, yeah, but, but I guess you get a list of the things that they want you to look into in that case.

[00:37:13] Otto: Yeah, exactly, so I would say that a function in this case, I would call it like, in the context of testing, you would say the smallest thing that you can test. Like, like a really big thing would be, like, the game works. That’s a really huge thing to test, and it depends on so many factors. But, a really small thing is, for example, you can jump with the space key, or the, you can move with the, with the arrow keys, or you can, if you click on this thing, it does X, for example.

Why Don’t Programmers Find All the Bugs Themselves?

[00:37:49] Rune: Well, if those things are like, surely they test them themselves or, or, or is that like sort of like a, one thing I noticed with like proper programmers, they’re really good at it. They can just see that everything works in code, but it doesn’t work in the system itself or in the software itself. It’s almost as there is a little bit of disconnect. I suppose that’s when where that kind of bug can come in that press enter and obviously he should jump, like surely they’ve tried that but I can imagine that in that case I’m sort of answering my own question, but I can then imagine that they send you a new update for a game and it was just the programmer who looked in the codes, yeah, it works, but it actually didn’t work in the, the real game because some of those functions you just mentioned they seem like the most standard simple things that would take two seconds to try before you send a an updated version.

[00:38:45] Otto: You would think, but, not everyone, but also there’s things such as, for example, it’s called race conditions. So basically imagine two functions racing to do something. So, for example, we can take a scenario where an NPC that says, you know, for example, you talk to him and then he’s going to say, oh, it’s this person’s birthday today or something.

[00:39:16] And then to do that in the backend, he, you talk to him and then immediately you get the he’s, there’s no wait period, and he says immediately what he’s supposed to say about somebody’s birthday. But to do that in the back end, then maybe, hey, he has to go into the database or something and list out all of the NPCs, see what birthday you have, and then come up to the right, and maybe you have, 3,000 NPCs in the game or something.

[00:39:46] And then he has to list the thing before, it gets displayed on screen, because otherwise if you know, the displaying on screen happens, then, you know, today it’s, and then it’s just an error code’s birthday today. So that occurs, and that’s just, the, the issue there is that you know, the one thing is faster than the other thing.

[00:40:06] And that happens sometimes, but maybe for the developer, he has a really fast computer, so it always, always happens in, you know, fast enough, maybe. And that’s something that, you know, it works for the, the developer every time, no problem, but somebody on a slower computer, for example. So that’s, you know, since I thought about, you know, a while back, like, why do bugs happen at all? And, I think it’s, I come, came to the conclusion that software is logic and nobody can have perfect logic all the time and think about all of the cases and everything about how a computer works, so there’s always gonna be bugs no matter what future we live in because all software is built by human logic in the end.

[00:41:03] You know, we can come to a larger discussion about AI and that…

[00:41:07] Rune: Yeah but if AI is learning from us, like I was going nuts the other day because I had left a, I don’t know what that symbol is called. It’s a dot and then there’s a colon underneath it, semicolon, is that a thing?

[00:41:22] Ah, so a list of things, I can’t remember what it was, but it was a list of things. And the last one in this list should not have that colon, it should just be nothing there.

[00:41:35] And I got this error message, I showed the ChatGPT the error message, and then I showed him the code and he couldn’t figure it out either. He thought it looks great to me, but it was just, you know, you had to get rid of one of those. I don’t know it’s you know, it’s just weird because sometimes he seems to ChatGPT seems to think like a human and he don’t seem to find the bugs either quite often, I think.

[00:41:59] Otto: Yeah, yeah, yeah, exactly. So I think just testing is important because, and also to do it like in the scenario where it’s the, the closest to the real thing possible. So for example, doing, something that I, learned when I started with game development was that I, you know, it worked great in the Unity editor, but when I built it to, like a, a build, then all of a sudden there was all of these bugs that didn’t happen because in the build process, it works differently after it’s like an exe file than when it’s in the Unity editor. So, trying it the closest to the real thing possible, is also important because of these things, I think.

[00:42:52] Rune: Yeah, man, that’s I feel like I got another approval on my approach, how to fix my games.

Stress Test

[00:42:59] Otto: Yeah, cool. and, there’s a, a last test that I, type of test that I think it’s also important to, to try, when doing your game, which is a Stress Test, which is what you mentioned. So, that could be, for example, it could be something like, you, instead of having one NPC in a, a place you have 3,000 or something and see where the limit is, how much, does it work, or is there some sort of bottleneck after you have maybe 4 NPCs, maybe it starts to lag and that’s just some bug that, that is creating, it’s unnecessary work or something, for example, so that could be important, but a Stress Test, as I understand it at least, could also be something else where instead of just having a lot of stuff, you try to do stuff fast, for example. So it could be instead of, you know, you go into this menu and click buy on some item, but instead of clicking buy once, you just hammer the button, the mouse button as fast as you can, like to try to get, so you can get like, more than one click per frame, basically.

[00:44:17] So, if that perhaps makes it so that instead of buying one thing for ten in-game dollars, you buy maybe six or something because you clicked fast enough or something like that, or you could, you know, switch from one item to the other, or maybe, you know, stuff like this creates, can create bugs where it’s duplication of items, or it could be other things.

[00:44:44] So that’s also important to test to always just try to click, things there, there was actually, I think I have to link it, it’s a really fascinating talk, about testing from James Bach, who is this famous tester. but he is, you know, as I said, a really famous tester, done this, for maybe 30 years or something, really good at it.

[00:45:11] So, he’s done things like where he’s on vacation and then he sees like this, you know, computer on, uh like a screen on the hotel lobby, and it says like, Hey, welcome! Here’s the, like, tourist information. Then he just goes and does things like this stress, stress test and stuff. And then he just clicks everything and, and eventually he gets access to the computer and then he can just put up whatever he wants on it, because of testing. So, really fascinating. yeah.

Structured Testing vs. Unstructured Testing

[00:45:47] Otto: And I do have, like a last piece of, like testing things or methodology, I guess you could call it. So the thing we’ve talked about until now is structure testing, which is what I’ve done most of the time. So where you basically, you get a list of these are the functions and this is the things you should test.

[00:46:14] And then, you know, you go through the list and then you’re complete and then you redo it a little now and then. But there’s also, let me double check the, the name of it. let’s see here. Yeah, I think it’s, Functional Testing, I think is the official name for it. So Functional Testing instead, somebody have to, to I don’t or maybe it’s Un, Non-Structure Testing.

[00:46:48] Not sure. Anyhow, somebody can fact check me, I will, if I find the method, what it’s called, I’ll put it on screen here. Anyhow, so the thing I’m thinking about is this so instead of having a this list of things to, to test, instead, you do it by intuition each time. So instead of having like these are the types of functions that we have.

[00:47:16] These are the things you can do. Instead you do it on intuition. So instead of going through the list you just go into the game or the software or whatever it is and try to find the functions that exist in the game or the software. And try to test them based on what you find. And the advantage of this is, this is something that I would like, would have liked to have tried more in my professional life.

[00:47:42] But I think that big companies especially like Structured Testing because it’s, you know, you can tick that box that you’ve checked everything, but that’s also one of the weaknesses with Structured Testing, because if you haven’t written down or created a test for something or you haven’t written down, this thing can be done with this, you know, then maybe you will miss a bunch of functions and, things you can do just because you have this structured list and it’s supposed to be.

[00:48:13] And you also sort of can fall into the trap of writing how it should be tested. Because if you do that, then you will say that, you know, it’s supposed to be do, done this, and then you maybe have a bug, but you haven’t tested in every possible way, or you haven’t tested it by thinking outside the box, so then you will just leave a lot of bugs for customers to find.

[00:48:37] So, doing it unstructured is, I think, more interesting, and I think something that I would like to try more. so then basically just go into, to the game, try everything, and just see what happens, and then try to, to find everything, basically.

[00:48:57] Rune: Mm. Interesting.

[00:49:00] Otto: Yeah.

[00:49:00] Rune: It’s sort of, sounds like I know, I don’t know if it’s a thing, but my brother mentioned that, when he worked for some game company in the past, like they would sort of send a instructed review almost to outlets and they will just sort of read the review that the developer wrote about their own game and then just sort of, in a way you’re you’re influencing the reviewers or maybe it’s not a review, more like an article about the game, so you can sort of control the message in a way.

[00:49:32] Which, I mean, they definitely have their upsides if you, but, but best is if they play it themselves and see what they think.

[00:49:44] Yeah. Interesting.

[00:49:46] So when you play Sunset Moon, it will be a non-instructed.

[00:49:50] Otto: Cool, that’s, that’s the way I like it. And I think, you know, a lot of the things that I think can be fixed in testing that is suboptimal is that people really seem to be afraid of finding bugs when they should be afraid of not finding bugs, I would say. because, you know, it’s maybe it will set you back, you know, you can’t keep your schedule because you found a lot of bugs.

[00:50:23] So then consciously or unconsciously, you sort of structure your work around testing to not find a lot of bugs, or something like that. And then the result is that either you don’t find bugs and then you have created the software fast, I guess, and then you keep some sort of budget. but then that’s, the risk is that your software will suck.

[00:50:50] The customers won’t like it or it won’t function and then you will have, a big problem where this bug might be everything from performance issues to this thing doesn’t work or it could be a security issue even that, you know, a hacker could get all of your customers data because this bug that you wouldn’t want to test for or, you know, so I would say, to, anyone who thinks about the testing or developing that it’s more expensive to have a bunch of bugs in the thing that you’re creating, rather than taking the time to fix them, basically.

[00:51:36] Rune: I recognize myself with a KnifeBoy 1 there, like I, I would, it was almost as if I knew there’s a bug over there so I’m not gonna go there. So it’s fine. And then and sometimes I would reason that well, no one is gonna go there anyway. But I did learn that when I took to Taipei Game Show like how to watch people play and it was just shocking how some people play games. It was just button mashing all the time, and I would be like what is going on? Why are you…? I didn’t say it to them, but I was thinking like, what is, what are you doing? Like, what are you trying to do? You’re just standing there jumping like a crazy person, like what’s going on? But that was interesting though to see how other people play your game, because it’s, it was not the way I played it.

[00:52:19] That was a lesson learned. but I was thinking, I wanted to wrap up the episode because on our end the internet is lagging like crazy. But I don’t think the listeners or viewers will see that.

Real-World Applications of Testing Skills

[00:52:36] Rune: So if I may, I wanted to ask one final question before we wrap up this episode and that is work damage in real life.

[00:52:47] And now you have to be honest, how do you look at your own life when you’re walking around there with this problem solving mind after work? If your toaster breaks or if you’re I don’t know, you’re fighting with your girlfriend or something, do you approach these things as a game tester or do you, you turn it off?

[00:53:09] Otto: Well, speaking to a partner, I don’t think it would be wise to think like a tester. You know, I heard a joke once that, you know, a developer, he goes into a bar and orders, 1 beer, and then he says, job well done, you can order beers. And a tester goes into a bar and tries to order -1 beers, and orders, 3 beers at once. And then he orders, 4,000,000,000 beers, and then he orders 🙂 beers, and so on. and, speaking backwards and stuff, I don’t think that would, make stuff better. yeah. But, in reality, I think, yeah, I’ve, I have, gotten a lot of use for it, you know, especially, you know, creating games and, really, I’m trying at least in, in the time I have to spare, I try to break it when I create a new function and also there’s been also, things where I’ve programmed, programmed stuff myself in, in my work and then try to break it, there was one, really big project where I had to work overtime on, on a weekend and you know, I worked maybe, feels like 36 hours in a row almost but then at the end of it, I tested it like crazy because you know, at that point I did not want to, I worked this much and then there was this deadline.

[00:54:43] So I did not want to ruin the whole process. So I tested everything and tested the, you know, with that it was, some sort of software, but I tested everything with the crazy characters and backwards and negative numbers and everything, so, yeah. I have gotten a lot of use for it. Sort of a fun story.

[00:55:06] But, so I was getting back from a trip with my brother and a friend and, we were waiting to, I think, return a rental car or something, but we couldn’t get the, so it was a sort of a automatic, car kiosk where you just typed in your registration number and then the slot opened and then you leave the keys and then that’s it.

[00:55:32] But it didn’t work for any, for some reason, so we were just stuck there and just wanna move on. So then, since we were bored, I sorta did that thing that I remembered this other guy did, and just clicked around the screen. It was this, you know, it’s just, it looks like a ATM machine, basically, there’s just this big metal box, and then it’s this touch screen in the middle, so I clicked it a little bit, and then eventually I got to Windows.

[00:55:59] So I can open, like, the command prompt and stuff. But then, my brother and his friend stopped me from doing the thing because they thought I would get in trouble. Or something. So I, I’ve, I have been able to do some stuff in the real world.

[00:56:16] So, yeah. I guess I’m a little bit damaged.

[00:56:22] Rune: Yeah, it sounds like it.

[00:56:24] Otto: Nice. Yeah, well, I think, that’s probably it for, for the testing episode. Hope you got some…

Reading Comments

[00:56:34] Otto: Oh! Before we end the episode, actually, we have, now that we have, two comments actually on a video, I would actually read some comments so that you people out there understand that you can comment and leave voicemails and we’ll, we’ll see them. So, the comment we got was on episode, 55, which was “2023 in Review”. So, let’s see, that’s when we talked about what we did during 2023. So, I guess we talked a lot about our game development and stuff. So, from the let’s see here, her name is “sietherine”, I guess, left two comments, so she said first, and I guess about your game and I had, Red Colony in the background, so I think that’s the game she commented on, but she said “Cool looking game. The art is great and really consistent.”

[00:57:39] Rune: Oh, cool. Nice.

[00:57:43] Otto: Yeah. She always said, also said, about Unity, I think. so she said “I am still on the Unity train here, but I am on a very old version. It’s frustrating at times, but I’m, I am air gapped and don’t want the horrible choice of updating and terms make me additionally hesitant.”, so I guess that’s about you know, that person is on an old version of Unity and it sucks but there’s no choice other than updating and you don’t want to do that, which I get.

[00:58:19] Rune: Yeah, I know that feeling. I hate updating. yeah. Well, cool. That’s fun to hear. Keep, keep riding that Unity train. I think it will be fine in the end.

[00:58:35] Otto: Yeah, nice. I will, I will reply to her there actually. Rune votes Unity, I vote Godot.

[00:58:47] Nice, Yeah, so that’s, comments left on Rumble, actually.

[00:58:53] Rune: Look at that.

Episode Outro

[00:58:58] Otto: Yeah. Okay, so yeah, I think, that’s it for the episode. Thanks so much, sietherine for the comments. Please leave more, Yeah, I think that’s it. Thank you so much for being my amazing co-host. Thank you for listening. Leave a comment or a voicemail if you have anything to say to us. And if you have any other testing tips or stories, please leave them in the comments.

[00:59:27] And I’ll link the stuff we talked about as usual in the description below. Have a good one!

[00:59:37] Rune: Bye!