Tag Archives: card

Card Importance Testing in the simulator

This is the third installment in a series of three posts about the new features present in 1.2 which has recently become available on the app store.

The first two discussed the per-participant statistics and HELO (Hearthstone ELO), and this final post will focus on Card Importance Testing which is another new feature.

One of my goals in writing Deck Simulator for Hearthstone was to create a tool which would allow adjustment of existing decks to improve their overall effectiveness, or simply put, win rate. While it is possible to create a bunch of variations of a deck with different card variations and test them, doing so can be time consuming and tedious. Because of this, I decided to try and add a feature which would help guide how to modify a deck.

Because of the large number of cards available (a few hundred including neutral cards), rather than focusing on what card(s) to add, I decided to focus on determining which cards to remove. After all, since all legal decks in Hearthstone require 30 cards, if you want to add a card you must first remove one. (Note: Below you will see a reference to using a  28- or 29-card deck, which is a special feature only present in the simulator)

Imagine you knew which of your cards really were doing the lions share (helping win games) and which were just going along for the ride. This would be one major step to finding how to improve your deck, since you could replace those cards with other cards that might have a greater positive effect on the deck as a whole.

I’ve added this type of Card Importance testing via a single button available on the “Configure Simulation” screen, aptly titled “Test Card Importance”. To use this, simply choose a AI Player and Deck, and click the button. This will add several participants (AI + deck combinations) to the list of participants:

  • Two participants where the selected deck is added, unchanged.
  • One participant with a modified deck where all cards of a certain name are removed from the deck. This can be one or two cards depending on how many were in the deck, and these holes will not be filled with other cards. For example if your deck contains two instance of “Arcane Missles”, there will be one participant with both of these cards removed (down to a 28 card deck), and whose name will be of the form :(removed XXX) > original deck name (due to space considerations the player being used will not be shown in the name as it usually is).

The reason for the first unchanged deck is so you can play each deck missing one of the card names against the unchanged deck, and compare to see whether the removal of thiscard decreased or increased the overall win rate. Alternatively, if you want to compare against another deck you can add that deck first before clicking the “Test Card Importance” button, and all modified decks will play against that instead.

(The reason for the second unchanged deck is to help guide the right number of games per match. More on that below.)

To illustrate a real example, I did the following:

  • Selected “Evl. AI (strong)” as the AI Player (or just leave it as this is the default)
  • Selected the deck “Evolved Paladin 215” and clicked “Add to simulation”
  • Selected the deck “J’s Mage Deck” and clicked “Test Card Importance” (blue button on right middle)
  • Made sure Simulation Type was set to “All play first”
  • Changed the slider “Matches per pairing” to 1001 (maximum value)
  • Clicked “Start simulation”

Here is an excerpt from the results I got:

  • [1921] Evl AI (strong) < Evolved Paladin 215 < Win 80.9% of 23023>
  • [1121] (removed ‘Arcane Explosion’) > J’s Mage Deck <Win 24.2% of 1001>
  •  …
  • [1062] (Unchanged) > J’s Mage Deck <Win 19.4% of 1001>
  • [1018] (Unchanged) > J’s Mage Deck <Win 19.3% of 1001>
  •  …
  • [883] (removed ‘Ragnaros the Firelord’) > J’s Mage Deck <Win 15.4% of 1001>

The most important thing to know about reading these results is that they are backward from what you might usually expect. For example the fact Ragnaros is on the bottom of the results (with a 15.4% win rate versus an 19.3-4% one for the unchanged decks) actually means Ragnaros is one of the main contributors to this deck since without him it’s performance falls significantly. Conversely, the deck actually performed *better* with two Arcane explosions removed from the deck.

So, from this little experiment we learned some information about one card that we probably wouldn’t want to remove from the deck (Ragnaros) and one that is a good candidate for swapping out for something else (Arcane Explosion).

The reason for the two unchanged decks is to determine how stable the simulation results are based on the number of matches per pairing. Ideally, since these decks are the same, you should get nearly the same win rates, and in my simulation with 1001 matches per pairing you see only a 0.1% difference, which means the results are generally valid. However if you were to run a solution with only 100 matches per pairing, you might get a variation of 5-6% between these two win rates, which tells you there isn’t enough data to properly capture the decks’ strengths. If you have the time and can wait for the simulation to finish, more matches is always better.

I choose “Evl. AI (strong)” here instead of the stronger “Evl. Planner (med)” because it’s significantly faster, but once I narrowed down to a few candidate cards to remove I suggest running another test with the planner type to further validate things. Also because in “All play first” games the statistics tend to be a bit less stable,  you may want to run one or more simulations with “All play all” to see how your modified deck(s) do against a few different types of decks. For these follow up runs, you can simply delete the participants  you don’t care about and re-run a different simulation as needed.

Remember, each time you run one of these simulations you should expect to get slightly different results just because the random number generator will be a bit different. But if you have enough matches per pairing, you shouldn’t see the win rates change more than few percent points between runs. You’ll find a clump of cards which are consistently on the top (the weak ones) and another clump that are consistently near or at the bottom (the strong ones).

Deck Simulator for Hearthstone is currently available on the Apple App store for $0.00, and you can see the details of it here:

https://itunes.apple.com/us/app/deck-simulator-for-hearthstone/id914214349?mt=8

Deck Simulator for Hearthstone V1.2 Released on Apple App Store!

I’m very happy to announce Deck Simulator for Hearthstone‘s 3rd version (1.2) is now available on the Apple App store!

For this release I decided to take a break from adding new cards and focus on adding a few features that will make the tool easier to use and more productive.

For example, I’ve added a very cool feature that allows you to evaluate to what extent each card in a deck is contributing to it’s overall strength, and also added a rating system for decks to easily compare them (HELO = Hearthstone ELO).

For full details of the release please see the app’s page here:

https://itunes.apple.com/us/app/deck-simulator-for-hearthstone/id914214349?mt=8

The app is currently completely free, with no ads, though will only be this way for a limited time.

I’ll be writing some more posts in the feature highlighting some of these new features and discussing how to use them.

RNG, Hearthstone, and the simulator

The other day I was speaking with a friend and when I mentioned Hearthstone he immediately said “I don’t like Hearthstone very much because there is too much RNG”.

RNG, which stands for “Random Number Generation”, is defined by Hearthstone Wiki as “the unpredictable outcome of a situation featuring a significantly randomized element”. In other words, the more RNG there is, the more possibility that your win or loss was a result of random odds as opposed to purposeful strategy and good deck construction. Chess would be an example of a game with zero RNG since it’s completely non-random, and a slot machine would be an example of a game that is nearly 100% RNG. All card games that involve shuffling have RNG to a certain extent.

I would agree that there is a major element of randomness in Hearthstone. After all, every game you start with a hand that is random drawn from your 30-card deck, and many card effects have a major random element. For example, Arcane Missles, which deals 3 damage randomly split among characters. Or one of my favorite cards, Ragnaros the Firelord, which deals 8 damage to a random player or minion.

Surely, there are things which offset the randomness of the game, such as the redraw system, and a strategic player who calculates correctly can make choices where the outcome of random effects is most likely to benefit them – for example killing off certain minions and letting random effects take down the remaining ones.

Having said that, I would argue that the result of a single game between two decks in Hearthstone doesn’t say too much about either the strength of the decks or the ability of the players. I actually think in this case the number of turns before the game ended more accurately predicts how close the decks are in strength. The more turns the loser managed to stay alive, the closer the decks are.

The realization that Hearthstone contains a good amount of RNG is another thing that motivated me to create Deck Simulator for Hearthstone. Surely playing a handful of matches with a real opponent gives you some idea of the strengths of each deck and player, but what if you could to run 100, even 1000 games in a matter of seconds or minutes?

That is exactly what the simulator allows, and even if the AI players are not exactly ‘ideal’ in terms of playing the perfect move in all situations, as long as they play reasonably good moves the win/loss ratio between two decks should give a good indication of which deck is stronger.

As I mentioned in the last post, the number of games played between each to pairings during a simulation is determined by the “Matches per pairing” slider, which defaults to 20. The reason I set 20 as the default is I felt it was the lowest value that gave some a good estimate of the relative deck strengths. Any less than that and RNG takes over and the result isn’t too statistically meaningful.

The maximum matches per pairing which permitted in the simulator is 1000, which should give a very accurate idea of deck’s relative strengths. However, depending on the  AI players you have chosen and the speed of your device, this could take quite some time (an hour or more).

Therefore, effective use of the simulator requires picking a good number of matches per pairing. Here are my general recommendations:

 

20: Use this if you want a very, very rough estimation of the decks’ strengths

100: Use this for a rough estimate that carries more weight

200-300: A good middle ground for simulation accuracy vs wait time. This is the most you’ll need for most situations.

1000: Only needed when you really want to be sure the relative strengths of decks, such as when you decide to craft new cards to make a new deck or improve an old one. I recommend limiting the number of participants to 2 only a small handful to avoid long simulation wait times.

 

While there are several different ways to measure statistical reliability and I may add some of these in upcoming releases, there is a very simple way to check how accurate your results are. Do a full simulation run then take a screenshot to capture the top of the results table. Then re-run the simulation (via the green “Restart simulation” button on the top right part of the simulation screen) and take another snapshot. You can do the same thing again two or three more times if you like.

Then compare the results of each simulation run. The less number of matches you use, the more variation you will see between simulation results. If you see one run where a certain deck is #1 (highest on the results list) and another where it is #5, you know you need to increase your number of runs and try again. You should aim for a number of matches where the general ranking and order is mostly consistent, but the win ratios can be a few percentages different between simulation runs.

I’m curious to hear about your experiences using the simulator and what sorts of parameters and simulation types you’re using, so please feel free to comment to this post!

(Just for the record, I don’t think Hearthstone has too much RNG, but rather a reasonable amount of it given it is a card game)

References

http://hearthstone.gamepedia.com/RNG

http://hearthstone.gamepedia.com/Random_effect

(Featured image of Hearthstone card “Ragnaros the Fireload”, taken from Hearthstone wiki (http://hearthstone.gamepedia.com/Ragnaros_the_Firelord) but is trademarked and copyrighted by Blizzard Entertainment)

Why create a Hearthstone Simulator? The story behind why.

Before I spend time talking about how the simulator works and how to use it effectively, I’d like to give a little history as to why I was motivated to create it in the first place.

A few months back, I was pretty into Hearthstone and trying to figure out how to adjust my decks to perform better, and how to build good decks from scratch. I knew some of the basics of deck construction (like a proper mana curve and trying to choose cards that compliment one another), but actually putting that into practice was tricky.

More than that, the bigger problem was that actually testing any of my deck-building ideas took so much time. I had to first decide what cards I wanted, then win enough games to buy cards, dust the ones I didn’t need, and then make the cards I wanted. Then I had to play several games with real opponents to get a feel for whether the deck was any good.

To top things off, since the ladder system in Hearthstone adjusts to your wins and losses, it’s hard to get a real absolute measure of your deck’s strength. For example, if you loose enough games you’ll eventually be put against players who have bad deck-making skills and bad tactics. Just because you win those games, it doesn’t mean your deck is any good.

So one day I thought to myself – what if I could estimate whether one deck was better than an other by using a simulator that mimicked the rules of game? 

One reason I enjoyed and immediately got addicted to Hearthstone was that it was quite similar to Magic: The Gathering, which is a (physical) collectable card game that has managed to maintain some level of popularity since its creation in 1993. While it shares some of the basic concepts with Hearthstone, such as ‘mana’, ‘spells’ and ‘creatures’ (equivalent to Hearthstone ‘minions’), the rules are greatly simplified. For example, Hearthstone doesn’t contain any ‘land’ type cards and minion cards’s effects cannot be activated at will. In fact, except for secrets, there is no way the other play can interject in the middle of your turn. This makes the number of possibilities, and number of rules drastically simpler in Hearthstone, and was one of the reasons I thought making a computerized simulation was even possible.

Another reason I decided to actually attempt to build such a simulation was because a little searching around on the net didn’t show any similar apps or websites out there. There are a bunch of ‘deck managers’ and ‘Arena simulators’ but I never came across anything that actually simulated the game play-by-play. (After trying to develop one myself I finally figured out why – It’s quite a challenging task!)

I’ve always been into simulations since I was younger so I decided to put that passion, some coding skills, and my interest in Hearthstone into something that I hoped would turn out to be a lot of fun.

I think I’ll leave it at this for today, but please be sure to check out Deck Simulator for Hearthstone on the Apple App store!

 

Deck Simulator for Hearthstone: accepted to Apple App Store!!!

After a long period of expectant waiting, I’m very happy to announce that Deck Simulator For Hearthstone has been accepted by Apple and is now available in the Apple Store! I came into the process expecting to be in the review queue for around 6 to 7 days, but the actual wait was closer to two weeks. Fortunately, once the app got into the review process things went smoothly from there, and it was accepted without any issues.

This was our first app and we weren’t sure what to expect, but it was overall a great learning experience.

If you want to find it easily, you can do a quick search for Hearthstone and it should come up in the top ten results (it was #7 when I did a few minutes ago). You can recognize it easily by the blue and green icon, which is shown at the bottom of this post.

In future posts I plan on giving some more detail into the development that went behind this tool as well as how to effectively use it, since there isn’t currently much in-game documentation. Though DSfM (or HSSIM as we like to call it internally) is technically in the “game” category, what it really is a research tool to help you understand and maximize your enjoyment of Blizzard’s wonderful Hearthstone game, which is also free on the app store. If you haven’t tried out Hearthstone, please check out it out I’m sure you’ll quickly get addicted!

So feel free to try out our app and give feedback, since refining any software to reach greatness requires an active user community.

Here is a link to the app:

https://itunes.apple.com/us/app/deck-simulator-for-hearthstone/id914214349?mt=8

dropspiral