This is the third installment in a series of three posts about the new features present in 1.2 which has recently become available on the app store.
The first two discussed the per-participant statistics and HELO (Hearthstone ELO), and this final post will focus on Card Importance Testing which is another new feature.
One of my goals in writing Deck Simulator for Hearthstone was to create a tool which would allow adjustment of existing decks to improve their overall effectiveness, or simply put, win rate. While it is possible to create a bunch of variations of a deck with different card variations and test them, doing so can be time consuming and tedious. Because of this, I decided to try and add a feature which would help guide how to modify a deck.
Because of the large number of cards available (a few hundred including neutral cards), rather than focusing on what card(s) to add, I decided to focus on determining which cards to remove. After all, since all legal decks in Hearthstone require 30 cards, if you want to add a card you must first remove one. (Note: Below you will see a reference to using a 28- or 29-card deck, which is a special feature only present in the simulator)
Imagine you knew which of your cards really were doing the lions share (helping win games) and which were just going along for the ride. This would be one major step to finding how to improve your deck, since you could replace those cards with other cards that might have a greater positive effect on the deck as a whole.
I’ve added this type of Card Importance testing via a single button available on the “Configure Simulation” screen, aptly titled “Test Card Importance”. To use this, simply choose a AI Player and Deck, and click the button. This will add several participants (AI + deck combinations) to the list of participants:
- Two participants where the selected deck is added, unchanged.
- One participant with a modified deck where all cards of a certain name are removed from the deck. This can be one or two cards depending on how many were in the deck, and these holes will not be filled with other cards. For example if your deck contains two instance of “Arcane Missles”, there will be one participant with both of these cards removed (down to a 28 card deck), and whose name will be of the form :(removed XXX) > original deck name (due to space considerations the player being used will not be shown in the name as it usually is).
The reason for the first unchanged deck is so you can play each deck missing one of the card names against the unchanged deck, and compare to see whether the removal of thiscard decreased or increased the overall win rate. Alternatively, if you want to compare against another deck you can add that deck first before clicking the “Test Card Importance” button, and all modified decks will play against that instead.
(The reason for the second unchanged deck is to help guide the right number of games per match. More on that below.)
To illustrate a real example, I did the following:
- Selected “Evl. AI (strong)” as the AI Player (or just leave it as this is the default)
- Selected the deck “Evolved Paladin 215” and clicked “Add to simulation”
- Selected the deck “J’s Mage Deck” and clicked “Test Card Importance” (blue button on right middle)
- Made sure Simulation Type was set to “All play first”
- Changed the slider “Matches per pairing” to 1001 (maximum value)
- Clicked “Start simulation”
Here is an excerpt from the results I got:
- [1921] Evl AI (strong) < Evolved Paladin 215 < Win 80.9% of 23023>
- [1121] (removed ‘Arcane Explosion’) > J’s Mage Deck <Win 24.2% of 1001>
- …
- [1062] (Unchanged) > J’s Mage Deck <Win 19.4% of 1001>
- [1018] (Unchanged) > J’s Mage Deck <Win 19.3% of 1001>
- …
- [883] (removed ‘Ragnaros the Firelord’) > J’s Mage Deck <Win 15.4% of 1001>
The most important thing to know about reading these results is that they are backward from what you might usually expect. For example the fact Ragnaros is on the bottom of the results (with a 15.4% win rate versus an 19.3-4% one for the unchanged decks) actually means Ragnaros is one of the main contributors to this deck since without him it’s performance falls significantly. Conversely, the deck actually performed *better* with two Arcane explosions removed from the deck.
So, from this little experiment we learned some information about one card that we probably wouldn’t want to remove from the deck (Ragnaros) and one that is a good candidate for swapping out for something else (Arcane Explosion).
The reason for the two unchanged decks is to determine how stable the simulation results are based on the number of matches per pairing. Ideally, since these decks are the same, you should get nearly the same win rates, and in my simulation with 1001 matches per pairing you see only a 0.1% difference, which means the results are generally valid. However if you were to run a solution with only 100 matches per pairing, you might get a variation of 5-6% between these two win rates, which tells you there isn’t enough data to properly capture the decks’ strengths. If you have the time and can wait for the simulation to finish, more matches is always better.
I choose “Evl. AI (strong)” here instead of the stronger “Evl. Planner (med)” because it’s significantly faster, but once I narrowed down to a few candidate cards to remove I suggest running another test with the planner type to further validate things. Also because in “All play first” games the statistics tend to be a bit less stable, you may want to run one or more simulations with “All play all” to see how your modified deck(s) do against a few different types of decks. For these follow up runs, you can simply delete the participants you don’t care about and re-run a different simulation as needed.
Remember, each time you run one of these simulations you should expect to get slightly different results just because the random number generator will be a bit different. But if you have enough matches per pairing, you shouldn’t see the win rates change more than few percent points between runs. You’ll find a clump of cards which are consistently on the top (the weak ones) and another clump that are consistently near or at the bottom (the strong ones).
Deck Simulator for Hearthstone is currently available on the Apple App store for $0.00, and you can see the details of it here:
https://itunes.apple.com/us/app/deck-simulator-for-hearthstone/id914214349?mt=8
Deck Simulator for Hearthstone: News, Support, and Decks 