Boring dice testing

[Anderas]

So i was actually testing my dice.
Yes.

Currently i am doing a data science training in the evenings after the work, which is all about statistics, python, machine learning, artificial intelligence... meeep... rewind... what was the first lesson?

Yes. Statistics.

We have to generate a write-up of some data that we generate ourselves as end-project of the first lesson.
I was shifting it to later, because "generating data" is the part that was quite boring.
Now i did it, finally.

I took the dice tower of Zombicide, put some felt inside because that thing is awfully loud otherwise, and then i rolled five different colored dice, 150 times.
They were white, turquoise, blue, clear and red.
As if that wasn't enough, i repeated the test with 8 of those sets.
Yes it was boring.

Turns out, half of the white dice are broken.
The Turqouise dice are acceptable.
And.

ALL of the Blue, clear and red dice are just broken. Each single one of them prefers one number over some other.
Interestingly, it has nothing to do with weighting. The transparent dice display an air bubble at the corner between 3 and 4, but the defect seems to be independent from that bubble.


How's that possibly on topic with Heroquest?

Well, all those dice were from chessex, with a somewhat flawed production process: They get out of their form, then they get dipped into paint, then they are stonewashed until the mold lines are gone, the paint sticks only in the pips, and they are not straight anymore.

The HQ Dice are not like that. They are made from wood, and the symbols are printed. So i guess there is no dice-destroying stonewashing process involved.
I just came to appreciate another little piece of our game.

[knightkrawler]

Chessex, of all companies...
Not so broing at all, if you ask me.

[Goblin-King]

Boring to generate the data, perhaps - But the data itself is very interesting indeed.

[Anderas]

Those are the dice i tested



It's worth noting that the turqoise seem to be a slightly different model than the other three. The pips are closer to the center, the dice are more rounded. The "holes" are no holes, just parts with transparent plastic. This photo shows a nice indent on the red dice's one that will certainly have some influence.

It's also worth noting that you can't buy those models anymore: I bought them 20 years ago when i was still young.
So everything i did applies to these dice, but not necessarily to dice bought today or other dice models.

I checked first for the deviation from the standard, but then i checked also for the reason of the deviation. After all, it could be the die is just doing it's work, and part of being random is showing one number more than the others.

So i checked the standard deviation not only on the total result, but also during the entire process for each group of 30 rolls. I assume if there is a technical defect, it will be visible all the time. If it is just a lucky streak, it will be visible once and then disappear. That way, two suspicious dice have been found innocent ( ) despite a strange result, on the other hand eight more dice have been found guilty despite having an overall deviation close enough to look inconspicuous on first glance.

[whitebeard]

So for each individual die, you only rolled it only 30 times?

For the dice which are concluded to be defective, what confidence interval do the observations lay outside of?

While I understand there are poorly weighted dice out there from inadvertent manufacturing flaws. I`m having a hard time believing this behaviour is conclusively observable for any such die in only 30 rolls. At this point I either run the calculations myself, or shut-up. Shutting-up.

[knightkrawler]

He rolled each die 150 times.
He did write down observations within each set of 30 rolls, though, to see if problems of the macrocosm apply to the microcosm and repeat themselves destinedly or coincidentally.
-----------------------------------------------------------------in the words of a wayward philosopher

[Anderas]

I rolled each die 150 times.
If one side count was equal or outside 2.5 standard deviations I said it was defect.
Once I extended the test to 180 rolls because of two dice hanging around 2 standard deviations and the other method has also not been conclusive.

Then I started to check the sub-series of 30 rolls, not to see the deviation but more to see if the deviation was more or less regularly on the upper or lower side. Checking that diagram made me confident enough to rule out even some dice that were merely 2 standard deviations from the average.

It's worth noting that you would probably not see or feel the errors when playing games like 40k. Mostly the dice were preferring one side on the cost of a neighbor, like, preferring the 5 on the cost of the 4. So if you play a game and you want to roll 3+ then you wouldn't see a difference.

But I found also that my greenish turquoise dice are significantly better... well, they stay mostly inside 1.8 deviations. So I will use them in the future.

[whitebeard]

I rolled each die 150 times.

Got it. I saw 5 types and 150 rolls, repeated for 8 of each die from the set. Then somehow you have groups of 30 for one die... and 150 / 5 = 30. So while I understood you did a lot of rolling, it seemed that perhaps not enough for an individual die. Sounds like a fun Friday night

For the defective dice, you may be able to support "weighting" of the die from your data if the number directly accross from the high number is the low number. Weighting with edge bias if two adjacent faces are high and the two across are low. And weighting on the vertex (corner) if three adjacent sides are high. While a geometric anomaly (smaller faces or different edge contours) could provide any other result. And for this you may need more rolls.

A die has six sides and you are requiring all of them to be within some sigma value to not be defective. Ignoring the obvious constraint, and choosing 2 sigma (5% of perfectly good sides will be called bad), then you have a 6 x 5% = 30% (EDIT: yeah that's not quite right) chance of calling a good die bad. Of course the constraint changes everything... do you have this answer? You could ignore the constraint if you look at only one value on the die. How many of your 5x8 = 40 dice are outside of 2 sigma for the side with three pips? Is that answer outside of 2 sigma?

[Anderas]

Yes, i made that calculation, too. Not the one for all 40 dice, though. I also did not yet do my summary or a write-up so i am still perfectly open to change everything.

Anybody who wants to play around with it, here's the raw data as csv.
Link to my dropbox
You are teaching, or not? If you want, use it, i don't reserve any rights on it.


A side with 2.2 standard deviations should appear in 1.4% of the cases if everything's perfectly normal.
That one out of six sides is so far away is then 7.9%, or 3 dice out of 40. But i did see 13 Dice beyond that border.
Same with any other randomly chosen borderline.
I was getting so mad that at one time, i was cross-checking with excel generated random numbers just in case i calculated something wrong somewhere, but no - the excel random numbers behave as expected.

I guess my rolling method preferred geometrical failures and tends to ignore weighting failures.
I used a dice tower, I put felt all the way in the dice tower, and I rolled always 5 dice together.

That means:
* the surface supports rolling because it is soft
* The dice tower does not support bouncing, as it is small and there is felt inside
* the surface puts the breaks on the speed of a die very fast
* to have a realistic environment, i rolled always 5 dice together. So they are also colliding with each other, invading the personal space of each other... like on the real gaming table. I am only interested in failures that are visible in such a real life situation.


I did not want roll a lot more of course, so instead i separated my notes into subsets of 30, assuming i would see technical defects all the time and luck only sometimes.
And that analysis, though not perfect mathematically speaking, made a good enough impression to convince me. Do you know how to calculate the probability of having the deviation 4 out of 5 times on the same side of the mean? Can i just work with the probs or do I have to take care for something?

This here is an example. On the right, you see the % of the total result appearing. 2 and 3 are the numbers that are the most off. But with 3.5% or 1.8 standard deviations they are not very far off, so that makes an impression of being normal: We would expect such a result in roughly 20% of the dice, or 8 out of 40. (i did see 24 like that)

However, if you divide into subsets, you see that the three was always below or exactly at the average. I made it blue for above the average and red for below the average.


And that's where i say, ok, it wasn't a streak changing the entire result, it looks like a technical failure.
This one i rolled 180 times because i couldn't decide for "guilty". That means each subset has 36 rolls.
The day i encode all that in Python, i will have one subset more instead. That seems to be cleaner.

[whitebeard]

Some food for thought.

What is the spec you are looking for with these dice? You are currently testing against a perfect random process, but how good is "good enough"?

I ask this because the more samples you collect, the greater the precision to which you are able to show that ANY die is not perfect. If you roll a million times, you could show that a die which is great for all practical purposes is outside of 5 sigma of a perfect die. In short, perfect should not be your objective.

How bad are these dice really?


Also, you have omitted the constraint in one of your assertions. It's not terribly important, but it does change the expected value for the number of results outside of a sigma value. Once you have one side which registers an extreme value (because of randomness or because of defect), the liklihood that the other sides will be wrong is not random. So there is not a 7.9% chance that a perfect die shows outside of 2.2 sigma, it is even lower.