$H$ -&- $T$

Suppose that each person, seeing two heads, would guess that they have tails; and conversely if they see two tails would guess that they have heads. Any person seeing two colleagues with different faces showing will pass.

Then we can enumerate all 8 possibilities, and see what happens.

So in 6 of the 8 equally likely cases, the team wins. Probability is 6/8 = 0.75 for this strategy. I haven't proven that this is the optimum strategy.

Let $V = \{-1,1\}^n$ be the set of all possible hat colour arragements, and let $d(\mathbf{a},\mathbf{b}) \; = \; \tfrac12\big(|a_1-b_1| + |a_2-b_2| + |a_3 - b_3|\big)$ be the Hamming distance function on $V$ . A strategy is a triple $F = (f_1,f_2,f_3)$ , where each $f_j$ is a function $f_j : \{-1,1\}^2 \to \{-1,0,1\}$ . Thus, for a hat arrangement $(a,b,c) \in V$ , this strategy tells Player $1$ to pick $f_1(b,c)$ , Player 2 to pick $f_2(a,c)$ , and Player 3 to pick $f_3(a,b)$ . A player picking $0$ represents a player passing. For any strategy $F$ , let $W_F \subseteq V$ be the set of colour arrangements for which $F$ wins.

If $(a,b,c)$ and $(-a,b,c)$ both belong to $W_F$ , it is clear that $f_1(b,c) = 0$ (if Player 1 did not pass, she would be wrong for one of these two configurations). Similarly, if $(a,b,c)$ and $(a,-b,c)$ both belong to $W_F$ , then $f_2(a,c) = 0$ , and if $(a,b,c)$ and $(a,b,-c)$ both belong to $W_F$ then $f_3(a,b) = 0$ . Since we cannot have $(a,b,c) \in W_F$ with $f_1(b,c) = f_2(a,c) = f_3(a,b) = 0$ , we deduce that at least one of $(a,b,c)$ , $(-a,b,c)$ , $(a,-b,c)$ and $(a,b,-c)$ must belong to the complement $W_F'$ of $W_F$ .

In other words, for any $\mathbf{a} \in V$ , there exists $\mathbf{b} \in W_F'$ such that $d(\mathbf{a},\mathbf{b}) \le 1$ . Since the number of elements of $V$ that can be withing Hamming distance $1$ of a single element of $V$ is $4$ , we deduce that $W_F'$ must contain at least $2$ elements, and hence $W_F$ has at most $6$ elements. Thus any strategy $F$ has at most $\tfrac34$ chance of winning.

The standard strategy $F$ , where each player passes if they see two hats of different colours, and chooses the opposite colour if they see two hats of the same colour, has $W_F' = \{(1,1,1),(-1,-1,-1)\}$ , and so has optimal probability of success $\boxed{\tfrac34}$ .

N.B. This argument can be extended to handle games with $2^n-1$ players.

It is easy to see, that at least two of the letters always has to be the same (by applying the pigeonhole principle) and also that there are 2×2×2 = 8 different outcomes in total (of which all letters are the same in 2 cases (p = 2/8 =1/4) ).

If each of the three members is guessing only (the letter which he/she cannot see on the other members), when he/she sees that the two other members have the same letter on their foreheads (there is at least one of them who sees this) and passes otherwise, then they only lose, when all the letters are the same (in which case there are 3 wrong guesses).

In all the other (6 out of 8) cases they win with 1 correct guess and two passes.

Hence, the probability of winning with this strategy is:

$\frac {6}{8} = \boxed { \frac {3}{4} }$

The solution presented by Zee Ell gives a high probability of $\tfrac34$ . It is the best strategy that is symmetric and deterministic .

But could there be a better strategy in which different players use a different strategy; or in which they randomly decide whether or not to guess, or what to guess, with a probability dictated by what they see?

One would need to analyze such a general strategy (which involves 24 degrees of freedom, if I am not mistaken), or otherwise prove that no better probability than $\tfrac34$ may be obtained. After all, the participants are smart Brilliant members...

1)There are 8 possible scenarios corresponding to 8 possible outcomes of 3 coin tosses. The optimal strategy cannot guarantee you the victory in every single of the 8 possible configurations, since this scenario is indistinguishable from having perfect knowledge of the outcome of the coin tosses in advance.

2) Hence, there is at least 1 configuration in which the optimal strategy will lose. Since there are groups of symmetric configurations (HHT and its 2 permutations, TTH and its 2 permutations, HHH and TTT), any strategy that loses on 1 configuration will also lose on its symmetric partners. So it is more convenient to construct a strategy that loses on, say, HHH or TTT rather than on one of the others, which have a larger number of symmetric partners.

3) Once the optimal strategy loses on HHH, it also loses on TTT and viceversa. Hence the optimal strategy can't win with a probability greater than 6/8 = 3/4.

4) There exist a strategy that wins with probability 3/4 (if you see HH say T, if you see TT say H, else pass). Hence, by the previous point, it is also optimal.

So I decided to take a jab at the non-deterministic (and symmetric*) case:

Since each player can see the other players, there are three choices in what they see: $HH$ , $HT$ , or $TT$ **. Upon seeing these options, there are three choices in what to say when they all guess: $h$ ("heads"), $t$ ("tails"), or $p$ ("pass"). Let's define $\rho_{XY \rightarrow a}$ as the probability of each player guessing $a$ after seeing $XY$ where $XY\in\{HH, HT, TT\}$ and $a\in\{h,t,p\}$ . Exhausting the permutations, we have nine different $\rho$ s.

Given this set of $\rho$ s, we would like to find the probability that you will win a random game. First to wrap our heads around this, let's take one of the eight sample games as an example given a set of $\rho$ s; let's try the game where the players are assigned two $H$ s and one $T$ (there are three of these, namely, $HHT$ , $HTH$ , and $THH$ ). Two of the players have $H$ and see $HT$ and one player has $T$ and sees $HH$ . Given this particular game, in order to win, the players who have $H$ and see $HT$ need to guess $h$ OR $p$ , AND the player who has $T$ and sees $HH$ needs to guess $t$ OR $p$ , AND not everyone can guess $p$ . Hence the probability $P(win HHT)$ ( $=P(win HTH)=P(win THH)$ ) of winning this type of game is

$P(win HHT)=(\rho_{HT \rightarrow h}+\rho_{HT \rightarrow p})^2(\rho_{HH \rightarrow t}+\rho_{HH \rightarrow p}) - {\rho_{HT \rightarrow p}}^2\rho_{HH \rightarrow p}$ .

Now since there are eight possible games evenly distributed, we may write the probability of winning a random game as

$P(win random game) = \frac{1}{8}\sum_{X,Y,Z\in\{H,T\}}{P(win XYZ)} \\ = \frac{1}{8}\left(\left[(\rho_{HH \rightarrow h}+\rho_{HH \rightarrow p})^3 - {\rho_{HH \rightarrow p}}^3\right] \\ + 3\left[(\rho_{HT \rightarrow h}+\rho_{HT \rightarrow p})^2(\rho_{HH \rightarrow t}+\rho_{HH \rightarrow p}) - {\rho_{HT \rightarrow p}}^2\rho_{HH \rightarrow p}\right] \\ + 3\left[(\rho_{HT \rightarrow h}+\rho_{HT \rightarrow p})^2(\rho_{TT \rightarrow h}+\rho_{TT \rightarrow p}) - {\rho_{HT \rightarrow p}}^2\rho_{TT \rightarrow p}\right] \\ + \left[(\rho_{TT \rightarrow t}+\rho_{TT \rightarrow p})^3 - {\rho_{TT \rightarrow p}}^3\right]\right)$

In the last equality, the first line is the game with all $H$ s; the second line, the games with two $H$ s; the third line, the games with one $H$ ; and the last line, the game with all $T$ s. As a quick solidarity check, you can plug in the values of the maximum deterministic solution already discussed in these comments, to check this equation. Doing so gives you 3/4 as expected.

You can eliminate three of these $\rho$ s by normalizing the probabilities: for example, $\rho_{HT \rightarrow h}+\rho_{HT \rightarrow t}+\rho_{HT \rightarrow p}=1$ . This will leave you with a six dimensional problem

From this point forward it is a matter of solving for the maximum value of this six dimensional function within the boundaries of $0 \leq \rho_{XY \rightarrow a} \leq 1$ for all $XY\in\{HH, HT, TT\}$ and $a\in\{h,t,p\}$ . At this point, I'll let someone else take over and do some coding or some really tedious calculus (using a lot Lagrange multipliers). The boundaries of this problem are the most annoying since you have to address them each. There will be $3^6=729$ cases since you can choose each variable to be fixed at the lower bound, the upper bound, or be free. Coding will be a lot simpler, but you will have to choose many random starting points in the space to ensure that you are covering all your bases. I suspect that $\frac{3}{4}$ is the best as it is in one of the corners or the 9-dimensional solid hypercube, but I would greatly like it if someone with more time than I have right now would check this out. Good luck with coding or math if you feel up to it.

*I feel like asymmetric solutions are outside of the rules since there is no way to communicate and decide who does what.

**Again, they can't determine an ordering so seeing $TH$ and $HT$ have to be the same due again to lack of communication.

I got this no better than 3/8; note that each one has options to either pass or say the correct answer thus probty. of winning is \(\frac{1}{2} )+ \(\frac{1}{2} \times $\frac{1}{2}$ )+ \(\frac{1}{2} \times $\frac{1}{2}$ \times $\frac{1}{2}$ ) = 3/8