Count them up

Recall that $x^2 \equiv 0, 1 \pmod{4}$ . So,

$a^2 + b^2 - 8c \equiv (0 \text{ or } 1) + (0 \text{ or } 1) - 0 \equiv 0, 1, 2 \pmod{4}$

Thus, the sum can never be equal to 3.

---

Similar approach working mod 8 due to the coefficient of 8:

Recall that $x^2 \equiv 0, 1, 4 \pmod{8}$ . So,

$a^2 + b^2 - 8c \equiv (0, 1 \text{ or } 4) + (0, 1 \text{ or } 4) - 0 \equiv 0, 1, 2, 4, 5 \pmod{8}$

Thus, the sum can never be equal to 3.

$x \equiv 0,1,2 \pmod 3 \\ x^2 \equiv 0,1 \pmod 3$

So different combinations of $a,b,c$ are shown below:

Case(1) : If $a=3l,b=3m,c=3n$ then :

$9l^2+9m^2-24n=3 \\ \implies 3 l^2+3 m^2-8n=1$

If $l,m$ both odd or both even then L.H.S. will be even and R.H.S. is odd which is not possible.

So, either $l\, \text{or}\, m$ will be odd-even or even-odd to sarisfy the equation.Let, $l=2p$ and $m=2q+1$ .

$3(4p^2+4q^2+4q+1)=8n+1 \\ \implies 12k+3=8n+1 \\ \implies 12k+2=8n \\ \implies 6k+1=4n$ .

L.H.S. is odd and R.H.S. is even that's not possible.So there is no solution.

Case(2): if $a=3l+1,b=3m,c=3n+1$ then :

$3l'+9m^2-8(3n+1)=3 \\ \implies 3l'+9m^2-24n=11 \\ \implies 3(l'+3m^2-8n)=11$ .

Here $3 \not | 11$ so there is no solution

Case(3): : if $a=3l,b=3m+1,c=3n+1$ .

Same logic as above.

Case(4) : If $a=3l+1,b=3m+1,c=3n+2$ then

$3l'+3m' - 8(3n+2)=3 \\ \implies 3l'+3m'-24n=19 \\ \implies 3(l'+m'-8n)=19 \\ \implies 3(l'+m'-8n)=19$ .

Here $3 \not | 19$ .So, there is no solution.

So, there are no $(a,b,c)$ exists.