Lets Draw the figure

Equilateral triangle would give the maximum area. P(6,8) to O(0,0) is 10. (3-4-5 rt. triangle. ). So r=10 * Sin(60/2)=5.

Equation of the chord of contact is (QR) is $6x+8y-r^2=0$ .

PM = $\frac{|6.6+8.8-r^2|}{\sqrt{6^2+8^2}}=\frac{|100-r^2|}{10}$

OM = $\frac{|0+0-r^2|}{\sqrt{6^2+8^2}}=\frac{|r^2|}{10}$

So, QR = 2. QM = $2\sqrt{(OQ)^2-(OM)^2}$ = 2 $\sqrt{r^2-\frac{r^4}{100}}$

Suppose area of PQR is $\delta$ = $\frac{1}{2}.QR.PM$

$\delta = \frac{1}{2}.2.\sqrt{r^2-\frac{r^4}{100}}.\frac{|100-r^2|}{10}$

$(\delta)^2 = \frac{r^2(100-r^2)^3}{1000} = \alpha$ (Say)

$\frac{dz}{dr} = \frac{1}{1000}[r^2-3(100-r^2)^2.(-2r) + (100-r^2)^3.2r]$

= $\frac{2r(100-r^2)^2}{1000}(100-4r^2)$

We have $\frac{d^2z}{dr^2}<0$ , so we get maximum at $\frac{dz}{dr}=0$ .

Solving $\frac{dz}{dr} = 0$ we get $r=5$ $\ne 10$ as P lies outside the circle.

So $\delta$ is maximum at $\boxed{r=5}$