Sorry calculus, not gonna work here

Since $x, y > 0$ , we can use the AM-GM inequality as follows:

$\begin{aligned} \frac {3x}2 + \frac {3x}2 + \frac {4y}3 + \frac {4y}3 + \frac {4y}3 & \ge 5 \sqrt [5] {\left(\frac {3x}2\right)^2 \left(\frac {4y}3\right)^3} & \small \color{#3D99F6}{\text{Equality occurs when }x=\frac 23, \ y=\frac 34} \\ 3x + 4y & \ge 5 \sqrt [5] {\left(\frac {3x}2\right)^2 \left(\frac {4y}3\right)^3} \\ 5 & \ge 5 \sqrt [5] {\left(\frac {3x}2\right)^2 \left(\frac {4y}3\right)^3} \\ \implies \sqrt [5] {\left(\frac {3x}2\right)^2 \left(\frac {4y}3\right)^3}& \le 1 \\ \left(\frac {3x}2\right)^2 \left(\frac {4y}3\right)^3 & \le 1 \\ \frac {16}3 x^2y^3 & \le 1 \\ \implies x^2y^3 & \le \frac 3{16} = \boxed{0.1875} \end{aligned}$

---

Calculus gonna work here

Despite the title, Calculus can work here using Lagrange multipliers as follows.

Let $F(x,y,\lambda) = x^2y^3 - \lambda(3x+4y-5)$ , then we have:

$\begin{aligned} F_x (x, y, \lambda) & = \frac {\partial F(x,y,\lambda)}{\partial x} = 2xy^3 - 3\lambda \\ F_y (x, y, \lambda) & = \frac {\partial F(x,y,\lambda)}{\partial y} = 3x^2y^2 - 4\lambda \\ F_\lambda (x, y, \lambda) & = \frac {\partial F(x,y,\lambda)}{\partial \lambda} = -3x-4y+5 \end{aligned}$

Equating $F_x (x, y, \lambda)$ , $F_y (x, y, \lambda)$ and $F_\lambda (x, y, \lambda)$ to 0, we have:

$\begin{cases} 2xy^3 = 3\lambda & ...(1) \\ 3x^2y^2 = 4\lambda & ...(2) \\ 3x+4y = 5 & ...(3) \end{cases}$

From $(1)$ and $(2): \quad \frac 23 xy^3 = \frac 34 x^2y^2$ $\implies \frac 23 y = \frac 34 x$ $\implies y = \frac 98 x$

$(3): \quad 3x + \dfrac 92 x = 5$ $\implies x = \frac 23$ $\implies y = \frac 34$ .

Therefore, the minimum value of $x^2y^3 = \left(\frac 23\right)^2 \left(\frac 34\right)^3 = \frac 3{16} = \boxed{0.1875}$

Let $p=x^2 y^3$

From the given eqaution :

$3x+4y=5$

Differentiating w.r.t. x we get:

$3+4\frac{dy}{dx}=0$ so $\frac{dy}{dx}=-3/4$

Now differentiating $p$ w.r.t. x we get:

$\frac{dp}{dx}=2xy^3 + 3x^2y^2\frac{dy}{dx}$

To get extremum make $\frac{dp}{dx}=0$

So, putting the value of $\frac{dy}{dx}=-3/4$ we get:

$2xy^3 - \frac{9}{4}x^2y^2=0$

On rearranging we get: $8y=9x$

putting this in the linear equation we get the values of x and y where p will attain maximum value.

$x=2/3 , y=3/4$

Putting this in p we get the vmaximum value $\boxed {0.1875}$

relevant wiki: Lagrange multipliers

Let $L(x,y) = x^2\cdot y^3 + \lambda(3x + 4y - 5)\Rightarrow$ $\dfrac{d}{dx} L(x,y) = 2x\cdot y^3 + 3\lambda = 0$ $\dfrac{d}{dy} L(x,y) = x^2\cdot 3y^2 + 4\lambda = 0.$ Substituing $\lambda$ in the 2 previous equations,we get $\frac{2x\cdot y^3}{3} = \frac{ x^2\cdot 3y^2}{4} \Rightarrow \frac{2y}{3} = \frac{3x}{4}\Rightarrow$ $\Rightarrow y = \frac{9x}{8}.$ Now, substitute $y = \frac{9x}{8}$ in $3x + 4y = 5 \rightarrow 3x + \frac{9x}{2} = 5 \rightarrow x =\frac{10}{15} = \frac{2}{3} \rightarrow y = \frac{3}{4}$ . Furthemore, we can see that the hessian matrix $d^2 L(2/3, 3/4)$ is a defined negative matrix(it's left to the reader as an exercise), so $x^2\cdot y^3$ has a maximum for $(x,y) = (2/3, 3/4)$ which is $(\frac{2}{3})^2\cdot(\frac{3}{4})^3 = \frac{3}{16}$ .

use am gm inequality by taking the terms $\frac{3x}{2}$ , $\frac{3x}{2}$ , $\frac{4y}{3}$ , $\frac{4y}{3}$ , $\frac{4y}{3}$ and you will get the solution

Assume x=sin a

Y=cos a .

Maximum value of 3 sin a+ 4 cos a=5.

So just find max value of sin^2 a cos^3 a :)

Use lagrange multipliers