\(% Differentiation % https://tex.stackexchange.com/a/60546/ \newcommand{\Diff}{\mathop{}\!\mathrm{d}} \newcommand{\DiffFrac}[2]{\frac{\Diff #1}{\Diff #2}} \newcommand{\DiffOp}[1]{\frac{\Diff}{\Diff #1}} \newcommand{\Ndiff}[1]{\mathop{}\!\mathrm{d}^{#1}} \newcommand{\NdiffFrac}[3]{\frac{\Ndiff{#1} #2}{\Diff {#3}^{#1}}} \newcommand{\NdiffOp}[2]{\frac{\Ndiff{#1}}{\Diff {#2}^{#1}}} % Evaluation \newcommand{\LEvalAt}[2]{\left.#1\right\vert_{#2}} \newcommand{\SqEvalAt}[2]{\left[#1\right]_{#2}} % Epsilon & Phi \renewcommand{\epsilon}{\varepsilon} \renewcommand{\phi}{\varphi} % Sets \newcommand{\NN}{\mathbb{N}} \newcommand{\ZZ}{\mathbb{Z}} \newcommand{\QQ}{\mathbb{Q}} \newcommand{\RR}{\mathbb{R}} \newcommand{\CC}{\mathbb{C}} \newcommand{\PP}{\mathbb{P}} \renewcommand{\emptyset}{\varnothing} % Probabililty \DeclareMathOperator{\Cov}{Cov} \DeclareMathOperator{\Corr}{Corr} \DeclareMathOperator{\Var}{Var} \DeclareMathOperator{\Expt}{E} \DeclareMathOperator{\Prob}{P} % Distribution \DeclareMathOperator{\Binomial}{B} \DeclareMathOperator{\Poisson}{Po} \DeclareMathOperator{\Normal}{N} \DeclareMathOperator{\Exponential}{Exp} \DeclareMathOperator{\Geometric}{Geo} \DeclareMathOperator{\Uniform}{U} % Complex Numbers \DeclareMathOperator{\im}{Im} \DeclareMathOperator{\re}{Re} % Missing Trigonometric & Hyperbolic functions \DeclareMathOperator{\arccot}{arccot} \DeclareMathOperator{\arcsec}{arcsec} \DeclareMathOperator{\arccsc}{arccsc} \DeclareMathOperator{\sech}{sech} \DeclareMathOperator{\csch}{csch} \DeclareMathOperator{\arsinh}{arsinh} \DeclareMathOperator{\arcosh}{arcosh} \DeclareMathOperator{\artanh}{artanh} \DeclareMathOperator{\arcoth}{arcoth} \DeclareMathOperator{\arsech}{arsech} \DeclareMathOperator{\arcsch}{arcsch} % UK Notation \DeclareMathOperator{\cosec}{cosec} \DeclareMathOperator{\arccosec}{arccosec} \DeclareMathOperator{\cosech}{cosech} \DeclareMathOperator{\arcosech}{arcosech} % Paired Delimiters \DeclarePairedDelimiter{\ceil}{\lceil}{\rceil} \DeclarePairedDelimiter{\floor}{\lfloor}{\rfloor} \DeclarePairedDelimiter{\abs}{\lvert}{\rvert} \DeclarePairedDelimiter{\ang}{\langle}{\rangle} % Vectors \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\bvect}[1]{\overrightarrow{#1}} % https://tex.stackexchange.com/a/28213 % \DeclareMathSymbol{\ii}{\mathalpha}{letters}{"10} % \DeclareMathSymbol{\jj}{\mathalpha}{letters}{"11} % \newcommand{\ihat}{\vect{\hat{\ii}}} % \newcommand{\jhat}{\vect{\hat{\jj}}} \newcommand{\ihat}{\textbf{\^{ı}}} \newcommand{\jhat}{\textbf{\^{ȷ}}} \newcommand{\khat}{\vect{\hat{k}}} % Other Functions \DeclareMathOperator{\sgn}{sgn} \DeclareMathOperator{\tr}{tr} % Other Math Symbols \DeclareMathOperator{\modulo}{mod} \newcommand{\divides}{\mid} \newcommand{\notdivides}{\nmid} \newcommand{\LHS}{\text{LHS}} \newcommand{\RHS}{\text{RHS}} \newcommand{\degree}{^{\circ}}\)
Since \(P(x) = Q(x) R'(x) - Q'(x) R(x)\), we notice that \[ \frac {P(x)}{Q(x)^2} = \DiffOp {x} \frac {R(x)}{Q(x)}. \]
Hence, \[ \int \frac {P(x)}{Q(x)^2} \Diff x = \frac {R(x)}{Q(x)} + C \] where \(C\) is a real constant.
Since \(R(x) = a + bx + cx^2\), we have \(R'(x) = b + 2cx\). We let \(P(x) = 5x^2 - 4x - 3\) and \(Q(x) = 1 + 2x + 3x^2\), and hence \(Q'(x) = 6x + 2\).
Hence, \[ 5x^2 - 4x - 3 = (1 + 2x + 3x^2)(b + 2cx) - (6x + 2)(a + bx + cx^2). \]
Notice that \begin {align*} \RHS & = [6cx^3 + (3b + 4c) x^2 + (2b + 2c)x + b] - [6cx^3 + (6b + 2c)x^2 + (6a + 2b)x + 2a] \\ & = (-3b + 2c) x^2 + (-6a + 2c) + (-2a + b). \end {align*}
Hence, we have \[ \left \{ \begin {aligned} -3b + 2c & = 5, \\ -6a + 2c = -4 \iff 3a - c & = 2, \\ -2a + b & = -3. \end {aligned} \right . \]
Notice that \[ 1 \cdot (-3b + 2c) + 2 \cdot (3a - c) + 3 \cdot (-2a + b) = 0, \] and \[ 1 \cdot 5 + 2 \cdot 2 - 3 \cdot 3 = 0, \] which means that these three equations are linearly dependent. Hence, let \(a = 0\), and hence \(b = -3\), \(c = -2\), \(R(x) = -3 x - 2x^2\), which gives \[ \int \frac {5x^2 - 4x - 3}{(1 + 2x + 3x^2)^2} \Diff x = \frac {- 3x - 2x^2}{1 + 2x + 3x^2} + C_1. \]
Letting \(a = 1\), and hence \(b = -1\), \(c = 1\), \(R(x) = 1 - x + x^2\), which gives \[ \int \frac {5x^2 - 4x - 3}{(1 + 2x + 3x^2)^2} \Diff x = \frac {1 - x + x^2}{1 + 2x + 3x^2} + C_2. \]
Notice that \[ \frac {1 - x + x^2}{1 + 2x + 3x^2} - \frac {- 3x - 2x^2}{1 + 2x + 3x^2} = \frac {1 + 2x + 3x^2}{1 + 2x + 3x^2} = 1, \] and the integrals just differ by a constant. Different choices of \((a, b, c)\) lead to results which only differ by a constant.
The differential equation we are attempting to solve is equivalent to \[ \DiffFrac {y}{x} + \frac {\sin x - 2 \cos x}{1 + \cos x + 2 \sin x} y = \frac {5 - 3 \cos x + 4 \sin x}{1 + \cos x + 2 \sin x}. \]
The integrating factor is \begin {align*} I(x) & = \exp \int \frac {\sin x - 2 \cos x}{1 + \cos x + 2 \sin x} \Diff x \\ & = \exp \int - \frac {\Diff (1 + \cos x + 2 \sin x)}{1 + \cos x + 2 \sin x} \\ & = \exp (- \ln \abs *{1 + \cos x + 2 \sin x}) \\ & = \frac {1}{1 + \cos x + 2 \sin x}, \end {align*}
and hence \[ \DiffOp {x} \frac {y}{1 + \cos x + 2 \sin x} = \frac {5 - 3 \cos x + 4 \sin x}{(1 + \cos x + 2 \sin x)^2}. \]
Let \(Q(x) = 1 + \cos x + 2 \sin x\), and let \(P(x) = 5 - 3 \cos x + 4 \sin x\). We have \(Q'(x) = 2 \cos x - \sin x\)
Set \(R(x) = a + b \sin x + c \cos x\) for some real constant \(a, b\) and \(c\). We have \(R'(x) = b \cos x - c \sin x\). Hence, \[ 5 - 3 \cos x + 4 \sin x = (1 + \cos x + 2 \sin x)(b \cos x - c \sin x) - (2 \cos x - \sin x)(a + b \sin x + c \cos x). \]
We expand the brackets on the right-hand side, and we have \begin {align*} \RHS & = b \cos x - c \sin x + b \cos ^2 x - c \cos x \sin x + 2b \sin x \cos x - 2c \sin ^2 x \\ & \phantom {=} - 2a \cos x - 2b\sin x \cos x - 2c \cos ^2 x + a \sin x + b \sin ^2 x + c \sin x \cos x \\ & = (a - c) \sin x + (b - 2a) \cos x + (b - 2c) \left (\sin ^2 x + \cos ^2 x\right ) \\ & = (b - 2c) + (a - c) \sin x + (b - 2a) \cos x, \end {align*}
and hence by comparing coefficients, we have \[ \left \{ \begin {aligned} b - 2c & = 5, \\ a - c & = 4, \\ -2a + b & = -3. \end {aligned} \right . \]
Notice that \[ 1 \cdot (b - 2c) + (-2) \cdot (a - c) + (-1) \cdot (-2a + b) = 0, \] and \[ 1 \cdot 5 + (-2) \cdot 4 + (-1) \cdot (-3) = 0, \] so the system of linear equations is linearly dependent. Hence, set \(a = 0\), and we have \(b = -3, c = -4\), and we have \(R(x) = -3 \sin x - 4 \cos x\).
Hence, \begin {align*} \int \frac {5 - 3 \cos x + 4 \sin x}{(1 + \cos x + 2 \sin x)^2} & = \int \frac {P(x)}{Q(x)^2} \Diff x \\ & = \frac {R(x)}{Q(x)} + C \\ & = - \frac {3 \sin x + 4 \cos x}{1 + \cos x + 2 \sin x} + C, \end {align*}
and hence \[ \frac {y}{1 + \cos x + 2 \sin x} = - \frac {3 \sin x + 4 \cos x}{1 + \cos x + 2 \sin x} + C, \] which means the general solution to the differential equation is \[ y = - (3 \sin x + 4 \cos x) + C(1 + \cos x + 2 \sin x). \]