jmc

If $x^2-x-1$ is a factor of $a{x}^{17}+b{x}^{16}+1,$ then both the roots of $x^2-x-1$ must also be roots of $a{x}^{17}+b{x}^{16}+1.$ Let $s$ and $t$ be the roots of $x^2-x-1.$ Then we must have $$a{s}^{17}+b{s}^{16}+1=a{t}^{17}+b{t}^{16}+1=0.$$Since $s$ is a root of $s^2-s-1=0,$ we have $s^2=s+1.$ This equation lets us express higher powers of $s$ in the form $Ms+N,$ for constants $M$ and $N.$ We have $$\begin{array}{rl}{s}^3& =s^2\cdot s=(s+1)s=s^2+s=(s+1)+s=2s+1,\\ s^4& =s^3\cdot s=(2s+1)s=2s^2+s=2(s+1)+s=3s+2,\\ s^5& =s^4\cdot s=(3s+2)s=3s^2+2s=3(s+1)+2s=5s+3,\end{array}$$and so on. Seeing a pattern, we guess that $$s^n=F_{n}s+F_{n-1},$$where $\{F_n\}$ are the Fibonacci numbers (with $F_1=F_2=1,$ and $F_n=F_{n-1}+F_{n-2}$ for $n\ge 3$). We can prove this formula with induction (see below). This means that $$s^{16}=F_{16}s+F_{15}=987s+610\text{ and }{s}^{17}=F_{17}s+F_{16}=1597s+987.$$Thus, $$a{s}^{17}+b{s}^{16}+1=(1597a+987b)s+(987a+610b)+1,$$so it must be the case that $1597a+987b=0$ and $987a+610b=-1.$ This system has solutions $a=\boxed{987}$ and $b=-1597.$

Proof of formula: We already did the base cases of the induction. If $s^n=F_{n}s+F_{n-1}$ for some value of $n,$ then $$\begin{array}{rl}{s}^{n+1}=s^n\cdot s& =(F_{n}s+F_{n-1})\cdot s\\ & =F_n{s}^2+F_{n-1}s\\ & =F_n(s+1)+F_{n-1}s\\ & =(F_n+F_{n-1})s+F_n=F_{n+1}s+F_n.\end{array}$$This completes the inductive step. $\square$