jmc

Choose side "lengths" $a,b,c$ for the triangle, where "length" is how many vertices of the octagon are skipped between vertices of the triangle, starting from the shortest side, and going clockwise, and choosing $a=b$ if the triangle is isosceles: $a+b+c=5$, where either [$a\le b$ and $a<c$] or [$a=b=c$ (but this is impossible in an octagon)]. Options are: $a=0$ with $b,c$ in { 0,5 ; 1,4 ; 2,3 ; 3,2 ; 4,1 }, and $a=1$ with { 1,3 ; 2,2} $5/7$ of these have a side with length 1, which corresponds to an edge of the octagon. So, our answer is $\boxed{\frac{5}{7}}$