Not Medical Advice: No, as olive pits are not poisonous. If you ate one it would probably go through you undisturbed.
However, a possible dental damage and danger of olive pits was reported by CBS News in 2011. A liberal democratic congressman from Ohio, Dennis Kucinich, filed a lawsuit against a congressional cafeteria for dental damage he allegedly suffered after biting into "dangerous substances" - namely an olive pit in a sandwich wrap he bought there.
The civil suit filed in 2011 in the Superior Court of the District of Columbia said the wrap "contained dangerous substances, namely an olive pit," that a consumer would not reasonably expect to find in the product served. The suit said that Kucinich suffered "serious and permanent" dental and oral injuries requiring multiple surgical and dental procedures.
Kucinich's lawsuit said the Ohio Democrat bought the sandwich on or about April 17, 2008. He seeks damage for "past and future medical and dental expenses, compensation for pain, suffering and loss of enjoyment and other damages."
Check out Healthy Olive Recipes and Cooking Tips from Eating Well.