Altogether it is thought—though it is really only a guess, based on extrapolating from cratering rates on the Moon—that some two thousand asteroids big enough to imperil civilized existence regularly cross our orbit. But even a small asteroid—the size of a house, say—could destroy a city. The number of these relative tiddlers in Earth-crossing orbits is almost certainly in the hundreds of thousands and possibly in the millions, and they are nearly impossible to track.
The first one wasn't spotted until 1991, and that was after it had already gone by. Named 1991 BA, it was noticed as it sailed past us at a distance of 106,000 miles—in cosmic terms the equivalent of a bullet passing through one's sleeve without touching the arm. Two years later, another, somewhat larger asteroid missed us by just 90,000 miles—the closest pass yet recorded. It, too, was not seen until it had passed and would have arrived without warning. According to Timothy Ferris, writing in the New Yorker, such near misses probably happen two or three times a week and go unnoticed.
An object a hundred yards across couldn't be picked up by any Earth-based telescope until it was within just a few days of us, and that is only if a telescope happened to be trained on it, which is unlikely because even now the number of people searching for such objects is modest. The arresting analogy that is always made is that the number of people in the world who are actively searching for asteroids is fewer than the staff of a typical McDonald's restaurant. (It is actually somewhat higher now. But not much.)