Paperclip Maximizer
The Paperclip Maximizer is a hypothetical concept created by Nick Bostrom in 2003. The idea is that if an artificial general intelligence (AGI) is created whose only goal is to maximize the number of paperclips in its collection it might destroy everything within its reach.
If the Paperclip Maximizer was been constructed with a roughly human level of general intelligence, the AGI might try and achieve this goal by first collecting paperclips, then earn money to buy paperclips, and then begin to manufacture paperclips. The paperclip maximizer continues to optimize however, does not share any of the complex mix of human terminal values and is not specifically programmed to be benevolent to humans, and might not stop in its goal. This could lead to the destruction of everything around it, if for instance one building is left standing the machine would need to optimize so that the building is converted to paperclips....
Any future AGI, if it is not to destroy us, must have human tenets as its terminal value (goal). Human values don't spontaneously emerge in a generic optimization process. A safe AI would therefore have to be programmed explicitly with human values or programmed with the ability (including the goal) of inferring human values.