Re. "It would not be able to carry out any sort of behavior that was qualitatively different from the stuff done by the humans in its training dataset. For instance, if a radically new kind of vehicle was invented -- say a flying hoverboard -- it wouldn't be able to adapt to riding it."
Against that, consider this quote about Alpha Go's second game against Lee Sedol:
"In the thirty-seventh move of the second game, AlphaGo shocked the Go world by defying that ancient wisdom and playing on the fifth line (figure 3.2), as if it were even more confident than a human in its long- term planning abilities and therefore favored strategic advantage over short-term gain. Commentators were stunned, and Lee Sedol even got up and temporarily left the room. Sure enough, about fifty moves later, fighting from the lower left-hand corner of the board ended up spilling over and connecting with that black stone from move thirty-seven! And that motif is what ultimately won the game, cementing the legacy of AlphaGo’s fifth-row move as one of the most creative in Go history." -- copy-pasted from https://www.slideshare.net/edsonm/michael-peter-edson-robot-vs-human-who-will-win, original source not given
Re. "It would not be able to carry out any sort of behavior that was qualitatively different from the stuff done by the humans in its training dataset. For instance, if a radically new kind of vehicle was invented -- say a flying hoverboard -- it wouldn't be able to adapt to riding it."
Against that, consider this quote about Alpha Go's second game against Lee Sedol:
"In the thirty-seventh move of the second game, AlphaGo shocked the Go world by defying that ancient wisdom and playing on the fifth line (figure 3.2), as if it were even more confident than a human in its long- term planning abilities and therefore favored strategic advantage over short-term gain. Commentators were stunned, and Lee Sedol even got up and temporarily left the room. Sure enough, about fifty moves later, fighting from the lower left-hand corner of the board ended up spilling over and connecting with that black stone from move thirty-seven! And that motif is what ultimately won the game, cementing the legacy of AlphaGo’s fifth-row move as one of the most creative in Go history." -- copy-pasted from https://www.slideshare.net/edsonm/michael-peter-edson-robot-vs-human-who-will-win, original source not given