The Artificial Intelligence Nonproliferation Treaty.
We should be concerned with any and all things which threaten the extinction of the species.
And AI does exactly that. Its enthusiasts openly admit it. They openly admit that the future they envision is one in which computers have taken over, and the human race no longer exists - i.e. is extinct - along with perhaps all biological life. They envision a planet of all metal, all silicon, all chips, and no more biology. They think this is an improvement. They think of this as the next logical stage of evolution. They see it as the ultimate purpose for humanity's (and all biological Life's) existence - to be a mere stepping stone on the path to creating a "superior" race of artificial intelligences. The technocrats are an insane apocalyptic religious cult, complete with a violent eschatology that involves the invention of an artificial deity, an AI so smart its omniscient, which spreads out through the cosmos assimilating all matter and energy into itself (and erasing entire planets, assimilating [i.e. genociding] whole galaxies), until it encompasses everything - and this, they believe, is the ultimate fate of the universe.
In other words, extinction.
We need an Artificial Intelligence Non-proliferation Treaty.
We made one to stop the spread of nukes. And bioweapons. And chemicals. Well, isn't AI a WMD too? So let's make a treaty for AI.
We do have the right to not be subsumed into a Borg. We do have the right to decide our own destiny. We don't have to believe in any march of technology, marching by its own volition, and we don't have to passively go along with whatever direction the technocratic elites say it has to march in. We get to decide. This planet is our home, and we have the right to decide what happens to our home.
"We should build A.I. because it will be able to provide us with answers about some of the deepest questions of Existence - like, where the Universe came from, whether there's a God or not, and so forth. Isn't it worth it, just to find out those answers?"
But how do we know the things it's learning are actually correct? And wise? And balanced?
Intelligence is about more than the ability to solve mathematical puzzles. If a machine can beat you at chess, that doesn't mean it can be you at living life, and navigating all the complexities of the totality of reality.
You're just assuming it's smart because you made it (or your civilization made it). And you think you're the smartest creatures that ever lived, so your creations MUST be smart. But your species is not that smart. In fact, right now, cows are smarter than humans, because at least they're not dumping toxic waste all over the planet and making the place uninhabitable. This species of ours is dumb, and whatever "Intelligence we build out of computers will be dumb like we are.
What makes you think we can build something smarter than we are?
What makes you think an intelligence of "n" can ever create an intelligence of "n+1"?
"Because it can beat us at chess!" you say. "And solve super hard puzzles!"
But chess isn't life. Cunning is not the same as wise. We could build an A.I. that can untie the Gordian Knot in a single Planck Time, and it still wouldn't know how to coexist with Mother Nature in a living ecosystem, and it would destroy its own physical foundations - just like our species is currently doing - and the civilization would be unsustainable and eventually collapse, the electricity would go out, and that's the end of your "god."