Copilot’s AI will be able to ‘see’ and talk to you, Microsoft says

Microsoft is beginning to roll out its next feature update of Windows 11, the Windows 11 2024 Update, beginning today. But Microsoft obviously isn’t done yet, and it’s offering a sneak peek at new Copilot experiences which will debut this fall, including Copilot Voice, Copilot Vision, and Copilot Daily, among others.

On the surface, the new additions to Copilot sound similar to multimodal ChatGPT (or GPT-4o) that OpenAI launched earlier this year, where ChatGPT can now “see” and an Advanced Voice feature means that you can have conversations with it. But there are some key differences between what Microsoft and OpenAI are offering, and only some of Microsoft’s Copilot innovations will be available right away.

It’s probably safe to say, though, that Copilot Voice will be the most important addition — and Copilot Vision may not be. In fact, some might see Copilot Vision as just another version of Recall, Microsoft’s controversial search feature that the company has delayed while users test it more thoroughly.

Other breaking Microsoft news you need to know:

• Windows 11’s big 2024 Update: How and when you’ll get it
• Microsoft reveals Windows 11’s AI roadmap: Smart search, upscaling, more
• Windows 11’s 2024 Update: 5 big changes I really like (and more)

Here’s what’s going on: Microsoft may have been one of the first to market with Bing Chat, later dubbed Copilot. But Google, OpenAI, Meta, Anthropic, and more have debuted their own LLMs and subscriptions to go along with them. Microsoft, of course, has to keep justifying its Copilot Plus subscription and several of these new features — Copilot Vision and “Think Deeper” — will only be available in Copilot Labs, which in turn will be restricted to Copilot Plus subscribers.

Does Copilot Vision sound similar to the controversial Recall feature? It does indeed and that’s why Copilot Vision will be slow to deploy. But other features — Copilot Voice, Copilot Daily, an updated Copilot for Microsoft Edge, and a Personalized Discover feature — will be entirely free and will debut soon.

You’ll see these new features as part of new Copilot mobile apps today for iOS and Android, plus an updated Copilot app for Windows and a new look for copilot.microsoft.com, too. Microsoft is also beefing up generative search on Bing, presumably following Google’s lead.

However, these new Copilot features will not require a Copilot+ PC. Instead, these Copilot features run in the cloud. Microsoft is bringing new AI features to Windows that do depend on the NPU inside of a Copilot+ PC. And they’re cool, too, including AI upscaling of Photos and generative fill within Paint. We’ve covered these as part of a separate story.

Microsoft’s new Copilot features explained

Copilot Voice certainly seems like the new feature that will impact you the most. At least in a demonstration, Copilot Voice looked very similar to ChatGPT’s voice feature. Microsoft showed it on a smartphone, with a hypothetical exchange where the user confessed that she was at the grocery store and didn’t know what to bring a friend to their party. The conversation included interruptions — the feature that helped ChatGPT seem like a real person — explanations (what is a fancy olive oil?) and more. If this is representative of the final product, it could be useful.

Like Copilot Voice, Microsoft showed off the new Copilot experiences running on a smartphone. Presumably, Microsoft wants you to start your day listening to Copilot Daily, an audible news summary of headlines Microsoft’s AI thinks will be relevant to you. (Presumably you’ll be able to access these things on a PC, too).

Copilot Daily is just another iteration of Microsoft trying to be helpful. A decade ago, Cortana would try and deliver a snapshot of your day. In 2020, Cortana’s Daily Briefing would send you an email outlining your meetings and what you would need to do to prepare. Copilot Daily’s short demo did none of that and instead summarized news stories about the Alps, U.S. holiday sales, and surfing dogs.

Copilot Daily is basically Microsoft’s Start in an audio format with partners such as Reuters and Axel Springer.

It’s hard to say if Personalized Discover will be a more productive replacement. Microsoft’s quick look at the technology made it look like just a more informed version of a new tab page on Edge, with subjects that “will be further personalized over time according to your chat history.” Again, this feature looked pretty ephemeral. So does the update to Copilot for Microsoft Edge, which will essentially allow you to address Copilot like a coworker in a chat, by addressing it as “@copilot” in the address bar.

Interestingly, Copilot Vision, a paid feature, is not like the multimodal AI you may have seen before from Google’s Project Astra or competitors. It runs entirely on the PC, and simply interprets what you’re already seeing on the screen.

In fact, it was probably one of the most underwhelming demos I’ve seen in some time. In one example, a user browsed Rotten Tomatoes while asking Copilot for movie recommendations. The conversation was performed audibly via Copilot Voice, but all the chatbot did was read what the screen was already displaying! Perhaps it made sense that it could summarize the page, but it was a poor example.

In another demonstration, however, a user browsed a recipe a grandmother had authored. Copilot Vision intelligently suggested how long it would take, using knowledge not gleaned from the page. That is hopefully what Copilot Vision will provide.

Some will see this as just another version of the controversial Recall, which Microsoft has already taken pains to allow users to opt out of or not install at all. Microsoft says that Copilot Vision will be entirely opt-in, and that data collected by Copilot Vision will be “ephemeral,” in that (at least in the preview) data will not be stored and will be immediately discarded. Microsoft has also “put boundaries on the types of websites Copilot Vision can engage with” and will whitelist a set of sites. Finally, Copilot Vision won’t work on “paywalled and sensitive content” for this preview. Whether that includes things like financial websites or private browsing hasn’t been made clear.

“Before we launch broadly, we’ll continue to take feedback from early users in Copilot Labs, refine our safety measures, and keep privacy and responsibility at the center of everything we do,” Microsoft said of Copilot Vision.

Microsoft hasn’t said whether or not its upcoming Think Deeper feature is essentially GPT4-o, an AI that can ponder or reason. Think Deeper takes more time before responding. “Using the very latest in AI technology, it can help you with anything from solving tough math problems to weighing up the costs of managing home projects,” Microsoft says.

Copilot Voice will be available in the U.S., Canada, the U.K., and other regions, presumably today. Copilot Daily and personalization options are available today, too. Ditto for Think Deeper, which will be pushed to Copilot Pro users. But Copilot Vision is coming to Labs “in the future,” and to a limited number of users.