Siri for Self-Drive Cars: Genius or Patenting the Obvious?

You knew Apple wasn’t going to stay out of the car businesses. Their latest plan is not to build a complete car – it takes skill Apple doesn’t yet have in-house – but to be the concierge and brains of a self-driving car. In other words, Siri for Everything in an autonomous or semi-autonomous car, and not just Siri/CarPlay for music and navigation we have today.

Details are in Apple’s patent application US20200026288, “Guidance of Autonomous Vehicles in Destination Vicinities Using Intent Signals,” which was made public Thursday. In other words, you tell Siri what you want to do generally – find a barbecue joint, or the mall entrance nearest the food court – and Siri would figure out what to do and where to park.

Some of the inferencing algorithms based on a back-and-forth with the (non) driver, plus gestures holding and pointing the phone, seem pretty smart. But we doubt Apple is going to lock up all rights to a smartphone that listens to the passengers, asks a follow-up question, gets more refined guidance, then tells the car where to go, or figures how to pick out the best Asian fusion restaurant 90 minutes into your drive.

‘A Non-Trivial Challenge’

Apple’s patent application was originally filed Aug. 2, 2019, and unsealed this week. Broadly, Apple wants a patent to direct autonomous vehicles using a combination of voice commands, gestures (with your hands, the direction of your face or eyes, or tipping / pointing the phone), and touchscreen gestures. Apple says of its application:

This disclosure [patent application] relates generally to systems and algorithms for using various types of signals from occupants for controlling the motion of autonomous or partially autonomous vehicles. ‘

… Given the multiplicity of choices that are typically available with respect to vehicle trajectories in real-world environments, occupant input or guidance with regard to selecting vehicle trajectories (without requiring tradtional steering, braking, accelerating and the like) may be extremely valuable to the motion control components of such vehicles. However, providing interfaces for such guidance which are intuitive and easy to use, especially within environments such as parking lots for which detailed and/or accurate mapping data may not be available, may present a non-trivial challenge.

The figure above is a flow diagram, Apple says, illustrating aspects of how the car, phone, and passenger might interact via “intent signals.” You know where you want to go generally (“the mall”) but you don’t know exactly which parking spaces are available. (Hashtag that one #firstworldproblem.) Apple and Siri will fix that.

The “navigation manager” starts with general ideas such as you might want to park near the main entrance of the mall. That’s where the passenger and phone interact to find what you really intend to do, that is which store or section of a store you want to visit.

Park Closest to Zumba Class

Another part of the application shows how the driver and car would interact. When you’re close to the ideal spot to park but there’s actually, say, two spaces available, Siri can ask which one, and the driver/passenger can respond with a gesture of the phone in the direction of the preferred spot and the phone knows the direction through its compass and accelerometer. Or a camera in the car tracking the driver’s or lead passenger’s eye movements, sees where the driver’s hand points, or even the direction the eyes take.

Some of the for-instances seem intended for Californians who don’t want to park the Tesla 150 feet from the mall entrance when there’s AI out there that could find you the free parking space only 130 feet away. That’s important: It saves you unnecessary steps en route to Zumba class.

There’s the usual “find me some coffee” concepts, or figuring how to park near the garden center at Lowes (looking for bunches of shrubs and small trees is an algorithm I’ve used personally). You might also be concerned about how much power Apple has to direct your purchases. We didn’t see anything in Apple’s patent application that would let you plug in your own recommendation engine and set of preferences; your definition of “upscale Mexican” in California might not be El Torito even if that’s where Apple wants you to go. (An issue that applies to all cars that recommend services.)

On the other hand, if you’re planning to have the car parallel park in an urban area, you might prefer to be on the side of the street where you have to cross (as a pedestrian) against heavy traffic, or it’s raining and you want the quickest dash into the shop. Assuming there’s even one free parking space in town, let alone two on the same block.

Are Others Now Solving the Problems Apple Posits?

I’ve written in the past that when you drive to, say, M&T Bank Stadium to see the Baltimore Ravens play at The Bank (as the stadium is called), you really want directions to E Lot if that’s closest to your seat, or you’d like Siri to negotiate the best price for parking within a 10-minute walk, and make sure that includes private lots not controlled by the stadium, and you want to make sure Siri is getting you the best price, not just collecting a spiff for Apple.

Apple assumes many of these bits of drill-down information are unknowns. But certainly, the mall parking lot and its parking lines have been mapped, even if they’re not yet on Google Maps (let alone Apple Maps). Drones or cameras looking down from the mall buildings could tell which exact parking spots are free today when you pull in. When cars have 5G telematics, all that will be in the head unit already, accurate to the minute.

Overall, Apple has lots of good ideas in its patent application. It remains to be seen who might challenge the application on the grounds that some of Apple’s concepts are thoughtful but either obvious or already in use, generally, under different terms. Who hasn’t, as a passenger, pointed out an open parking spot while holding the phone in your hand and used it as part of your gesture? In-car cameras are already tracking the driver’s position in the seat, as well the hand gestures (BMW does that now), and many are tracking the driver’s head position and even direction of the eyes to measure distraction and wakefulness.

This project may be Apple’s fallback to building its own car. (That and Apple CarPlay, which is wildly successful.) From 2014 to 2019, roughly, Apple’s Project Titan was a ground-up autonomous, electrified vehicle project. Apple found out that building a car is enormously complex, there are regulatory hurdles to clear far tougher than for phones or PCs, and you can’t build a world-class auto factory in a couple of years. Apple also found out that not everyone wants to run a contract factory for Apple, including BMW and Daimler, and if there was an agreement, divorce court would have followed closely. Too many egos and everyone would want the final say.

Continue reading

Tesla Built a Supercomputer to Develop Camera-Only Self-Driving Tech

Tesla is talking about what it sees as the next leap in autonomous driving that could do away with lidar and radar, leaving self-driving cars to get around with regular optical cameras only.

Tesla Rolls Out $200 Monthly Subscription for ‘Full Self-Driving’

Some vague language on Tesla's part means that vehicles marketed as having full Autopilot capabilities might need an additional $1,500 hardware upgrade to use FSD.

Elon Musk Says Tesla Full Self-Driving Is Almost Ready for Release

Musk says the company is finally set to release the Full Self-Driving (FSD) update for compatible Tesla vehicles in the US, but this isn't the first time he's said that. If it's true this time, the software will begin rolling out at midnight on Sept. 10 to beta testers. If all goes as planned, the final launch could be just a few weeks later.

Walmart is Using Self-Driving Trucks in a 7 Mile Delivery Loop

It's the first time ever that an autonomous vehicle company has removed drivers from a delivery route's middle mile.