Saturday, March 19, 2022

Tesla Safety Score is Frustrating

Lots of digital ink has been spilled about Tesla's Full Self Driving beta, so I won't retread it here, except to say that I was hoping to be able to talk about my experience using it by now, and I can't, because I haven't. I have growing annoyance about the fact that I paid a substantial premium for a set of features I have yet to be able to fully use on my car, nearly 18 months into ownership. I do use Autopilot extensively, and some of its features were only available if you opted for the FSD upgrade, so it's not a completely wasted investment, but it's also been 6+ months since I requested access to the beta without getting it. 

What "requesting access to the beta" actually means is that I agreed to let Tesla grade my driving to prove that I'm a safe enough driver to be a good beta tester. It's sort of like one of those insurance company nanny widgets that you plug into the OBDII port to get a discount because you drive fewer miles or more safely than the average driver in your area. The FSD beta was initially opened to drivers with a perfect 100/100 safety score, and supposedly Tesla decrements the minimum acceptable number periodically when they want to let another tranche of people into the beta. I'm seeing things online that they're at 98 or 97 right now. But I still have no FSD beta access, so let's review what this means in real-life driving situations. 

Tesla's Safety Score is a daily score that contributes to a 30-day rolling average, which is the figure that Tesla looks at to determine who is a safe enough driver to be eligible for FSD beta. The cynic in me thinks that this is actually Tesla's way of making sure that those with more consistently complex driving environments don't get access to FSD and end up having a bad experience when the limitations of the relatively stupid robot that people are expecting to drive like a human are exposed. Anyway, my safety score tends to be in the 96-98 range. I've managed to get it to 99 periodically, but apparently not for long enough to qualify for the beta. There are 5 categories, and each is graded as green/yellow/red (increases your score/improving this will increase your score/decreases your score) based on your performance against an opaque figure Tesla calls "fleet median". I call it opaque because I don't know if it's the entire Tesla fleet or just those actively trying to qualify for FSD beta but not in the program yet, or if it includes those using FSD, or what geographic area is included - US only vs worldwide, etc. The app says "Median value for the Safety Factor based on the vehicles used to calibrate the safety score" which makes me think maybe it's none of those, and instead a carefully-selected control group of people who are also capable of consistently driving like the EPA range test group - the safest, most conservative few percent of drivers in the easiest traffic environments. Below is each category, and some thoughts about why I am where I am.

  • Forward Collision Warnings (per 1k mi)
    • This cares a lot about closing speed vs speed of the vehicle in front of you, and whether you're still holding the accelerator rather than braking to adjust, but it's a bit aggressive. I would be willing to own it if this was a situation where I wasn't paying attention and almost rear-ended someone, but that's not what happens in my case. Almost every one of these dings I've had are times where the warning triggers as I'm reacting to whatever set it off, which is frustrating, because I'm being graded on my reaction time rather than how safely I'm driving. Plus this one often goes hand-in-hand with a ding in the next category, so often it's a double-whammy. Given the amount that I drive, I almost always have one of these in the 30 day history, but usually not more than 1.
  • Hard Braking (>0.3g)
    • Assuming that you're not driving aggressively (accelerating hard and then braking hard) and are trying to drive smoothly, there are two main ways that this gets triggered, and in my opinion both are a bit unfair to use as an indicator of unsafe driving, since in most cases, it's exactly the opposite. Either you're braking hard to adjust for sudden traffic slowdowns, someone who does something you're not expecting, an obstacle in the road, etc. or you're trying to use regen braking/one pedal driving as much as possible and get caught by the fact that (as I discussed in a previous post) the regen braking level is not consistent and have to apply the friction brakes to slow down faster. This doesn't count when Autopilot is active, so there's another incentive to use Autopilot as much as possible. 
  • Aggressive Turning (>0.4g)
    • These cars have a low center of gravity, so they handle pretty well. Take advantage of that with even slightly spirited driving, and this metric is toast. So if we resign ourselves to having No Fun in the twisties, the other big trigger for this is when you are using Autopilot on secondary roads and it requires intervention - you're more likely to have to provide an aggressive, last-minute steering input to correct for Autopilot failing to turn aggressively enough on its own to properly follow the lane, or misjudging which lane it's supposed to be in as it crosses an intersection, etc. This would be a lot more fair if it didn't include incidents where autopilot was disengaged by the maneuver, but it's clear from my scores that this isn't what happens. 
  • Unsafe Following Distance (appears proportional to speed, but no quantitative metric is given)
    • This one is most frequently triggered when you're dealing with multi-lane roads and have to get into another lane because you need to turn or exit or pass, or if someone cuts you off. City traffic means aggressive drivers that will take an appropriate following distance as an opening to get in front of you, and the need to occasionally do the same to other drivers to force your way into a given lane. Consistent use of Autopilot helps, but Autopilot tends to be very conservative when dealing with requested lane changes in heavy traffic and so sometimes you have to override it to get where you need to be in time to make the desired maneuver. 
  • Forced Autopilot Disengagements (if you're not holding the wheel for long enough that Autopilot shuts itself off)
    • This is the only one I've had a perfect score on the entire time, because to trigger this, you have to have been not been applying sufficient pressure to the steering wheel for the better part of 60 seconds, then ignore both a visual and auditory warning. It's not difficult to stay on top of this one, and arguably this is a really good metric to disqualify people from participating in the beta because they're demonstrably not willing to properly supervise the robotic student driver to keep it from doing something stupid or unsafe. 
I see some things online indicating that there are ways to game the system. I'm not really interested in doing that, because I'm more interested in seeing whether it's actually possible to qualify in an area full of traffic, aggressive drivers and difficult situations if you don't cheat. Also while it's annoying that I still don't have access, this sort of reminds me of the reasons why I always stayed a release behind current when I had a Mac - there is definitely benefit to letting other people rush in on release day to be the unpaid beta testers and find all the broken stuff that shouldn't have escaped internal testing so that the system sucks less by the time I use it. 

No comments: