Dan Luu

@danluu.com

3603 Followers 13 Following

Active on Mastodon: https://mastodon.social/@danluu. Also trying https://threads.net/@danluu.danluu

In general, when I talk to software folks about testing, I'm coming from such a different place that they immediately look at me like I'm an alien, so let's talk about how we tested at this hardware company I worked for, Centaur, which informs my biases about how I like to work. Some of the things that we did that were or are unorthodox in the software world are:

Hired dedicated QA / test engineers, with testing being a first-class career path on par with being a developer
No code review by default
Virtually no hand-written tests
Constant testing via what programmers sometimes called property based testing, randomized testing, fuzzing, etc., although we just called those tests (hand-written tests were called "hand tests").
Large regeression test suite (3 months wall clock to execute on compute farm)
No unit tests

Just to give you an idea of the general structure, when I left (in 2013), we had about 1000 machines generating and running tests at all times for roughly 20 logic designers and 20 test engineers. This was on prem and the machines took up half a floor of the building we were in.

The general structure was that we had maybe 20% of machines running regression tests, and 80% generating and running new tests. Three months of regression tests is too much to gate commits on, so there was a much shorter list of tests that took maybe 10 minutes or so to run that people would run before committing. Those pre-commit tests would run on a special setup to run as quickly as possible, with overclocked machines that were the fastest machines money could buy, as well as a different simulator setup.

After the first three such calls this month, I was really starting to wonder what had changed. Had we accidentally published my phone number, somewhere? So when the fourth tech support call came through, today (which began with a confusing exchange when I didn’t recognise the name of the caller’s charity, and he didn’t get my name right, and I initially figured it must be a wrong number), I had to ask: where did you find this number?

“When I Google ‘Three Rings login’, it’s right there!” he said.

The ranking below is mainly based on how well vehicles scored when the driver-side small overlap test was added in 2012 and how well models scored when they were modified to improve test results.

Tier 1: good without modifications
Volvo
Tier 2: mediocre without modifications; good with modifications
None
Tier 3: poor without modifications; good with modifications
Mercedes
BMW
Tier 4: poor without modifications; mediocre with modifications
Honda
Toyota
Subaru
Chevrolet
Tesla
Ford
Tier 5: poor with modifications or modifications not made
Hyundai
Dodge
Nissan
Jeep
Volkswagen

These descriptions are approximations. Honda, Ford, and Tesla are the poorest fits for these descriptions, with Ford arguably being halfway in between Tier 4 and Tier 5 but also arguably being better than Tier 4 and not fitting into the classification and Honda and Tesla not really properly fitting into any category (with their category being the closest fit), but some others are also imperfect. Details below.

Please see link for text version of table

There's a common narrative that Microsoft was moribund under Steve Ballmer and then later saved by the miraculous leadership of Satya Nadella. This is the dominant narrative in every online discussion about the topic I've seen and it's a commonly expressed belief "in real life" as well. While I don't have anything negative to say about Nadella's leadership in this post, this narrative underrates Ballmer's role in Microsoft's success. Not only did Microsoft's financials, revenue and profit, look great under Ballmer, Microsoft under Ballmer made deep, long-term bets that set up Microsoft for success in the decades after his reign. At the time, the bets were widely panned, indicating that they weren't necessarily obvious, but we can see in retrospect that the company made very strong bets despite the criticism at the time.

In addition to overseeing deep investments in areas that people would later credit Nadella for, Ballmer set Nadella up for success by clearing out political barriers for any successor. Much like Gary Bernhardt's talk, which was panned because he made the problem statement and solution so obvious that people didn't realize they'd learned something non-trivial, Ballmer set up Microsoft for future success so effectively that it's easy to criticize him for being a bum because his successor is so successful.
Criticisms of Ballmer

For people who weren't around before the turn of the century, in the 90s, Microsoft used to be considered the biggest, baddest, company in town. But it wasn't long before people's opinions on Microsoft changed — by 2007, many people thought of Microsoft as the next IBM and Paul Graham wrote Microsoft is Dead, in which he noted that Microsoft being considered effective was ancient history:

A few days ago I suddenly realized Microsoft was dead. I was talking to a young startup founder about how Google was different from Yahoo. I said that Yahoo had been warped from the start by their fear of Microsoft. That was why they'd po…

mattnewton 1 hour ago | parent | next [–]

When I worked at Google I got re-orged into the same division as pixel / android.

My director wore an Apple Watch and had an iPhone for personal use, and I am pretty sure I saw an Apple Watch on my VP too. Nobody was expected to eat the dog food and so few did. This was crazy to me coming from Apple- I remember several internal sites would ask you to file a radar (bug report) on why you switched to chrome from safari if you opened them in chrome. So many crazy issues I saw and reported didn’t actually matter to many high ranking members of the pixel team because they didn’t use the devices after 5pm.

There is a lot of incredible talent in that team but I think Google needs a minor culture shift to compete with Apple here.

This is a summary of the publicly available documents on the 2011-2012 FTC investigation of Google's allegedly antitcompetive actions in search and ads, followed by a tech-focused analysis of the decision from someone who's worked at the two companies that are discussed in the most detail in the memos (Google and Microsoft), worked in search, and worked closely with ads teams on optimizing ads ranking algorithms. I've seen a number of law-focused and economics-focused analyses, but I haven't seen a tech-focused analysis in a level of detail I find satisfying. In particular, a number of key arguments in the memos rely on evidence and inferences that would've seen implausible to someone who was familiar with tech, which I haven't seen discussed.

The law-focused and economics-focused analyses tend to avoid digging into this and, while there have been some articles written about tech errors for a lay audience, they've tended to explain that the inferences that were made were wrong in r...

@ZTGallagher • 1mo ago

I know a guy who was a QA tester for Obsidian Entertainment working on Neverwinter Nights 2. At the end of development, they invited all the QA testers to the parking lot for a celebration party.
There was no party, they disabled all their keys when they got out there, and told them they were all fired. And that was it...

Table showing that many websites are unusable with devices that are commonly used in non-rich countries.

See link for details.

$Intel Processor Instability Causing Oodle Decompression Failures RAD has become aware of a problem that can cause Oodle Data decompression failures, or crashes in games built with Unreal. We believe that this is a hardware problem which affects primarily Intel 13900K and 14900K processors, less likely 13700, 14700 and other related processors as well. Only a small fraction of those processors will exhibit this behavior. The problem seems to be caused by a combination of BIOS settings and the high clock rates and power usage of these processors, leading to system instability and unpredictable behavior under heavy load. As far as we can tell, there is not any software bug in Oodle or Unreal that is causing this. Due to what seem to be overly optimistic BIOS settings, some small percentage of processors go out of their functional range of clock rate and power draw under high load, and execute instructions incorrectly. This is being seen disproportionately in Oodle Data decompressi...$