Lyncredible

Calling dibs on DIBS

Mon, 30 Oct 2023 00:00:00 +0000

I joined Uber in October 2015 to work on the brand-new Driver Incentives Backend System. Jonah Cohen, my soon-to-be manager, called dibs on naming it DIBS just a few weeks before I arrived. Fast forward eight years, and now I am calling dibs on recounting the tale of DIBS¹.

Driver Incentives and Carrots

2015 was the heyday of price wars in the ride-sharing market. Most riders likely encountered aggressive promotions and jaw-droppingly low prices, clearly underwritten by the platforms themselves. However, the rider subsidies were mere shadows compared to the incentives lavished² upon drivers. At first, these incentives served to lure drivers, ensuring ample supply aligned with anticipated high demand. But as competition intensified, such incentives evolved from a recruitment tool to an essential strategy for driver retention³.

Uber operated by cities, with driver incentives locally managed by each city’s operations team⁴. These incentives were structured as weekly campaigns for drivers. A typical campaign might state: earn an additional $1,000 next week if you complete a total of 40 trips either from or to downtown Manhattan, between 7am and 9am or between 4pm and 7pm, and from Monday to Friday.

Like any other great business software, driver incentives at Uber started as spreadsheets. Some tenacious Driver Operations people, or DOps, started the program by emailing campaign details to drivers in their cities at the beginning of each week. When the week ended, they wrote SQL queries⁵ based on the campaign rules to count qualifying drivers and calculate earnings. The incentives were paid out by uploading CSV files with driver UUIDs and dollar amounts.

Sometime in 2014, the Driver Incentives product team was formed at Uber’s headquarters in San Francisco. They built out the software to manage the campaigns from creation to payout. Named Carrots⁶, the software was a game-changer to DOps who routinely worked 12+ hours a day and 6+ days a week. Here’s the magic it brought⁷:

One-Stop Management: DOps could now juggle campaigns all in one web app. Say goodbye to the chaotic dance of copy-pasting across documents, spreadsheets, and endless chat windows.
Instant Visibility: Whip up a campaign, and bam! It’s right there in the Uber Driver App. Emails? Who needs ‘em!
Error-Free & Unattended Calcs: Qualification and earnings got the automated touch, thoroughly checked for any sneaky bugs. So, no more clumsy fingers messing up SQL or twiddling thumbs waiting for queries.
Easy Peasy Payout: Everything’s integrated! DOps just hit a button after a quick peek at the pre-payout report. CSV uploads were so yesterday.

From Incentica to DIBS

There was just one hiccup: Carrots used the same Vertica⁸ database that DOps ran their SQL queries on. The team bought the beefiest Vertica box on the market, affectionately dubbing it Incentica. However, even this powerhouse was nearing its breaking point. Vertica wasn’t built for horizontal scaling. It could not house all of Uber’s data or deal with the erratic, fast-growing query demands.

Now, here’s a quirky thing: Uber’s idea of a week ended at 4am every Monday, based on each city’s local time. This meant that, for the most part, Incentica had a leisurely pace throughout the week. However, come Monday morning in East Asia, queries started flooding in. And as the clock struck 4am in city after city, the floodgates opened wider. The real kicker? The Carrots team had to constantly babysit these queries. So, their workweek effectively began at noon on Sundays to align with 4am Monday in Beijing. And if you were the unlucky on-call engineer, you were probably pulling an all-nighter on Sunday to ensure Incentica could withstand the onslaught from the Middle East, Europe, and eventually the Americas.

Then there was the ETL pipeline responsible for transferring data from the primary trips database to Incentica. The pipeline was delayed and lossy due to legacy constraints, which meant the calculations could still end up off the mark despite all the waiting and babysitting.

Hailing from Seattle, the DIBS team was tasked with rectifying these issues. Our solution seemed straightforward. DIBS would harness streaming-based aggregation to update calculations for each driver and every campaign in real time. Picture this: hundreds of worker nodes running Kafka consumers, all eagerly subscribed to trip completion events. The workers would coordinate to partition and process the events, making the system infinitely scalable⁹. The streaming architecture not only distributed the workload evenly throughout the week, eradicating those dreaded Monday query spikes, but it also had a cherry on top: drivers could monitor their real-time progress in the Uber app as the week unfolded. As an added safety net, We opted to shield against duplicate event deliveries¹⁰ by recording all processed trip UUIDs for each driver and campaign. Looking back, the design might seem deceptively simple or even naive, but guess what? It did the trick.

We called dibs on the first full week of 2016 for DIBS’ prime-time debut.

Scaling to sound sleep

And that was the week when I paid my fair share of Sunday all-nighters.

DIBS actually operated smoothly throughout Sunday night. Yet, both I and Greg, the Carrots on-call engineer, kept getting questions from DOps across various cities. They lamented the absence of calculation results for their campaigns. A closer look revealed that these campaigns had remained in the draft state all week. DOps were supposed to click the “launch” button before the week started to kick off streaming aggregation in DIBS. Their oversight was understandable; before DIBS, they had grown accustomed to waiting until week’s end to click “launch”. This was due to the batch-oriented nature of the Incentica solution, which could only process results after the week concluded.

That night I learned more about Kafka than the previous few months combined. Greg and I scrambled to “launch” the campaigns and then rewind the Kafka consumer offset to the previous Sunday. To expedite things, we added a ton more workers¹¹ to the Kafka consumer group, enabling DIBS to churn through the entire week’s trips in just a couple of hours. Ultimately, we managed to complete calculations for most campaigns just a few hours past the Monday 4am cut-off. The only challenge left? Tackling those sleep-depriving Sunday nights.

Going back in time

In the week that followed, we worked with DOps to launch campaigns on time. Every DOps individual we spoke to was understanding and sympathetic. They acknowledged the advantages of punctual campaign launches, such as allowing drivers to view their real-time progress towards campaigns within the app. While they wanted to help the engineers stop dreading about their on-call shifts, they also politely requested a buffer. What if a campaign was launched a few minutes late? Or if a mistake in the campaign rules was discovered later in the week?

That turned out to be a fascinating engineering problem for the DIBS team to solve. We built an automated backfill solution for late campaigns. To achieve this, we maintained two kafka consumer groups, both processing the same trips dataset. The primary one always processed trips in real time, while the secondary one was responsible for backfilling late campaigns¹². When a campaign was launched late, the backfill consumer group would automatically rewind its offset to match the campaign’s intended start time, and backfill any trips that had happened prior to the late-launch. We also introduced various optimizations, such as batching multiple late campaigns for backfilling and meticulously filtering out irrelevant trips, ensuring the backfill process was swift and efficient.

The automated backfill was a smashing success. Every Carrots campaign, even those with delayed launches, was promptly calculated and paid out with little human intervention. Both Carrots and DIBS engineers enjoyed 8 hours of sound sleep during their on-call shifts.

At least that was the case until DIBS launched in China two months later, when every assumption we made about the system had to be changed.

Everything was bigger in China

DIBS made its debut in China around March 2016, right when the rivalry between DiDi and Uber was escalating from fierce to downright incendiary. With a dense population and aggressive promotions/incentives, Uber was clocking millions of trips every week in each of China’s major cities. This volume dwarfed even the busiest Uber markets elsewhere in the world. At the peak, Shanghai saw ten times more trips than New York City in a typical week. The sheer scale would have completely destroyed Incentica. Fortunately, for DIBS, accommodating the colossal Uber China traffic was just a matter of deploying more worker nodes.

The real challenges was, interestingly and somewhat expectedly, in backfills. Uber’s DOps in China adopted a strategy of intentionally delaying campaign launches in Carrots until the week’s end. This tactic was devised to keep DiDi in the dark about Uber’s campaign rules, preventing them from undercutting us¹³. Instead, they assured drivers that Uber would either match DiDi’s incentives or provide superior ones, but the precise formula would remain unknown until the week concluded.

This approach wreaked havoc on DIBS. Our primary, real-time consumer sit there all day doing nothing, while the secondary, back-fill consumer bore the brunt of the workload, meticulously processing every single trip for each campaign in the few hours following each week’s close. The surge in compute and network demand during backfills was staggering, scaling to levels about 1,000 times higher than the usual rate.

So we set out to scale DIBS for China. The key insight was that most changes to campaign rules had no impact on the partial aggregation logic. For example, an incentive of $1,000 demanding 100 weekly trips to qualify would utilize the same trip counter as another incentive of $2,000 requiring 180 weekly trips. This meant DIBS could do real-time aggregation of the partial result, like the trip count, and reuse them for varying campaign rules, as long as the incentive structure remained consistent.

Each week we launched dummy campaigns¹⁴ featuring common incentive structures, performed partial streaming aggregations throughout the week, and when the week wrapped up, reused the partial aggregation results for the actual campaigns launched by DOps. We even automated the process of detecting prevailing incentive structures and initiating these dummy campaigns. At last, our pursuit of uninterrupted slumber was realized.

The trip that existed before it didn’t

A mysteriously missing trip broke our peaceful dream later that spring. A DOps reached out to report that a driver was unfairly deprived of their incentive. According to DIBS, they fell short by a single trip. That was baffling. Uber’s Kafka setup ensured “at-least-once” event delivery, implying DIBS should never under count¹⁵. If this driver was denied incentive due to a missed trip, how many other drivers might have been similarly shortchanged?

To unravel this, we manually initiated a backfill for the contentious campaign. Somewhat unexpectedly, the backfill detected the elusive trip and confirmed the accurate trip count for the concerned driver. Bolstered by this discovery, we triggered backfills for all campaigns from the previous week, comparing the results pre and post-backfill. The scrutiny unearthed a minuscule yet consistent pattern of omitted trips.

There was an undeniable glitch within the primary, real-time streaming consumer. It seemed as though certain trips were momentarily non-existent in real-time but subsequently found their way into the system, only to be captured during backfills. The pressing question was, why?

The database that was not Kafka

To answer that question, we delved deeper into the origins of our streaming data. As it turned out, DIBS was not pulling data from a Kafka topic. Instead, it was sourcing from a Schemaless table. Schemaless was Uber’s in-house online datastore, designed for vital business data like rider, driver and trip details. Built on top of sharded MySQL tables, Schemaless strictly adhered to append-only writes, which was tailor-made for stream processing tasks akin to DIBS. To further its compatibility, the Schemaless team ingeniously developed client-side stream-consumer libraries that mirrored Kafka’s API. It is no wonder that this whole blog post, up until this point, depicted DIBS as if we were using Kafka the whole time.

The missed trip broke that abstraction though. To understand why, we need to first dive into the data schema in Schemaless.

The Schemaless schema

All Schemaless datasets were stored in MySQL with a physical schema that could be depicted as follows¹⁶:

Column Name	Column Type	Note
`ID`	`BIGINT`	Primary key with auto-increment
`UUID`	`CHAR(36)`	Globally unique identifier, indexed
`Created`	`TIMESTAMP`	Timestamp auto-populated upon creation
`Payload`	`MEDIUMTEXT`	Serialized JSON payload

If we were to store a Uber trip object into a table like this, we would serialize the object into a JSON format, generate a UUID for the object, and append the resultant row to the table. The integer ID would be auto-generated by MySQL thanks to auto-increment. The Created timestamp will be auto-set with UTC_TIMESTAMP(). Here is a sample row in columnar format for clearer visualization:

Column	Value
`ID`	`42`
`UUID`	`c160a108-91c5-4af4-8133-727c36f1e277`
`Created`	`2023-10-29T19:01:04.146Z`
`Payload`	`{"city_id": "xxx", "amount_in_usd": 30, "distance_in_km": "18", ...}`

Note this physical table was just one shard of many. Schemaless partitioned each dataset among many underlying physical tables¹⁷. The globally unique UUID functioned as the pivotal sharding key. The integer IDs were unique within each shard, but were expected to overlap across different shards.

The “Kafka” consumer

The “Kafka” consumer workers streamed data from Schemaless by taking advantage of the append-only nature of the underlying tables. Each shard was earmarked for one exclusive worker¹⁸. This worker ran an infinite loop to consistently poll for fresh rows to process. The loop’s mechanics looked like the following:

Initialize per_shard_offset to 0.
Run the SQL query: SELECT id, uuid, payload FROM charges WHERE id > $per_shard_offset to fish out new rows.
Process the rows retrieved from Step 2 and update per_shard_offset to the highest ID identified among the new rows.
Circle back to Step 2.

At first glance, the logic appeared impeccable. Since each shard was singularly managed by one specific worker, there wasn’t any room for race conditions between two different workers. So, with the mystery deepening, we documented our quandary and sought insights from the Schemaless team. Upon reviewing our notes, Rene Schmidt, the esteemed architect behind Schemaless, instantly pinpointed the issue, stating, “It’s a race condition due to Repeatable Read”.

The race condition

Imagine two new rows being appended to the table concurrently as the DIBS consumer ran its SELECT query. This happened all the time because many trips were completing around the world at any given time. It was not rare for two to end up in the same shard and to be written to the same MySQL table simultaneously. Suppose the last known highest ID was 42 just before these events. The table below outlines a possible sequence of events:

Time	Rows	Appender #1	Appender #2	DIBS consumer
t0	1-42	Idle	Idle	`per_shard_offset` = 42
t1	1-42	Begin Tx	Idle	Idle
t2	1-42	Append #43	Begin Tx	Idle
t3	1-42	Idle	Append #44	Idle
t4	1-42, 44	Idle	Commit Tx	`SELECT ... WHERE id > 42` yields #44
t5	1-42, 43, 44	Commit Tx	Idle	Update `per_shard_offset` = 44

From the sequence, it’s evident that although Appender #1 initiated its transaction before Appender #2, the latter managed to append a row (with ID 44) and complete its transaction earlier. Unfortunately, the DIBS consumer executed the SELECT query post Appender #2’s commit but pre Appender #1’s commit, getting only row #44 in return. This means in its subsequent scans, DIBS would bypass any rows with IDs below than 44. As a result, row #43 slipped through the cracks, remaining undetected from the DIBS consumer’s point of view.

The workaround

The root of this race condition traced back to the default transaction isolation level of Repeatable Read in MySQL / InnoDB. It could be prevented by changing the transaction isolation level to Serializable. In that mode, the SELECT transaction would create a range lock on ID, essentially locking the range (42, +∞) in our earlier example. That lock would conflict with the single row locks established by appenders, such as a single row lock of 43 by Appender #1 in the same example. This means the SELECT transaction would wait for all in-progress append transactions to finish before executing the query, thus avoiding the race condition.

Yet, switching to Serializable mode was not free. It would incur a huge performance penalty, forcing many transactions to execute in serial rather than concurrently. The impact would reverberate any production systems interfacing with the database. Given the resulting dramatic plunge in throughput, this approach was not viable.

Our eventual solution was a bit of a workaround tailored to the Schemaless stream consumer library. The SELECT query was modified to exclude recent rows by examining the created timestamp:

SELECT id, uuid, payload
FROM trips
WHERE id > $per_shard_offset
  AND created < TIMESTAMP(
    DATE_SUB(UTC_TIMESTAMP(), INTERVAL 2 MINUTE))

By imposing a condition for the created timestamp to be at least 2 minutes old, the SELECT query effectively ignored all rows appended by recent transactions, and punted them to a subsequent loop iteration. The choice of a 2-minute buffer was enough to sidestep the race condition without adding too much delay. This adjustment avoided degrading overall throughput of the entire system, incurring only a minor delay on the stream consumer side.

DIBS until Teleportation

DIBS ultimately lived up to its promise of lossless aggregation. Incentica could finally be retired.

The entire DIBS team flew down to San Francisco to attend the joint Incentica-shutdown party. While in town, I had the chance to engage with several members of Uber’s China Growth organization, the largest user of DIBS. I was surprised to discover that numerous teams within China Growth had use cases mirroring Driver Incentives. Each program revolved around the theme of giving rewards to eligible riders or drivers upon meeting certain criteria.

On the following day, a TK¹⁹ Q&A with China Growth was on the agenda, so I asked Travis a question: “It seems almost half of China Growth are doing various forms of promotion and incentive programs. Assuming there will come a day when we stop the aggressive spending, what innovations should we be focusing on now to differentiate our product when that time arrives?”

TK’s answer was memorable: “Some day Uber will replicate Star Trek’s Transporters²⁰ and teleport people from Point A to Point B. But until then, we will keep doubling down on incentive programs because everyone else is spending like crazy.”

The Q&A was hosted within the confines of Uber’s headquarters at 1455 Market Street. Personally, its interior design reminded me strongly of the USS Enterprise. Beyond sci-fi analogies, it was evident that TK was alluding to the future of self-driving cars. I had always been somewhat skeptical²¹ about the strategy of betting on self-driving technology while bleeding cash in the core business. TK’s response did little to allay those concerns.

While in SF, I also had the privilege to present a company-wide tech talk on DIBS. As expected, the automated backfill solution was a hot topic of discussion. It was unconventional, caused spiky traffic patterns, and strained both upstream and downstream systems. I recognized and validated the concerns, but also emphasized the short-term necessity of the solution as Uber was spending billions of dollars on driver incentives per year. The vision was to transition to a more robust solution once the Big Data Platform²² was operational. “Or perhaps, when Teleportation becomes a reality,” I mused privately.

DiDi called dibs on us

Neither was fast enough. On August 1st, 2016, less than one year after the start of the DIBS project, Uber sold its China operations to DiDi, receiving a minority stake in the latter as part of the deal.

Most of the sophisticated features we had developed for DIBS lost their relevance overnight. Within a month, ownership of the DIBS system was transferred to the Carrots team. Subsequently, every member of the DIBS team embarked on new ventures within Uber.

Throughout the DIBS project, there had been internal reservations regarding the overarching strategy of the price war. However, such concerns never impeded our dedication to crafting the best possible solution within our set parameters. With the change in direction, there was no need to dwell on past accomplishments. After all, business is business, or in Basta’s words, “No sacred masterpieces”.

I drew inspiration from Matt Basta’s captivating piece titled No sacred masterpieces, which recounts how he “built Excel for Uber and they ditched it like a week after launch”. Intriguingly, both tales unfolded during the same time frame centered around the same driver incentives program. Yet, surprisingly, even after spending over 5 years together at Stripe following our Uber stints, Basta and I have never crossed paths.
To grasp the scale of the driver subsidies, consider this: drivers often earned more from incentives than they did from ride payouts.
I provided a theory on the intense price war in a previous post.
In many ways, Uber’s early operations echoed the franchise model of McDonald’s. City managers acted as the franchise owners, while the Uber app, internal tools, and playbooks served as the secret recipes for whipping up Big Macs. Many innovations sprouted from these city operations team, only to be replicated and scaled later with the help of the central teams at the headquarters.
If ChatGPT had been available back then, the DOps would surely have used it to write SQL queries.
In later years, given the broadly critical stance of the media towards Uber, the name Carrots might have sparked a minor PR hiccup. Fortunately, it didn’t. The team simply saw it as an easily recognizable codename.
The following marketing content was mine, but the thrill was ChatGPT.
Vertica, for those unfamiliar, is an analytics database that’s designed for very fast queries over very large sets of mostly read-only data.
The system was horizontally scalable up to the number of virtual partitions in Kafka, which was 4,096 at the time.
We also contemplated using Bloom filters in the design to quickly detect duplicates, but they were never implemented. Even for the most industrious driver, there just weren’t enough trips to justify it.
We probably scaled out the number of Kafka consumer workers by 10x, although my memory is blurred. This was feasible because, first, the trips data was already spread across 4,096 virtual partitions, and second, Uber’s compute infrastructure was adept at elastically scaling containerized workloads. The first point allowed us to scale out horizontally as long the number of worker nodes did not exceed 4,096. The second point was even more impressive considering that Will Larson’s team built the compute infrastructure in Uber’s own physical data centers.
Little did I know back then, but it turned out to be an anti-pattern to backfill by rewinding streaming consumer offsets. The proven way to do this was called a Lambda Architecture, where backfills ought to be performed using batch data processing techniques like Apache Spark. It did not matter though, because Uber had yet to build its lossless ETL pipeline to send trip data into its Hadoop-based Big Data Platform.
Basta alluded to the same competition dynamics in his article, where Excel formulas to calculate incentives must be kept confidential because Uber was worried that DiDi sent spies to intern at Uber and steal such secrets.
The dummy campaigns were not visible in the Uber Driver app.
As is mentioned earlier in this post, DIBS recorded trip UUIDs to protect against duplicate event deliveries to avoid over-counting.
The physical schema was simplified to focus on the race condition. Check out Schemaless on Uber’s Engineering Blog for the complete definition.
DIBS primarily consumed from the trips dataset in Schemaless, which had 4,096 shards.
The converse was not true. Each worker could stream from multiple shards.
TK is the preferred moniker for Travis Kalanick, co-founder and then CEO at Uber.
See Transporter (Star Trek) on Wikipedia.
See my previous post Uber had no upside which covered my doubts about the potential impact of self-driving technology on Uber’s business.
Sitting in the audience, Zheng Shao was one of the many who asked questions about the efficiency and scalability of the DIBS backfill. He would go on to build Uber’s Big Data Platform, making it possible for DIBS to migrate to a much more scalable Lambda Architecture.

New hire hacks

Mon, 25 Jul 2022 00:00:00 +0000

Summer time means I am going to help quite a few new grads to ramp up. This is highly important work because the new hires will substantially increase my team’s total throughput over time, but it is also super time consuming. Over the last few years I have learned a few hacks to make the process more effective.

Spin-up buddy

At Stripe, we appoint a spin-up buddy for each new hire. The spin-up buddy is an engineer who is already experienced on the team. They are the dedicated person to help the new hire ramp up. Generally, the spin-up buddy should serve as the information filter for the new hire, so that the new hire can focus on the most important stuff to learn and practice. Specifically, the spin-up buddy is responsible for a few concrete activities below:

Create a spin-up guide for the new hire. The guide lays out a tailored onboarding plan for the new hire. It includes the people to talk to, the documents to read, the technologies to learn, and the spin-up tasks to complete. The guide contains concrete week-by-week goals for the first 5~6 weeks, and also sets up some directional targets for the first 3 months.
Scope out a few bite-sized spin-up tasks for the new hire. These tasks are typically bugs or tiny feature requests from the backlog. Each of such tasks should be clearly defined and take no more than a couple of days to finish. The new hire can use these tasks to get their hands dirty and to have a sense of accomplishment early on.
Mentor the new hire on a slightly bigger project with some ambiguity. The definition of “bigger” can vary, but I generally find a sweet spot in the 2~3 weeks range. This gives the new hire an opportunity to apply their creativity to solving a new problem, which is another milestone worth celebrating for.

To make the spin-up buddy setup succeed, the buddy and the new hire need to work very closely. They can either be physically co-located, or, if working remotely, have frequent async message exchanges and sync pairing sessions.

I have found this setup to be beneficial for a few reasons:

It is scalable. The manager is no longer a bottleneck to ramp up new hires. The more already spun-up people you have on the team, the more new hires you can effectively spin up at the same time. The first-ever spin-up guide takes some effort to create, but subsequent ones only cost some incremental time.
It gives everyone opportunities to mentor other people. There is no leveling or seniority requirement for the spin-up buddy. They just need to be reasonably familiar with how the company and the team operates. Such mentorship experience is especially valuable for early-career engineers.

Hacking admin tasks

The various admin tasks can be tricky to get right. It is very common for new hires to be missing some arcane permissions or not invited to regular meetings. Below are a few hacks I practice to streamline such admin tasks:

Use email groups for calendar invites and access control as much as possible. Invitation to regular team meetings should be sent to the team email group. Permissions to access team resources like shared documents should also be granted to the team email group. This is straightforward to set up in Gmail. If your employer uses Outlook, I believe you can leverage distribution lists or security groups for similar purposes.
If your team has multiple email groups, e.g. foo-team and foo-user-newsletter, pick one group (e.g. foo-team) to host all team members, and subscribe that team group to other groups (e.g. subscribing foo-team to foo-user-newsletter, instead of directly adding people to both groups).
Use Slack user groups for channel membership. Create a team group and configure “default channels” for the group. Your new hire will be automatically invited to all configured “default channels” once they are added to the team group. This also works for private channels, which are usually annoying to manage. If your employer uses Microsoft Teams, I have no idea how Teams works¹. :(
Document the various admin tasks in the spin-up guide, and leverage the spin-up buddy to close the gaps.

That is it for now

Do you have any tricks to ramp up new hires? Please let me know. Happy spinning up to all new hires and their buddies!

Despite having worked for almost 6 years on Microsoft Lync, one of the predecessors to Teams, I have not used Teams at all. To my friends at Teams: let me know if this user group concept exists in Teams. Thanks!

Capital efficiency

Sun, 17 Jul 2022 00:00:00 +0000

The decision to work for a startup¹ can be stressful. Everyone wants to get rich, but it is an impossible task to accurately predict the trajectory of any given company. When I consult my mentors and friends, most of the advice I get is a bit handwavy, as is usually the case. I have spent quite some time thinking about this topic. Here is one important lesson I have learned: examine its capital efficiency when deciding to join a startup.

What is capital efficiency

Capital efficiency, as its name suggests, measures how efficiently a business uses capital. Loosely speaking, capital efficiency is defined as the ratio of money made over money spent during a certain period. It is obviously better to deploy $1 and make $10 in return, than to deploy $1 and make $2. In this context, the definition of capital is limited to tangible assets like cash, while intangible assets like human creativity and brand recognition are excluded.

There are much more formal definitions for capital efficiency. The formal definitions are not required to read this post, but I do recommend the following articles to learn more:

The Hype Factor by Dave Kellog
The Burn Multiple by David Sacks
The Venture Capital Efficiency Ratio by Sean Cantwell

Why I care about capital efficiency

The tech industry is defined by high-growth companies, and I certainly would like to work for a company that has a shot to become the next Amazon or Microsoft. This was why I joined Uber in 2015, although it did not go the way I had hoped. There are many explanations for why Uber has not achieved the high expectations². I don’t think there is a single correct reason. However, I do believe that Uber’s poor capital efficiency was a significant factor.

My argument is as follows. The high valuations of tech companies are betted on zero marginal cost. Such companies can spend most of their resources on upfront fixed-cost investments i.e., building great products. They can scale to an almost unlimited pool of customers rapidly and inexpensively. A poor capital efficiency, sustained for a long period, suggests that the marginal cost is not zero, but rather very high. This forces the business to pay its attention to the uncapped marginal cost, while the fixed-cost portion that fuels product innovation takes a backseat. That would be unfortunate for tech workers like me, whose skill set is concentrated in building products. Such skill set would not be as impactful and would not be valued as much in a business with high marginal cost.

We can turn our attention to the one of largest tech companies for example: Microsoft. The business model of pre-Internet Microsoft was primarily selling packaged software. The marginal cost of producing a DVD copy of Windows is almost zero, yet Microsoft could charge hundreds or thousands of dollars for a single copy. The profit from the high-margin packaged software business allowed Microsoft to invest massively and expand to many business lines. It is important to note that such investment was concentrated in the fixed cost of creating new products, rather than the marginal cost of replicating existing products. They started by selling BASIC interpreters and gradually moved to operating systems (MSDOS/Windows), productivity tools (Office), databases (SQL Server) and developer tools (Visual Studio), to name a few. Microsoft did all of this with no outside funding³.

On the other hand, capital inefficient companies are forced to spend a huge amount of money to maintain and grow its existing business. For example, Uber and Lyft could not differentiate their product experience, but had to resort to price wars to take market share. The implication here is that the product is fungible, and the tech workforce building the product is also replaceable.

Capital efficiency for startups

Once I was convinced capital efficiency is an important metric, the next question was how to measure capital efficiency for startups. Recall the loose definition above: capital efficiency is the ratio of money made over money spent during a certain period. Startups are private companies and do not report earnings. How could I know how much money a startup spent or made in any given year?

The answer is of course a big NO without insider knowledge⁴, but there is a reasonably good approximation: the fund-raising dilution ratio(DR). Imagine a startup raises $M in funding at a pre-money valuation of $V. All existing shareholders before this funding round is diluted by M/V * 100%⁵, and that ratio is a proxy for the company’s capital efficiency.

Let’s look at a couple of hypothetical examples:

Imagine Company A raises $1B at a pre-money valuation of $10B. The DR is 1B/10B * 100% = 10%, which means that all existing shareholders are diluted by 10%⁶.
Imagine Company B raises $100M at a pre-money valuation of $10B. The DR is 100M/10B * 100% = 1%⁷.

Both companies are valued at $10B before their respective funding round, but I would argue that Company B likely has a higher capital efficiency than Company A, because Company B’s DR is lower. Why is that? A company’s valuation is a function of the amount of money it expects to make in the future, while the amount of money being raised is an approximation of the money it will spend. Put the two together and you have a reasonable proxy to the company’s capital efficiency.

We can also go one step further and measure the Venture Capital Efficiency Ratio(VCER), which is defined as, for any given round, the pre-money valuation over total capital raised before the round⁸. It is the same idea but measured over all previous funding rounds cumulatively. Imagine a company is valued at $10B pre-money for a new round, and it has raised a total of $2B over all previous funding rounds, then its VCER is 5x at that round.

Here is how I think about the two metrics:

DR is a forward-looking metric for a single round. It shows how much of the company ownership is given away to new investors in exchange for the money raised to fuel future growth. DR predicts capital efficiency in future business growth. Lower DR is a proxy to higher capital efficiency.
VCER is a backward-looking metric over all previous rounds. It shows how much money the company has raised and possibly spent to grow the business and reach the current valuation. VCER validates capital efficiency in previous business growth. Higher VCER translates to higher capital efficiency.

Real-world examples

Uber

Date	Round	Amount	Valuation	Total Raised	DR	VCER
Aug 2009	Seed	200.0K		200.0K
Oct 2010	Angel	1.3M		1.5M
Feb 2011	Series A	11.0M	49.0M	12.5M	22.45%	32.7x
Dec 2011	Series B	37.0M	300.0M	49.5M	12.33%	24.0x
Aug 2013	Series C	363.0M	3.5B	412.5M	10.37%	70.7x
Jun 2014	Series D	1.2B	17.0B	1.6B	7.06%	41.2x
Dec 2014	Series E	2.8B	40.0B	4.4B	7.00%	24.8x
Jul 2015	Series F	1.0B	50.0B	5.4B	2.00%	11.3x
Dec 2015	Growth Equity VC	5.6B	62.5B	11.0B	8.96%	11.5x
Jan 2018	Late VC	1.3B	69.0B	12.3B	1.88%	6.3x
Aug 2018	Late VC	500.0M	72.0B	12.8B	0.69%	5.8x
Apr 2019	Late VC	500.0M	78.8B	13.3B	0.63%	6.2x
May 2019	IPO	8.1B	74.3B	21.4B	10.90%	5.6x

WeWork

Date	Round	Amount	Valuation	Total Raised	DR	VCER
Oct 2011	Seed	1.0M		1.0M
Jan 2012	Seed	6.9M		7.9M
Jul 2012	Series A	17.0M		24.9M
May 2013	Series B	40.0M		64.9M
Oct 2013	Series C	150.0M		214.9M
Dec 2014	Series D	355.0M	4.6B	569.9M	7.72%	21.4x
Jun 2015	Series E	434.0M		1.0B
Mar 2016	Series F	430.0M		1.4B
Oct 2016	Series F	260.0M		1.7B
Jul 2017	Series G	5.2B	20.0B	6.9B	25.80%	11.8x
Jul 2018	Late VC	500.0M		7.4B
Nov 2018	Growth Equity VC	3.0B	42.0B	10.4B	7.14%	5.7x
Jan 2019	Series H	1.0B	47.0B	11.4B	2.13%	4.5x
Mar 2021	SPAC IPO	1.3B	9.0B	12.6B	14.26%	0.8x

Zoom

Date	Round	Amount	Valuation	Total Raised	DR	VCER
Jun 2011	Seed	3.0M		3.0M
Jan 2013	Series A	9.0M	24.0M	12.0M	37.50%	8.0x
Sep 2013	Series B	6.5M	48.6M	18.5M	13.37%	4.1x
Feb 2015	Series C	30.0M	200.0M	48.5M	15.00%	10.8x
Jan 2017	Series D	100.0M	1.0B	148.5M	10.00%	20.6x
Apr 2019	IPO	356.8M	8.8B	505.3M	4.03%	59.6x

Stripe

Date	Round	Amount	Valuation	Total Raised	DR	VCER
Mar 2011	Seed	2.0M		2.0M
Feb 2012	Series A	18.0M	100.0M	20.0M	18.00%	50.0x
Jul 2012	Series B	20.0M		40.0M
Jan 2014	Series C	80.0M	1.8B	120.0M	4.44%	45.0x
Dec 2014	Series C	70.0M	3.5B	190.0M	2.00%	29.2x
Jul 2015	Series C	100.0M	4.9B	290.0M	2.04%	25.8x
Nov 2016	Series D	150.0M	9.2B	440.0M	1.63%	31.7x
Sep 2018	Series E	245.0M	20.0B	685.0M	1.23%	45.5x
Jan 2019	Series F	100.0M	22.5B	785.0M	0.44%	32.8x
Sep 2019	Series G	850.0M	35.3B	1.6B	2.41%	45.0x
Mar 2021	Series H	600.0M	95.0B	2.2B	0.63%	58.1x

A few notes regarding the data

Amount is the amount of money raised in the round.
Valuation is the pre-money valuation for the round.
DR is the dilution ratio for the round, i.e., the ratio of Amount over Valuation.
Total Raised is the cumulative total capital raised up until that round.
VCER is the Venture Capital Efficiency Ratio of Valuation for the round over Total Raised until the previous round.
All funding and valuation numbers are in US dollars.
The data is sourced from Dealroom and Crunchbase. Although I have worked at Uber and Stripe, I have only referenced public data when putting together this post. There is no guarantee if the data is accurate.
Only primary equity rounds are included. Debt rounds and M&A rounds are omitted. Secondary equity rounds are excluded because no new shares are issued in such rounds.
Extension rounds at the same valuation as the original rounds are combined into the original rounds to make DR calculation reasonable.
To simplify the result, I decided to omit data for Uber China’s funding rounds. As a result, Uber’s fund raising amount below is understated (because money raised for Uber China is not in the table), and hence Uber’s capital efficiency numbers are overstated.

My takeaways

It is typical for DR to start high - about 20% or higher for at least three of the four examples above. We don’t have valuation data for WeWork’s early rounds, but it is safe to assume the DR is also in that 20% range or higher. This makes sense because an early-stage company needs to make large upfront fixed-cost investments to create the product.
It is also typical for DR to trend lower in subsequent rounds. A low DR in these later rounds suggests that the company’s marginal cost to scale their business is low. That is a sign of a healthy business that is achieving economies of scale. The trend is most clear for Zoom’s data.
Conversely, a high DR in later rounds indicate high marginal cost. It implies problems in reaching economies of scale, as is shown in the trajectory of Uber and WeWork after the colossal rounds with billions of dollars raised. Note the absolute number of the DR can still be lower since the valuation is typically higher at these rounds. For example, Uber’s “Growth Equity VC” round is a good one to examine.
VCER is a more reliable metric over the long term. A VCER that consistently trends downwards is a sign of a business that has trouble reaching economies of scale, as is illustrated in Uber. A VCER that goes below 1x basically means earlier investors are close to being wiped out, which is what happened to WeWork. On the contrary, a VCER that steadily trends upwards indicates high capital efficiency as the business achieves economies of scale, as is the case of Zoom.
For both metrics, it is useful to examine both the raw number and the trend over time.

At the end, I want to note these capital efficiency metrics should be used as references to help understand startups. They are useful but do not tell the entire story. There is no replacement for analyzing the business, the people and the product. Personally I had lots of fun tinkering with these numbers and forming the idea in this post. As a parting gift, I would like to leave you with the IPO-round numbers for some of the most successful companies in the tech industry:

	Microsoft	Amazon	Google	Facebook
IPO Date	Mar 1986	May 1997	Aug 2004	May 2012
Total raised pre-IPO	1.0M	8.0M	36.1M	2.3B
Pre-money valuation	716.0M	384.0M	21.4B	88.0B
Amount raised in IPO	61.0M	54.0M	1.7B	16.0B
Post-money valuation	777.0M	438.0M	23.1B	104.0B
IPO DR	8.52%	14.06%	7.77%	18.18%
IPO VCER	716.0x	48.0x	593.7x	38.7x

I am using a very loose definition of “startups” in this post: any VC-funded private company that aspires to grow rapidly is a startup. Many such companies should no longer be called “startups” in later stages.
Here is my explanation for Uber’s underachievement: Uber had no upside.
Microsoft grew its business with no outside funding before the IPO. They raised $1 million to get a venture capital company onboard for “some adult advice”, but never spent the money.
Early stage startups are usually very open to share financial metrics with prospective candidates. Make sure to ask for such metrics if you are interviewing with one.
The definition of the dilution ratio is adapted from chemistry. According to Wikipedia, the dilution ratio is ratio of solute to solvent. Technically the pre-money valuation should be the solute, because existing shareholders are being diluted, not the new money raised. However, I prefer to treat the new funding as the solute, and the pre-money valuation as the solvent. This way the dilution ratio is usually a lower number, as is shown in the examples that follow.
If the pre-money valuation is the solute and the new money raised is the solvent, Company A’s dilution ratio will be 10:1.
If the pre-money valuation is the solute and the new money raised is the solvent, Company B’s dilution ratio will be 100:1.
The proper definition of VCER uses Enterprise Value, which is again not possible to get without insider knowledge, and hence I use the pre-money valuation instead.

Learning accounting to read earnings reports

Sun, 10 Jul 2022 00:00:00 +0000

Will Larson wrote this Reading a Profit & Loss statement article on his blog, which prompted me to share how I learned accounting to read earnings reports. This skill has not made me a good stock picker, but I found it helpful to understand the financial positions of my employers, which were not always in line with the narrative provided by higher-ups. This allowed me to make independent career decisions, and I hope it is a useful skill for you too.

As is the case for almost every other skill, my journey took some learning and practice. I went with the Introduction to Finance and Accounting Specialization by University of Pennsylvania on Coursera, but I think every 100-level undergrad course in the finance and accounting field works. There are also many accounting 101 books out there, although I haven’t read any and hence cannot make any recommendation.

Will’s blog post focused on the Profit & Loss statement (also known as the Income statement). In addition, I also found the Balance Sheet and the Cash Flow statement very useful. All of them are explained with examples in a typical finance and accounting course like the one above.

For practice, I spent some time going through historical earnings reports for a few public companies on EDGAR. Here are the reports I paid the most attention to:

S-1: the go-public report that contains historical financial statements for a few years leading up to the public listing
10-Q: the quarterly reports
10-K: the annual reports

These reports usually provides year-over-year comparisons for only a single year, so I built some spreadsheets with data going back 5 or 10 years to understand longer term trends. This exercise served as a good practice to solidify my knowledge, but it could get tedious over time. Luckily I found some free tools to help:

Stockrow provides free financial data going back 10 years. It also offers convenient deep links to 10-K and 10-Q reports on EDGAR.
Koyfin provides free financial data going back 2 years. It does have a nicer interface though.

And that is what I want to share on learning accounting today. Hope it helps!

Leadership is not a title

Sun, 13 Feb 2022 00:00:00 +0000

Yesterday I got a recruiting email for an Engineering Leadership opportunity, and I had got similar ones before. The title Engineering Leadership makes me quite uncomfortable, although it is understandable why the recruiters wrote it. For years, I was not able to tell the difference between leadership and management, especially in the engineering context. I have, however, come to my own definition over time. In my opinion, leadership is not a title reserved for (senior) management, but a skill that could and should be practiced by every individual.

Leadership versus Management

Leadership is indexed on creating value. Everyone could and should create value. A leader on a team cares first and foremost about how to maximize value creation by the entire team. The leader makes sure everyone on the team is aligned on a shared vision. They take on grunt work passionately if that is what it takes to succeed as a team. They will take accountability when there are signs of failure, but they will not worry about how to split the reward before the team succeeds.

Management is indexed on empowering people. Everyone could and should empower people. A manager on a team recognizes first and foremost that the power of any single person is limited. It takes a village, as they say. The manager will enthusiastically build the right team for the mission, create the right environment and culture for everyone to thrive, and coach each individual to maximize their potential.

Both leadership and management are important skills to have. The term management also happens to be the job title for a manager, primarily because contemporary performance review systems have to be scaled in a hierarchical organization¹. Despite not having the term leadership in the title, it is also absolutely critical for managers to possess the leadership skill because the performance of a manager is ultimately evaluated by the amount of value creation by their team. However, a manager’s job will be miserable if they are the only person to demonstrate leadership on their team. As is typical in the tech industry, many organizations have designated Tech Leads who obsess with value creation by their respective organizations.

Leadership is not technical expertise

Leadership is not technical expertise. As is defined above, leadership is not about proficiency in certain technologies or mastery of certain subjects. As a manager, one easy trap to fall into is not to give enough credit to rising leaders simply because they are not “senior enough”. An equally dangerous slope is to blindly require every “senior engineer” to developer their leadership skill. I have learned both lessons the hard way by committing the mistakes by myself. To illustrate, let me refer to the example of true leadership in a different field.

Tom Brady is probably the greatest of all time in American football. He played statistically at the highest level well into his forties and most recently won the Super Bowl at the age of 44² last year. In a sport that is dominated by raw athletic talent, Brady did that by being a true leader. His singular focus is to win, and he is willing to do everything it takes to win: from a very disciplined diet and lifestyle, to being the most hard-working player; from voluntarily taking pay cuts to keep his team competitive, to supporting and protecting the most impossible teammate. My personal favorite is Brady’s debunking of his own father (with a sense of humor) to protect his previous team and coach, who were criticized by Brady Sr. for treating his son unfairly. Brady’s name was never associated with agility, strength or velocity even in his twenties, yet he retired last week with the most Lombardi Trophies in history. It is too bad that Tom Brady, as Patrick Collison would say, is not going to make a few more slam dunks in Super Bowl LVI later today.

Leadership to be continued

I argued what leadership is not in this post. It is already getting long, so I will save what leadership is for another article. For those of you who are going to watch the Super Bowl, I predict there will be no fewer touchdowns scored than slam dunks. Enjoy the show!

Effective performance measurement systems are vital to the life and death of any aspiring organization, as is pointed by Paul Graham in a tweet and an essay. In a large enough organization, contemporary management principles center around breaking down objectives and holding individual managers accountable. This arrangement seems reasonably effective to me, but I would not rule out a future where management stops being a job title, but merely a skill for everyone to practice and deploy.
Technically Brady was 43 years, 6 months and 4 days old on the day of Super Bowl LV. I am quoting 44 as his nominal age under East Asian age reckoning.

Performance Outlook

Sat, 12 Feb 2022 00:00:00 +0000

I have a framework for career conversations. It comprises of ladder reading, performance heatmap/brag docs, and career development plans. Since writing that post, I have learned and evolved the way to create the career development plan. Let me call it the performance outlook¹. The idea is the manager and the employee can work together to write down their performance review for the next performance review cycle, which covers what success looks like for them using concrete examples. I would like to thank Joe Gershenson from whom I learned this technique.

Recapping Career Development Plan

Here is what I wrote for the original career development plan:

The final structured step would then be creating a career development plan. The plan would be comprised of four sections: current condition, target state, proposed projects and practical advice. The current condition and target state would be derived from the ladder reading and performance heatmap exercises. The proposed projects and practical advice would then be tailored to that condition and target. For example, suppose an engineer would like to move to the next level. They were technically competent, but needed to demonstrate project coordination and leadership skills. The proposed projects would be breadth-shaped and require cross-team or cross-functional collaboration. The practical advice would be to focus on communication and mitigating schedule risks.

Enter Performance Outlook

The core idea is to write down the performance review together with the employee as if time is up for the current review cycle. For example, suppose there is one performance review every year in January. The current time is February 2022, but we will pretend it is already January 2023. It is time to grade their performance for the calendar year of 2022. We will follow the usual format of evaluating performance: start with the impact (the what), and supplement with the contributions (the how). In particular, we will describe how the employee have improved their skills in areas of development that were highlighted in their most recent performance review.

There are a few key details I try to adhere to in authoring performance outlook:

Always use the present or past tense. This is counter-intuitive because the performance review is going to happen in the future. Everything written down in the exercise is imaginary no matter how likely or unlikely it appears. In fact, the uncertainty is exactly why it is important to write in the present or past tense. When writing in the future tense, I tend to leave a lot of room to account for variance, but that will only create confusion and anxiety for the employee. Using the present or past tense forces me to be crisp, which helps a lot in creating alignment on expectations.
Use specific and achievable examples. This is an extension of the first point above to make sure clear expectations are set. The typical employee ladder definitions can be very abstract, so it helps to be explicit here. For example, instead of “demonstrating leadership”, write down “identifying waste in our cloud spending and delivering $X in cost savings”.
Differentiate between success and outstanding success. It is very motivating to include stretch targets in the outlook, but the writing should make such targets clear to separate from the table stakes. A helpful way is to organize such targets under an explicit heading for outstanding success. An alternate is to create an entirely standalone performance outlook for the outstanding success scenario.
Write with strong yet loosely-held convictions. The write-up should express strong convictions for an opinionated future, but it should not enslave the employee or the manager to that particular version of the future. Both of them should be on frequent lookout for changes in the surrounding environment. Both should respond to changes by adjusting or rewriting the performance outlook promptly.

I have come to love this exercise. The bonus is it can be done by yourself too. It is always a pleasant conversation when an employee creates a performance outlook on their own. I have also been doing this exercise for myself.

Please give it a try and let me know if you like it.

I chose the word outlook due to its use in finance for earning expectations.

Cryptocurrency Mining: Decentralized Centralization

Fri, 14 Jan 2022 00:00:00 +0000

I dipped my toe into Ethereum mining over the holiday break. My intention was not to make money¹, but to learn the underlying infrastructure powering cryptocurrencies. After all, most of my career has been spent on dealing with distributed systems. The journey impressed me. I am fascinated to see how much centralization there is despite the best intentions that were put into designing a decentralized system. I am also amazed by the creative entrepreneurs who discovered and delivered on business opportunities through decentralized centralization.

Understanding cryptocurrency mining

To understand how cryptocurrency mining works under the hood, I followed the 2013 article titled How the Bitcoin protocol actually works by Michael Nielsen. The article strikes the balance of enough technical discussion and not too many implementation details. While it focused on Bitcoin, the general principle applies to Ethereum and other Proof-of-Work based cryptocurrencies.

In a nutshell, cryptocurrencies are distributed ledgers among a trustless network, and Proof-of-Work is the mechanism that allows the decentralized network to come to consensus, or agree on things like account balances and the order of transactions. The mechanics typically includes finding a nonce which, when concatenated to a message, produces a SHA256 hash that, when interpreted as an unsigned 256-bit integer, is less than the network-specified target, i.e. the mining difficulty. A nonce needs to be found for every block, which is produced regularly over time - about every 10 minutes for Bitcoin and every 10 seconds for Ethereum. The network participant who finds such a nonce first will get rewarded with a certain amount of the corresponding cryptocurrency. The reward is created out of thin air and part of the consensus protocol. Finding a nonce is computationally expensive, and the reward is an incentive for network participants to provide computing resources and to play by the rules.

The general concept seems reasonable so far. The protocol is beautifully designed to allow trustless consensus to form in a decentralized network. How does it work in practice then?

Mining pools

The first tactical decision for me was whether to join a mining pool. My GPU could compute hashes at a rate of about 33M per second(33MH/s), compared to the total hashing power of the Ethereum network at 1PH/s. In probability terms, that means I could expect to find a good nonce for every 30,000,000 blocks. Assuming a 10-second block time, that translates to getting a reward once every 9.5 years! That is obviously bad for a whole host of reasons:

That 9.5-year number is an expectation, not guaranteed. So I may have to wait even longer.
The total hashing power is likely to increase over time, furthering delaying my expected payout.
Any down time in my mining computer would work against me as well.
The Ethereum network will probably switch to Proof-of-Stake long before my expected payout, rendering my mining useless.

Given all of these, it is a no-brainer for a hobbyist miner like me to join an established mining pool. A mining pool participates on the Ethereum network as one single entity, but it leverages the computing power of all the members in the pool. Even a small-ish mining pool with 1% of the total hashing power is expected to get awards for 3.6 blocks every hour, which greatly smooths out randomness in the earning curve.

Joining a mining pool is straightforward, but it represents a philosophical shift. I would not need to trust anyone by mining directly on the network, but now I need to trust a central authority, the mining pool operator. After all, they get the block awards directly in their own wallet, and I need to trust that they will calculate my share correctly and pay me on time. In fact, the rules of my mining pool dictate that members get paid once they hit a certain amount², which will take me about 200 days to get. That is certainly better than once every 9.5 years, but in a sense I am offering a no-interest loan to the mining pool operator whom I was incentivized to trust despite not knowing anything about them.

A mining pool is obviously a great business with strong network effects. The more hashing power a mining pool has, the smoother the earning curve will be for its members. The more members it has, the more hashing power it gets. The pool operators charge 1% to 3% from their members, which is a classical software service with zero marginal cost. The strong network effects naturally lead to centralization. In fact, the leading Ethereum mining pool, Ethermine, accounts for more than 30% hashing power of the entire Ethereum network. The top three commands 56%, enough to launch a 51% attack. Ethereum mining is not something an average user can afford to do alone any more. The situation is not too different for Bitcoin or any other mainstream cryptocurrencies.

Mining software

The second tactical decision was to choose a mining software. My first intuition was to lean on open source mining software, but most of them have not received major updates for years. On the other end, closed source mining software is a booming business. They typically charge a commission of 1-2% for fully integrated solutions which push your GPUs to their limits while protecting their longevity. Such software is sketchy though. They use anti-reverse engineering techniques to protect the binary executables. This is perfect reasonable since otherwise it would be super easy to hack the binaries and avoid paying the commission. However, it does trigger plenty of warnings from most anti-virus software, and invites questions about what is actually happening under the hood.

Nevertheless, the business of closed source mining software seems to be getting more popular by the day. It is such a competitive business that they even employ consumer promotion tactics like lottery draws for loyal customers. This business is also built on network effects with typical software margins. The more money the developer makes, the better job they can do in improving usability and performance. The more usability and performance improves, the more users they will attract. The more users, the more commission they will make. This space is more fragmented than mining pools though, presumably because it is not super difficult to create high-performing miners. If mining pools are SaaS businesses, mining software is a freemium software business without App-Store-like distribution³. The centralization is still staggering and potentially dangerous since no outsiders know what is in the software binaries.

Decentralized centralization

Both mining pools and mining software are centralized businesses built on network effects inside a decentralized ecosystem. Credits must go to the early entrepreneurs who discovered and executed against these business opportunities. In a sense this is not too different from how aggregators like Google built powerful businesses. The Internet itself is decentralized by design. Google did not become powerful by controlling every website. It built a strong business by aggregating all websites and making it a no-brainer for consumers to ask Google for information. The centralization is a natural result of network effects. The zero marginal cost nature of Internet and software made such network effects stronger than ever.

Such centralization can also be dangerous though. Top mining pools can execute a 51% attack by just coordinating with a small number of counter parties. Popular mining software developers can change their commission rate without revealing it. Even if they are honest and operate with integrity, the centralization makes them ideal targets for outside attackers. Someone can launch simple denial of service attacks against top mining pools to cause major disruptions to the entire Ethereum network⁴. They can also plant malicious code into popular mining software without users noticing.

Ethereum’s giant migration from Proof-of-Work to Proof-of-Stake could probably alleviate the problem, but staking pools are already emerging to be the next natural point of centralization. Solana’s Proof-of-History is another interesting alternative, but it is an even high barrier of entry to run a Solana validator node.

Takeaways

Centralization naturally arises where network effects emerge. Decentralized networks bring down marginal cost and beget network effects. Moxie’s first impression of web3 discusses the ways centralization emerges in the user-facing application space on top of major cryptocurrencies. To be fair, the application developers do intend to reduce reliance on centralized services. My experience with mining shows that such centralization is even more lopsided in the infrastructure layer where blockspaces are provided. More interestingly, the infrastructure centralization aligns interests between big and small players, which means the centralization is unlikely to break. Personally I am not paranoid about the philosophical debate of centralization versus decentralization. It is much more interesting for me to watch dynamic businesses get built to meet real user demands in real markets.

At a recent high level of $4,000 per Ether, my estimated earning is $2 per day.
The payout threshold is perfectly reasonable because the transaction fee is astronomically high to transfer Ether from the pool’s wallet into my own wallet.
I have distinct memories of downloading random software over dial-up connections from the 1990s. It was both exciting and scary. I was burnt by viruses like the CIH badly in the early days.
If I understand correctly about how the Ethereum protocol works, sudden loss of major hashing power could lead to decrease of mining difficulty. An attacker would have the incentive to take down major mining pools and run their own mining operation on the side with reduced mining difficulty, which allows them to get rewarded richly within the rules of the network. An ambitious and resourceful attacker could even temporarily control more than 50% of total hashing power while the top pools are down.

Uber had no upside, Part Deux

Sat, 11 Dec 2021 00:00:00 +0000

My previous post, Uber had no upside, quickly became the top-ranked post by page views in one day. I am much grateful for your appreciation¹. I am equally inspired by the hard questions some have raised². In this follow-up piece, I would like to address the major points surfaced in the discussions. Please bear in mind this is not investment advice though.

A review of the discourse

Broadly speaking, there were three categories of discussions:

Uber has a deep moat in economy of scale through its network effects. It would be uneconomic for a new upstart to compete.
Uber could use its low-margin businesses to fund growth projects which could be high-margin, just like Amazon did with retail and AWS.
I was contradicting myself by saying Uber had no upside while still holding a meaningful position in $UBER stock.

Let me try to address each of them in the following sections.

How to compete with Uber economically

Reader Martin Kess wrote that Uber’s high utilization and reliability are a moat. This brings down cost-per-ride, and makes it difficult for a new upstart to compete. Let’s say, for a average $10 ride, the cost per ride is $8 for Uber but $12 for the upstart. The upstart could choose to subsidize $8 and bring price down to $4, but Uber would only need to burn $4 to compete. The upstart would obviously be in a very inefficient position. It would be unwise for any upstart to enter the market and start a price war even in a single metro area.

Or is it?

Contrary to intuition, it is Uber that would be in a very bad economic position if there were a price war. The problem is exactly Uber’s high market share. Let’s say a new upstart kicks off a price war in one city where Uber commands 100% of ride-hailing market, which is 1 million rides per day. The upstart sets a price of $4/ride (hence losing $8/ride) and quickly grows to 50k rides per day, or 5% of the market measured by the number of rides. Uber lowers its own price to $4/ride (hence losing $4/ride) to avoid further losing share (hence doing 950k rides/day). At this point, the upstart is burning $400k per day, but Uber is lighting $3.8M on fire per day, 9.5x the amount the upstart is spending!

The fundamental problem here is that Uber cannot selectively subsidizes rides that would be taken away by the upstart. Instead, Uber has to subsidize all rides in the city. Uber’s dominant position becomes its Achilles’ heel in a price war.

To make matters worse, the economics tip more into the upstart’s favor in a number of ways:

The more the upstart subsidizes, the higher the pressure Uber will feel. If the upstart sets the price at $2/ride, Uber will need to burn 11.4x as much in the above example³.
Because of #1, the upstart has pricing power despite being the smaller player. Uber has to be reactive and match the price even if it will feel more pain at lower price points.
There is no floor in the price war. $0 rides are plausible for the upstart, and the price can even go negative. That negative number could come in the form of rider promotions or driver incentives. This is even worse news for Uber because the math is not in its favor when the price drops even more into the negative territory due to #1.
The upstart’s efficiency will improve as it gets more market share, meaning that it could burn less money per ride. On the other hand, Uber’s efficiency will worsen as it loses market share, causing it to burn more money per ride. Ironically, the relative spending difference between the two will converge despite the efficiency change in opposite directions. Again, the important variable here is the market share for each player. Both efficiency and cash burn are functions of the volume.
Up until this point we have assumed the market to be static at 1 million rides per day. In practice, the market will expand due to lower prices. This is arguably bad news for Uber again, because it has to burn even more money comparatively as long as it has vastly higher market share.
Another huge assumption we made was that Uber had economy of scale, i.e. it could burn less per ride. That may be true, but the degree is probably low. Uber obviously incurs a large variable cost per ride - about 80% of its gross bookings goes to the driver⁴. Even within the 20% that Uber takes home with⁵, a large portion is used to pay variable costs like insurance. The fixed costs of paying the corporate employees and developing the software are where the economy of scale comes from, but that portion is small, only accounting for 4% of gross bookings according to Philo’s most generous estimation⁶. The new upstart does not need to spend billions or years on engineering to get an okay mobile app with passable user experience, and not a single venture capitalist will blink their eye to fund the fixed cost part of software development.

This was largely the playbook that Uber used to compete with Didi in China, coming in as the smaller player. Upon realizing its own disadvantageous position by being the dominant player in China, Didi pulled a masterful trick. It funneled investment into Lyft which staged a price war with Uber in the US from the advantageous market position of being the smaller player. In fact, Didi formed the anti-Uber alliance globally to finance local Uber competitors around the world. Uber was forced to leave China because it could not sustain the cash burn outside of China⁷. Didi could thank Sun Tzu for writing Besiege Wèi to rescue Zhào in the 2,500-year-old The Art of War.

Uber knew the precarious position it was in. That was why Uber kept raising billions after billions of dollars. Contrary to popular belief, the colossal funding rounds indicated not strength, but weakness. It was a last resort because Uber had no way to differentiate its product.

Uber’s high-margin dream

Several readers, in public⁸ or in private, has expressed that Uber could use its low-margin businesses to fund growth projects, some of which will hopefully be high margin. After all, this was how Amazon became a trillion dollar company, discovering AWS after spending many years in red ink for the retail business.

I actually gave one concrete example in my previous post. Meituan successfully built a high-margin ads business on the back of a low-margin food delivery business. However, I also explained why this is unlikely to happen for Uber, at least in the US market.

In more general terms though, the comparison between Uber and Amazon is a deceptively dangerous trap. Yes, both companies started with low-margin businesses involving moving atoms in the physical world. And yes, both companies spent their respective first decades losing money. However, there is at least one fundamental difference between the two from the financial perspective. Amazon was operating cash-flow positive⁹ but Uber was not¹⁰. To put things into perspective:

Amazon was founded in July 1994. It was operating cash-flow positive in its first full year in 1995, and it has been that way ever since for every single year, with the exceptions of the years 1999, 2000 and 2001, when Amazon was aggressively investing for growth at the peak of and after the burst of the dotcom bubble. Amazon did amazingly well in the 2008 recession, growing its operating cash-flow by 20% from 2007 to 2008, and then growing another 95% the following year!
Uber has never been operating cash-flow positive in any year since 2016, the earliest year for which there is publicly available financial data. From 2016 to 2020, Uber burnt through almost $13B cash in operating activities in the 5-year span. For context, Uber raised a total of $15B in cash from its founding in 2009 to mid-2016 when it reached a $68B valuation.

In other words, Amazon losing money was an accounting gimmick. It was reinvesting every penny of profit into future growth. One can read the 2004 shareholder letter from Jeff Bezos¹¹ to understand why Amazon singularly focused on cash flows instead of profit & loss¹². On the other hand, Uber did not have much extra money to invest into future growth, because it was burning though piles of cash in its core business for an extended period of time.

Amazon raised a total of $108M¹³ in outside capital in its entire history, and built a trillion dollar company by aggressively reinvesting operating cash flows. Uber has raised a staggering $25.2B¹⁴ to date, and is still operating cash-flow negative¹⁵. The margin for error was just too small for Uber.

My opportunity cost

Now on to the am-I-contradicting-myself question. The short answer is yes, but the full answer requires more nuance.

My primary consideration back in late 2017 was the opportunity cost of my career. It was important to work for an employer with (hopefully massive) growth prospects. I got more than half of my yearly compensation in the form of illiquid equity grants, so I had to think hard into the future and project the risk-adjusted payoff. I was given a meaningful 4-year RSU grant when I joined Uber in October 2015. I was awarded another sizable 4-year RSU grant in April 2017 due to good performance. I was walking away from more than half of my total equity grant in December of that year, because I saw no upside. I could get a much better deal elsewhere even if the new offer was lower on paper at that moment, because I would place much higher premium in the upside of the new company. From that angle, I would argue that my actions were consistent with my analysis.

There is also a real opportunity cost for my current $UBER holdings. As Murat U. and Hersh Mehta pointed out, I could have earned a higher risk-free return by selling and reinvesting into broad-market index funds immediately¹⁶. To be fair, my current holding is less than half of the total I acquired via my two-year employment. Nevertheless, I have to admit that I have nothing but emotional attachment by holding onto the remaining shares. I guess the takeaway is that even rational people with great analysis can make irrational decisions, which could be considered as my tiny contribution to disprove the Efficient Market Hypothesis :)

My tweet got 10 likes, which is the highest in over a year! :P
My LinkedIn post received a number of inspiring comments and questions on top of 100+ likes.
Please use this as an exercise to validate my math.
Technically driver earnings is not a GAAP expense in Uber’s accounting, but that is irrelevant to our discussion here.
According to Uber’s Q3 2021 report, Uber has a blended take rate of 21% across Mobility and Delivery.
In Ride-Hailing: Is It Sustainable?, Philo estimates that Uber will spend $2B in R&D on a little under $100B of gross bookings in 2021. Double that spending and you get a generous 4% number.
The speculation for the reason behind Uber’s China exit was a logical inference. I had no material non-public information at the time of the decision, nor do I have any today. The decision was way above my pay grade as a fungible software engineer.
See Hai Wang’s post on building deeper moat into Uber.
Source: Amazon’s 10-K filings
Source: Uber’s S-1 and 10-K filings.
You should really read all 24 shareholder letters Bezos wrote.
Bezos noted that, in his 2004 shareholder letter, Amazon does “not focus first and foremost, as many do, on earnings, earnings per share or earnings growth”. He also mentioned “that a focus on EBITDA … would lead to the same faulty conclusion about the health of the business”.
Source: Amazon Financials on Crunchbase
Source: Uber Financials on Crunchbase
According to its Q3 2021 report, Uber’s 2021 year-to-date operating cash flow at the end of September was negative $338M. It was still negative, but an order of magnitude lower than the past few years.
This post is not investment advice. I may transact in the securities mentioned in this post without providing any updates here.

Uber had no upside

Mon, 06 Dec 2021 00:00:00 +0000

I enjoyed a great read of Ride-Hailing: Is It Sustainable? by Philo from MD&A¹. It sent me back to late 2017 when I decided to leave Uber. At the time, many friends and colleagues asked me why. The answer was because I saw no upside in Uber. In this post, I would like to take a detour from the usual engineering management topics and talk about business. I hope it is useful since it is probably more important to decide whom to work for in a career. Please bear in mind this is not investment advice though.

(Edit: this post sparked some inspiring discussions on the Internet, which encouraged me to write a follow-up post. I do suggest reading both to get a full picture.)

Uber stabilized in late 2017

Uber had a turbulent year in 2017. From the Delete Uber campaign in January, to Susan Fowler’s heartbreaking blog post in February. From the unearthing of other harassment and business conduct claims throughout the spring, to the tragical death of Bonnie Kalanick, mother of Founder & CEO Travis Kalanick, over the Memorial Day weekend. The chaos finally came to an end when Dara Khosrowshahi took over as CEO before the Labor Day. Dara brought hope to the company when he projected an 18-month timeline to take Uber public².

I started interviewing in October, right after Dara started, and gave my notice in December. So it was understandable for my friends and colleagues to ask the “why now” question. A lot of Uber employees left in the chaotic first half of 2017, but things had stabilized over the fall. After all, who did not want the IPO excitement now that there was a time table to go public?

I was also doing well at Uber. I was an established tech lead trusted by my team, my manager, and my organization. I just had a stellar performance review in the spring of 2017. The accompanying stock refresh pushed my yearly compensation package to a level unmatchable from other tech companies³. Why leave?

Profitability was baked in

The answer was because I saw no upside in Uber. I did and still do believe Uber created massive value for the society. It expanded the ride-hailing market by at least an order of magnitude⁴. It was and still is literally impossible to get a taxi from the Seattle-suburban neighborhood I live in, but now I can get a Uber within 10 minutes just at my fingertips.

Uber was still losing money at the time, but I believed that the Uber business could be sustainable. However, I suspected that Uber’s sky-high valuation had already baked in long-term profitability of a low-margin business in a highly competitive market. Uber would need to deliver new high-margin business at scale to justify any future upside.

One data point was the valuation. I joined Uber in October 2015, when the valuation was $50B at a per-share price of $48.77. When I left Uber in December 2017, the valuation had risen to $72B, but the per-share price stayed flat at $48.77. That meant Uber the business did not create any value in the two-year span from the investors’ perspective. New capital infusion was responsible for all the valuation increase. All existing shareholders, including employees, were diluted as a result⁵.

Another consideration was Dara’s performance as the CEO of Expedia prior to joining Uber. It appeared that Dara excelled at optimizing operations and doing mergers & acquisitions, but he was not as good at leading product innovations and creating new business opportunities. If that were to hold true for Dara’s tenure at Uber, the best case scenario would be cutting losses and achieving profitability, which again implied limited upside.

These were my primary considerations when deciding to leave in late 2017. While I stand by that decision today, my thinking has certainly evolved over the past four years. The sections below have more.

The missing monopoly profits

There was an argument that Uber should have acquired or could still acquire Lyft to reap monopoly profits. That claim seemed dubious to me. We can leave the regulatory approval question aside, and just think about market dynamics for the moment. Uber didn’t seem to possess any deep moat other than the network effect built up in a two-sided marketplace connecting drivers and riders. That moat was fragile though, as Lyft had shown when they took advantage of Uber’s forgettable 2017 and grew from ~10% to ~30% market share in the US. It was not a stretch to imagine a new player entering the market and undercutting Uber, had Lyft been acquired or driven out of the business by Uber.

The counter argument is why a new investor will still fund a Uber-like business when they know it is a race to the bottom. That is a legitimate question, but I think someone might still do:

If Uber were a monopoly and were to raise prices, a sufficiently capitalized competitor would have the incentive to enter the market, compete on prices while offering similar quality of service, and take share from Uber.
If Uber, fully aware of the competitive threat, didn’t raise prices despite being a monopoly, it would continue to live on thin margins hence having limited upside.
Even in a competitive market where companies race to the bottom, there are still plenty of newcomers every now and then. The airline industry is capital intensive and known for being loss makers to promote their loyalty programs. There are also quite a few commercial airliners on the market.

Another counter argument is Uber’s network effect. The theory goes like more riders lead to higher utilization for drivers, which leads to higher earnings and more drivers on the platform, which then brings down prices and attracts more riders. AirBnB seems to have a strong network effect. It has been successfully fending off competitors so far. Why would Uber not benefit from the virtuous cycle and fight off any potential competitors? Here is a hypothesis:

Uber’s network effect is local to metropolitan areas. I believe most Uber trips are taken by local riders and operated by local drivers. A new upstart could focus on just one city, run promotions and pay incentives, and quickly undercut Uber’s business in that city. The capital requirement is manageable, and it could raise more capital and expand to other cities once it has shown some success.
AirBnB’s network effect is global. A typical traveler would not go to the same destination for repeat vacations. As a result, it would be hard for new upstarts to compete. They could focus on finding hosts in one hot destination, e.g. Hawaii, but they would still need to run national or global promotion campaigns to win mind share from travelers. Alternatively, they could focus on getting travelers from one source area, e.g. Seattle in winter, but they would need to sign up hosts in all the popular destinations. In other words, AirBnB’s network effect is more robust.

The hypothesis is partially validated when DiDi, Uber’s almighty competitor in China and globally, went public along with its financial data. Didi merged with Kuaidi and acquired Uber’s China business many years ago, but there are no signs of monopoly profits at all. Instead, Didi faced fierce, localized competition from Meituan and iZuche.com after the ride-hailing war in China was declared over in 2016 when Uber exited.

Could Uber lobby for favorable regulations to prevent other companies from even entering the market? It is possible, but it is probably not good for Uber either:

Regulations imply cost. They will cut into Uber’s already razor-thin margin. Just look at how hard Uber has been fighting against regulations to treat drivers as employees.
Artificial monopolies created by regulations tend to be fragile⁶. One only needs to look at the market value of New York City taxi medallion, a government-created monopoly that Uber destroyed not long ago.

The underachiever in Uber Eats

Uber Eats was believed to be the next growth engine after rides, and it is indeed a larger business⁷ than ride-hailing for Uber in 2021 due to the coronavirus pandemic. Nevertheless, it still under-achieved relative to its potential. Uber Eats was on track to be No.1 in the US food delivery category in 2017, but it is now a distant second to DoorDash. What if Uber Eats were to maintain its leading position? Would the food delivery business unlock a lot more upside for Uber?

I would still be quite skeptical in that world. Meituan is probably the most successful business built on the back of food delivery, but its profit driver is the high-margin ads business to rank restaurants in its app⁸. Amazon employs a similar tactic in its retail listing page to extract margin from third-party merchants. Meituan does the same to restaurants who want to be ranked higher in the app. That ads business requires high-density metro areas with hundreds of selections, which does not work in most of US suburbs⁹. There are exactly four Asian restaurants that Uber Eats can deliver to me under 30 minutes. It is irrational for any of them to pay Uber in exchange for higher ranking.

The pipedream of self-driving cars

The final argument is that self-driving cars will turn Uber into a cash cow. Uber’s largest expense is driver payouts¹⁰, and they can drastically lower prices for rides and still literally print money with self-driving cars. After all, isn’t that why Kalanick acquired Anthony Levandowski’s startup and got into an expensive legal battle with Waymo, Alphabet’s self-driving unit? The payoff has to be extremely high to justify the investment, right?

I am not buying that argument either. First, it was not a given that Uber would be the first company to commercialize its own self-driving technology. Of course this option is now officially dead after Uber sold its Advanced Technologies Group to Aurora. However, even before the deal, the fact that Uber had the largest ride-hailing network did not appear to give it any advantage in developing self-driving technology. Tesla has millions of cars and volunteer users/testers on the road. Waymo has the longest actively-operated fleet of test vehicles. Startups like Cruise and Aurora are focused bets that will either make self-driving work or go out of business. All things considered, it was unlikely for Uber to deliver either the first or the best commercial deployments of self-driving technology.

Now that Uber is no longer developing its own self-driving technology, could it license such technology once it is available and mature? Maybe. Would that unlock a lot more upside? Probably not. Let’s say some other company develops such technology and is ready to scale it. I think they have a lot of options favorable to them and unfavorable to Uber:

License the technology to Uber but ask for steep royalty premiums. Uber would still accept the deal because it would be cheaper than human drivers, but the leftover margin to Uber would be thin.
License the technology to Uber and Uber’s competitors. Capture all the value while the ride-hailing service providers race to bottom. This has happened in the PC industry where Intel and Microsoft captured all the value while the OEMs competed to death¹¹.
Create their own ride-hailing service and compete directly with Uber. They could offer superior experience and much cheaper rides thanks to their self-driving technology.
Partner with an established consumer brand to create a new ride-hailing service and compete with Uber. This is the same as #3, but they could offload the rider relationship to an entity with much better brand recognition, for example Apple, Amazon, Facebook or Google.

Uber definitely has the competitive advantage in scaling the ride-hailing marketplace. In particular, Uber has the operational excellence in managing its fleet of driver partners, but this is irrelevant in the self-driving future.

Will I be proven wrong?

I saw no upside for Uber four years ago. I still see very limited upside for Uber today. That does not mean I am right though. I am writing this to share my reasoning rather than my conclusion. In fact, I will be happily proven wrong if $UBER takes off some day, as I still hold a meaningful portion of my employee equity grant¹².

I found the article from the weekend longreads via The Diff by Bryne Hobart.
Uber did go public in May 2019, roughly 20-months after Dara became CEO.
I interviewed at Stripe and Facebook, among other companies. None of them were able to match my yearly total compensation package at Uber based on the then-current valuation for each company.
To learn how Uber expanded the ride-hailing market, read Ride-Hailing: Is It Sustainable?, the article I linked at the beginning of this post.
Uber went public at $45 per share and a $82.4B valuation. In other words, it had more dilution at lower price; late-stage investors lost money; most employees were underwater.
On the other hand, natural monopolies are more robust against regulations, even when the regulations were aimed at the monopolies themselves. GDPR was supposed to contain Facebook and Google. Instead, they became even more unstoppable in the EU because smaller competitors cannot afford the cost of GDPR compliance.
According to Uber’s Q3 2021 report, Delivery is ~30% more than Mobility in gross bookings, and marginally higher in revenue.
See online marketing services from Meituan’s 2020 annual report.
One primary reason for DoorDash’s success was their focus on suburbs.
Technically most driver payouts are counted in gross bookings but excluded from revenue, which means it is not an expense line item. Uber only books its own cut as revenue.
Remember IBM was the original PC OEM but they cut their loss long ago. Uber’s dominant position in the ride-hailing market does not provide any safeguards.
This post is not investment advice. I may transact in the securities mentioned in this post without providing any updates here.

Getting to know me

Sat, 06 Nov 2021 00:00:00 +0000

It’s been a long hiatus since my last post. Many thanks to all who are still tuned in. My last post was help me understand you, where I ask favors from new teammates to understand them better. This post is the mirror piece. I have found it useful to create a document for coworkers to understand me better. I call it the getting-to-know-yuan document.

Write for the audience

My primary audience of the document is my team. It would not be a terrible choice to name it help-you-understand-me, but the name deserves some considerations. Not everyone has asked me to help. What if they don’t need my help? For that reason, I settled on the name of getting-to-know-yuan.

As for the content, I find it helpful to cover my management philosophy and general operating approach. It is also beneficial to supplement with a management handbook which contains tactical advice on how to better work with me. A bonus section is how to tell I am grumpy and how to help me, although I am yet to created my own version of it.

It is super valuable for me to review it from time to time. To some extent, I think the document has provided more value to myself. It reminds me to follow the best practices I created for myself! It is also important to update the document as I learn new things.

Getting to know Yuan

Below is the latest version with confidential information removed. Feel free to tweak and reuse for your own benefits. I cannot claim credit either since mine is also based on many great examples I have seen at Stripe.

If you are reading this, I am really excited to be working with you.

Management Philosophy

My management philosophy is heavily influenced by systems thinking. To me, a manager’s performance is evaluated by the collective output of their team. Therefore, a manager needs to be flexible in doing whatever is needed from them to deliver the desired output. On a high level, to paraphrase Niels, I believe an engineering manager is responsible for creating healthy and well-executing teams:

Well-executing means:

Understand what the business needs from the team, and set the right goals for the team

Know the execution status for the goals

Help the team unblock when they encounter hurdles

Ask for help from their own manager and/or the organization when necessary

Healthy means:

Create an environment of trust so that

the manager and their reports can have difficult conversations

Disagreement and conflict can be surfaced in a constructive way

Make it sustainable so that people stay engaged and don’t burn out

To put it another way, my focus as an engineering manager is on projects and people. I would utilize tools like policies and processes to help with that, but the tools are a means to an end.

Here are a few books that played key roles in shaping my understanding of engineering management:

High Output Management, Andrew Grove

Only the Paranoid Survive, Andrew Grove

An Elegant Puzzle, Will Larson

The Manager’s Path, Camille Fournier

The Making of a Manager, Julie Zhuo

Operating Approach

Team meetings

I recognize the value of regular team meetings, including sprint planning, standups, etc. I believe good processes are not designed but evolved, so I tend to rethink each meeting’s value from time to time. For example, here is a story of my previous team on adopting and getting rid of standups.

On a high level, I think the purpose of regular team meetings is to:

Help with project execution

By broadcasting status updates and increasing awareness of each individual project among a larger group

By communicating blockers and asking for/offering help

By escalating issues and making decisions to address them

Help the team members bond with each other

By checking in on each other and sharing joys and tears

By asking for and offering help on key blockers

By trusting and empowering each other on major challenges

On a new team, I would like to keep existing meetings running first. In the meantime, I will seek to slowly evolve them to increase the value we get out of the meetings as a team. I don’t think there are universally good processes for all teams, but there are great processes tailored to each individual team.

1:1s

I spend about half of my time preparing for and attending 1:1s. I prepare high-level agendas before 1:1s, and I expect my counterpart to also think about topics to discuss beforehand. I wrote about my learnings to effective 1:1s. For my team members, I also dedicate one 1:1 per month to career conversations.

Planning

I prefer to have long-term visions (e.g. two years) and short-to-medium-term roadmaps to guide execution. The quarterly planning cycle enables teams to be agile, while we also need to execute against a compelling long-term vision to avoid being trapped in local maxima. My usual plan is to refresh team charters yearly to provide long-term direction, and to track OKRs more regularly to drive short-term planning and execution.

Management Handbook

Equity

I believe in equal opportunities among team members. It includes driving team planning and owning projects. It also includes facilitating regular team meetings and participating in toilsome tasks.

Pair work by default

Each work stream should have at least two engineers by default. Exceptions can be made, but generally I find pair-working as an effective way to keep the team focused and to increase throughput. It also helps increase the bus factor over time.

Communications

I think more clearly when reading and writing than during meetings, so I prefer written communications especially when we are about to make important decisions. As an extension, I prefer meetings with written agendas and/or pre-reads. This is not intended to be a replacement for meetings, but rather a way to make meetings more efficient. I do enjoy a healthy dose of unscripted chit-chat too, but I can also ramble a lot - please stop me early!

Feedback

I welcome feedback, particularly constructive ones to help me grow. I prefer giving and receiving feedback in 1:1. Upon receiving feedback, I might need time to think and reflect. That does not mean I disagree with the feedback - I might just ask for some personal space to learn and grow :)

Goal setting

I believe in driving business outcomes via metrics / OKRs. I have a strong preference in democratizing access to metrics so that they are easier to review and to track against. That way we could have leading indicators to tell us when things need attention. Otherwise, we could fall into a trap where we scramble to find metrics after the fact to justify the impact of our work.

Execution

I usually take a hands-off approach for project execution. I will default trust individuals to make forward progress consistently. To gain some visibility into progress, I will usually ask project owners to break down projects into milestones with estimated dates, to send regular updates on progress, and to update estimates when necessary. This practice is to hold ourselves accountable, but more importantly it helps us get better at estimation and execution over time.